Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: OpenStack: Dev

Release Upgrades (was [nova] [cinder] Nova-volume vs. Cinder in Folsom)

 

 

OpenStack dev RSS feed   Index | Next | Previous | View Threaded


vishvananda at gmail

Jul 12, 2012, 2:36 PM

Post #1 of 4 (160 views)
Permalink
Release Upgrades (was [nova] [cinder] Nova-volume vs. Cinder in Folsom)

On Jul 12, 2012, at 2:22 PM, Narayan Desai wrote:

> I think that the long term maintenance or removal of nova-volume in
> its existing form is orthogonal to the actual issue of continuity from
> one release to the next.

Agreed. Discussion the volume/cinder strategy is a separate topic. I've
taken the liberty of updating the subject to keep the discussions on point

>
> At this point, we've now run cactus, diablo and are in testing with
> essex. Each of these has effectively been a flag day for us; we build
> the new system, migrate users, images, etc, and let users do a bunch
> of manual migration of volume data, etc, while running both systems in
> parallel. This hasn't been as painful as it sounds because our
> understanding of best practices for running openstack is moving pretty
> quickly and each system has been quite different from the previous.

Upgrading has been painful and we are striving to improve this process
as much as possible.

>
> The lack of an effective process to move from one major release to the
> next is the major issue here in my mind. It would be fantastic if
> (some day, ha ha ha) you could apt-get upgrade from folsom to grizzly,
> but i think that is likely to be more trouble than it is worth. A
> reasonable compromise would be a well documented process as well as
> tools to aid in the process. Each real deployment will have a
> substantial set of local customizations, particularly if they are
> running at any sort of scale. While it won't be feasible to support
> any upgrade with these customizations, tools for the process (which
> may only be used a straw man in complex cases) would go a long way.

I would like to take this a bit further. Documentation is a great first step,
but I would actually like to have an actual Jenkins test that does the upgrade
from essex to Folsom with live resources created. I think this the only way
that we can be sure that the upgrade is working properly.

The first version of this doesn't even have to be on a full cluster. I'm thinking
something as simple as:

* configure devstack to checkout stable/essex from all of the projects
* run the system, launch some instances and volumes
* terminate the workers
* upgrade all of the code to folsom
* do any manual upgrade steps (nova-manage db sync, cinder migrate, etc.)
* run all the workers
* make sure the existing data still works and new api commands run

The manual upgrade steps should be contained in a single script so that the
distress can use it to help make their package upgrades and deployers can
use it for reference when upgrading their clusters.

This is something we can start working on today and we can run after each
commit. Then we will immediately know if we do something that breaks
upgradability, and we will have a testable documented process for upgrading.

Vish



_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


Gabriel.Hurley at nebula

Jul 12, 2012, 5:20 PM

Post #2 of 4 (157 views)
Permalink
Re: Release Upgrades (was [nova] [cinder] Nova-volume vs. Cinder in Folsom) [In reply to]

Fantastic idea! Automating that N+1 test and forcing the buildout of the migration script would be HUGE!

- Gabriel

> -----Original Message-----
> From: openstack-bounces+gabriel.hurley=nebula.com [at] lists
> [mailto:openstack-
> bounces+gabriel.hurley=nebula.com [at] lists] On Behalf Of
> Vishvananda Ishaya
> Sent: Thursday, July 12, 2012 2:36 PM
> To: Narayan Desai
> Cc: Openstack (openstack [at] lists)
> (openstack [at] lists)
> Subject: [Openstack] Release Upgrades (was [nova] [cinder] Nova-volume
> vs. Cinder in Folsom)
>
>
> On Jul 12, 2012, at 2:22 PM, Narayan Desai wrote:
>
> > I think that the long term maintenance or removal of nova-volume in
> > its existing form is orthogonal to the actual issue of continuity from
> > one release to the next.
>
> Agreed. Discussion the volume/cinder strategy is a separate topic. I've taken
> the liberty of updating the subject to keep the discussions on point
>
> >
> > At this point, we've now run cactus, diablo and are in testing with
> > essex. Each of these has effectively been a flag day for us; we build
> > the new system, migrate users, images, etc, and let users do a bunch
> > of manual migration of volume data, etc, while running both systems in
> > parallel. This hasn't been as painful as it sounds because our
> > understanding of best practices for running openstack is moving pretty
> > quickly and each system has been quite different from the previous.
>
> Upgrading has been painful and we are striving to improve this process as
> much as possible.
>
> >
> > The lack of an effective process to move from one major release to the
> > next is the major issue here in my mind. It would be fantastic if
> > (some day, ha ha ha) you could apt-get upgrade from folsom to grizzly,
> > but i think that is likely to be more trouble than it is worth. A
> > reasonable compromise would be a well documented process as well as
> > tools to aid in the process. Each real deployment will have a
> > substantial set of local customizations, particularly if they are
> > running at any sort of scale. While it won't be feasible to support
> > any upgrade with these customizations, tools for the process (which
> > may only be used a straw man in complex cases) would go a long way.
>
> I would like to take this a bit further. Documentation is a great first step, but I
> would actually like to have an actual Jenkins test that does the upgrade from
> essex to Folsom with live resources created. I think this the only way that we
> can be sure that the upgrade is working properly.
>
> The first version of this doesn't even have to be on a full cluster. I'm thinking
> something as simple as:
>
> * configure devstack to checkout stable/essex from all of the projects
> * run the system, launch some instances and volumes
> * terminate the workers
> * upgrade all of the code to folsom
> * do any manual upgrade steps (nova-manage db sync, cinder migrate, etc.)
> * run all the workers
> * make sure the existing data still works and new api commands run
>
> The manual upgrade steps should be contained in a single script so that the
> distress can use it to help make their package upgrades and deployers can
> use it for reference when upgrading their clusters.
>
> This is something we can start working on today and we can run after each
> commit. Then we will immediately know if we do something that breaks
> upgradability, and we will have a testable documented process for
> upgrading.
>
> Vish
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp



_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


mordred at inaugust

Jul 12, 2012, 5:22 PM

Post #3 of 4 (156 views)
Permalink
Re: Release Upgrades (was [nova] [cinder] Nova-volume vs. Cinder in Folsom) [In reply to]

On 07/12/2012 04:36 PM, Vishvananda Ishaya wrote:
>
> On Jul 12, 2012, at 2:22 PM, Narayan Desai wrote:
>
>> I think that the long term maintenance or removal of nova-volume in
>> its existing form is orthogonal to the actual issue of continuity from
>> one release to the next.
>
> Agreed. Discussion the volume/cinder strategy is a separate topic. I've
> taken the liberty of updating the subject to keep the discussions on point
>
>>
>> At this point, we've now run cactus, diablo and are in testing with
>> essex. Each of these has effectively been a flag day for us; we build
>> the new system, migrate users, images, etc, and let users do a bunch
>> of manual migration of volume data, etc, while running both systems in
>> parallel. This hasn't been as painful as it sounds because our
>> understanding of best practices for running openstack is moving pretty
>> quickly and each system has been quite different from the previous.
>
> Upgrading has been painful and we are striving to improve this process
> as much as possible.
>
>>
>> The lack of an effective process to move from one major release to the
>> next is the major issue here in my mind. It would be fantastic if
>> (some day, ha ha ha) you could apt-get upgrade from folsom to grizzly,
>> but i think that is likely to be more trouble than it is worth. A
>> reasonable compromise would be a well documented process as well as
>> tools to aid in the process. Each real deployment will have a
>> substantial set of local customizations, particularly if they are
>> running at any sort of scale. While it won't be feasible to support
>> any upgrade with these customizations, tools for the process (which
>> may only be used a straw man in complex cases) would go a long way.
>
> I would like to take this a bit further. Documentation is a great first step,
> but I would actually like to have an actual Jenkins test that does the upgrade
> from essex to Folsom with live resources created. I think this the only way
> that we can be sure that the upgrade is working properly.

++

> The first version of this doesn't even have to be on a full cluster. I'm thinking
> something as simple as:
>
> * configure devstack to checkout stable/essex from all of the projects
> * run the system, launch some instances and volumes
> * terminate the workers
> * upgrade all of the code to folsom
> * do any manual upgrade steps (nova-manage db sync, cinder migrate, etc.)
> * run all the workers
> * make sure the existing data still works and new api commands run
>
> The manual upgrade steps should be contained in a single script so that the
> distress can use it to help make their package upgrades and deployers can
> use it for reference when upgrading their clusters.

Yes - especially if it's a self contained thing like devstack is currently.

For the "upgrade all the code to folsom" step - let's chat about making
sure that we get the right hooks/env vars in there so that we can make
that "upgrade to tip of trunk in most projects + proposed patch in one
of them" - same as we do for devstack runs today.

> This is something we can start working on today and we can run after each
> commit. Then we will immediately know if we do something that breaks
> upgradability, and we will have a testable documented process for upgrading.

The creation of the self-contained script devstack was a HUGE step
forward for us for integration testing. I think a similar thing for
upgradability would similarly be huge.


_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


narayan.desai at gmail

Jul 12, 2012, 8:42 PM

Post #4 of 4 (156 views)
Permalink
Re: Release Upgrades (was [nova] [cinder] Nova-volume vs. Cinder in Folsom) [In reply to]

On Thu, Jul 12, 2012 at 4:36 PM, Vishvananda Ishaya
<vishvananda [at] gmail> wrote:

> Upgrading has been painful and we are striving to improve this process
> as much as possible.

I think that this needs to be a core value of the developer community,
if Openstack is going to become pervasive.

> I would like to take this a bit further. Documentation is a great first step,
> but I would actually like to have an actual Jenkins test that does the upgrade
> from essex to Folsom with live resources created. I think this the only way
> that we can be sure that the upgrade is working properly.

I don't want to dampen enthusiasm around this issue at all, but I
think this goal is pretty difficult to achieve, just due to the
existing complexity in real deployments. I'm also worried that this
would take away from a high level upgrade documentation goal.

>
> The first version of this doesn't even have to be on a full cluster. I'm thinking
> something as simple as:
>
> * configure devstack to checkout stable/essex from all of the projects
> * run the system, launch some instances and volumes
> * terminate the workers
> * upgrade all of the code to folsom
> * do any manual upgrade steps (nova-manage db sync, cinder migrate, etc.)
> * run all the workers
> * make sure the existing data still works and new api commands run
>
> The manual upgrade steps should be contained in a single script so that the
> distress can use it to help make their package upgrades and deployers can
> use it for reference when upgrading their clusters.
>
> This is something we can start working on today and we can run after each
> commit. Then we will immediately know if we do something that breaks
> upgradability, and we will have a testable documented process for upgrading.

Having a testable process for upgrading is definitely a start. I guess
that I am more or less resigned to upgrades being pretty effort
intensive, at least in the near term, so my personal bias is towards
getting pretty extreme documentation done first.
-nld

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

OpenStack dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.