Mailing List Archive: OpenStack: Dev

database migration cleanup

Index | Next | Previous | View Threaded

dprince at redhat

Apr 26, 2012, 12:24 PM

Post #1 of 20 (361 views)
Permalink

database migration cleanup

The OpenStack Essex release had 82 database migrations. As these grow in number it seems reasonable to clean house from time to time. Now seems as good a time as any.

I came up with a first go at it here:

https://review.openstack.org/#/c/6847/

The idea is that we would:

* Do this early in the release cycle to minimize risk.

* Compact all pre-Folsom migrations into a single migration. This migration would be used for new installations.

* New migrations during the Folsom release cycle would proceed as normal.

* Migrations added during Folsom release cycle could be compacted during "E" release cycle. TBD if/when we do the next compaction.

* Users upgrading from pre-Essex would need to upgrade to Essex first. Then Folsom.

--

I think this scheme would support users who follow stable releases as well as users who follow trunk very closely.

We talked about this at the conference but I thought this issue might be near and dear to some of our end users so it was worth discussing on the list.

What are general thoughts on this approach?

Dan (dprince)

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

vishvananda at gmail

Apr 26, 2012, 1:14 PM

Post #2 of 20 (336 views)
Permalink

Re: database migration cleanup [In reply to]

+1. Might be nice to have some kind of test to verify that the new migration leaves the tables in exactly the same state as the old migrations.

Vish

On Apr 26, 2012, at 12:24 PM, Dan Prince wrote:

> The OpenStack Essex release had 82 database migrations. As these grow in number it seems reasonable to clean house from time to time. Now seems as good a time as any.
>
> I came up with a first go at it here:
>
> https://review.openstack.org/#/c/6847/
>
> The idea is that we would:
>
> * Do this early in the release cycle to minimize risk.
>
> * Compact all pre-Folsom migrations into a single migration. This migration would be used for new installations.
>
> * New migrations during the Folsom release cycle would proceed as normal.
>
> * Migrations added during Folsom release cycle could be compacted during "E" release cycle. TBD if/when we do the next compaction.
>
> * Users upgrading from pre-Essex would need to upgrade to Essex first. Then Folsom.
>
> --
>
> I think this scheme would support users who follow stable releases as well as users who follow trunk very closely.
>
> We talked about this at the conference but I thought this issue might be near and dear to some of our end users so it was worth discussing on the list.
>
> What are general thoughts on this approach?
>
> Dan (dprince)
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

thierry at openstack

Apr 27, 2012, 1:03 AM

Post #3 of 20 (327 views)
Permalink

Re: database migration cleanup [In reply to]

Vishvananda Ishaya wrote:
> +1. Might be nice to have some kind of test to verify that the new migration leaves the tables in exactly the same state as the old migrations.

Sounds like a good way to avoid nasty regressions. And maybe run that
test (at least once) over the range of supported databases (SQLite,
MySQL, PostgreSQL) ?

--
Thierry Carrez (ttx)
Release Manager, OpenStack

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

eglynn at redhat

Apr 27, 2012, 2:45 AM

Post #4 of 20 (324 views)
Permalink

Re: database migration cleanup [In reply to]

> https://review.openstack.org/#/c/6847/

Nice!

> * Migrations added during Folsom release cycle could be compacted
> during "E" release cycle. TBD if/when we do the next compaction.

An alternative idea would be to do the compaction *prior* to the
Folsom relase instead of after, so that the cleanest possible
migration path is presented to non-trunk-chasing users. It could for
example be a task that's part of spinning up the first Folsom-RC.

Its unlikely that after new migrations are added after the release
candidate goes out the door (as these are generally associated with
non-trivial new features, which would have missed the boat at that
late stage). But if there are any, these would have to be adde to
the squashed aggregate migration from the get-go.

Cheers,
Eoghan

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

dprince at redhat

Apr 27, 2012, 7:02 AM

Post #5 of 20 (326 views)
Permalink

Re: database migration cleanup [In reply to]

----- Original Message -----
> From: "Eoghan Glynn" <eglynn [at] redhat>
> To: "Dan Prince" <dprince [at] redhat>
> Cc: openstack [at] lists
> Sent: Friday, April 27, 2012 5:45:27 AM
> Subject: Re: [Openstack] database migration cleanup
>
>
>
> > https://review.openstack.org/#/c/6847/
>
> Nice!
>
> > * Migrations added during Folsom release cycle could be compacted
> > during "E" release cycle. TBD if/when we do the next compaction.
>
> An alternative idea would be to do the compaction *prior* to the
> Folsom relase instead of after, so that the cleanest possible
> migration path is presented to non-trunk-chasing users. It could for
> example be a task that's part of spinning up the first Folsom-RC.
>
> Its unlikely that after new migrations are added after the release
> candidate goes out the door (as these are generally associated with
> non-trivial new features, which would have missed the boat at that
> late stage). But if there are any, these would have to be adde to
> the squashed aggregate migration from the get-go.

I thought about this... but that still leaves only a couple weeks to catch any issues that might come up in the release candidate phase. Also, using the RC makes the compaction point a bit more fuzzy for end users who are following trunk more closely. I do like that it would keep the release tree cleaner however.

Performing the compaction after release is sort of a middle ground approach which should allow us to clean house from time to time but also keep things stable around release time.

>
> Cheers,
> Eoghan
>
>

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

johannes at erdfelt

Apr 27, 2012, 7:20 AM

Post #6 of 20 (330 views)
Permalink

Re: database migration cleanup [In reply to]

On Thu, Apr 26, 2012, Dan Prince <dprince [at] redhat> wrote:
> The OpenStack Essex release had 82 database migrations. As these grow
> in number it seems reasonable to clean house from time to time. Now
> seems as good a time as any.

Mirations don't appear to be particularly slow right now, and it doesn't
appear that merging migrations will make them significantly faster.

What exactly is the benefit of doing this?

By merging migrations, we lose history in git for why the migrations
were added in the first place.

JE

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

sdague at linux

Apr 27, 2012, 7:21 AM

Post #7 of 20 (330 views)
Permalink

Re: database migration cleanup [In reply to]

On 04/26/2012 03:24 PM, Dan Prince wrote:
<snip>
> I think this scheme would support users who follow stable releases as well as users who follow trunk very closely.
>
> We talked about this at the conference but I thought this issue might be near and dear to some of our end users so it was worth discussing on the list.
>
> What are general thoughts on this approach?

Is there any support in sqlalchemy, or related tools, to handle
migrations the way rails does, where a schema file is created at the end
of every migration? It would be ideal if we both had a full migration
history, as well as a short cut at any snap shot to get to the end.

-Sean

--
Sean Dague
IBM Linux Technology Center
email: sldague [at] us
alt-email: sdague [at] linux

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

dprince at redhat

Apr 27, 2012, 8:47 AM

Post #8 of 20 (320 views)
Permalink

Re: database migration cleanup [In reply to]

----- Original Message -----
> From: "Sean Dague" <sdague [at] linux>
> To: openstack [at] lists
> Sent: Friday, April 27, 2012 10:21:17 AM
> Subject: Re: [Openstack] database migration cleanup
>
> On 04/26/2012 03:24 PM, Dan Prince wrote:
> <snip>
> > I think this scheme would support users who follow stable releases
> > as well as users who follow trunk very closely.
> >
> > We talked about this at the conference but I thought this issue
> > might be near and dear to some of our end users so it was worth
> > discussing on the list.
> >
> > What are general thoughts on this approach?
>
> Is there any support in sqlalchemy, or related tools, to handle
> migrations the way rails does, where a schema file is created at the
> end
> of every migration? It would be ideal if we both had a full migration
> history, as well as a short cut at any snap shot to get to the end.

Ah. Yes, the Rails schema.rb. I looked around for just this sort of thing and didn't find much. Python-migrate has some "experimental" support for generating models and I did make use of that initially. See 'create_model' below:

[root [at] nova migrate_repo]# python ./manage.py --repository=./ --url=mysql://nova:password [at] localhos/nova
Usage: manage.py COMMAND ...

Available commands:
compare_model_to_db - compare MetaData against the current database state
create - create an empty repository at the specified path
create_model - dump the current database as a Python model to stdout
db_version - show the current version of the repository under version control
downgrade - downgrade a database to an earlier version
drop_version_control - removes version control from a database
help - displays help on a given command
make_update_script_for_model - create a script changing the old MetaData to the new (current) MetaData
manage - creates a Python script that runs Migrate with a set of default values
script - create an empty change Python script
script_sql - create empty change SQL scripts for given database
source - display the Python code for a particular version in this repository
test - performs the upgrade and downgrade command on the given database
update_db_from_model - modify the database to match the structure of the current MetaData
upgrade - upgrade a database to a later version
version - display the latest version available in a repository
version_control - mark a database as under this repository's version control

----

python-migrate's 'create_model' does not however give you something that exactly matches the schema you'd get by running the all the migrations. So auto generation doesn't appear to be an option right now. It would be nice to contribute python-migrate in this regard and get better support for model generation, etc. Maybe a good long term goal?

Dan

>
> -Sean
>
> --
> Sean Dague
> IBM Linux Technology Center
> email: sldague [at] us
> alt-email: sdague [at] linux
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

dprince at redhat

Apr 27, 2012, 8:51 AM

Post #9 of 20 (318 views)
Permalink

Re: database migration cleanup [In reply to]

----- Original Message -----
> From: "Johannes Erdfelt" <johannes [at] erdfelt>
> To: openstack [at] lists
> Sent: Friday, April 27, 2012 10:20:38 AM
> Subject: Re: [Openstack] database migration cleanup
>
> On Thu, Apr 26, 2012, Dan Prince <dprince [at] redhat> wrote:
> > The OpenStack Essex release had 82 database migrations. As these
> > grow
> > in number it seems reasonable to clean house from time to time. Now
> > seems as good a time as any.
>
> Mirations don't appear to be particularly slow right now, and it
> doesn't
> appear that merging migrations will make them significantly faster.
>
> What exactly is the benefit of doing this?

Speed wasn't the primary motivation here I suppose. Do we really want to continue to maintain 100+ migrations in our codebase over the lifetime of the project? As we add more and more to our pep8/hacking tools this could become an annoying burden right? I mean why require end users to run migrations that add and drop the same table repeatedly?

>
> By merging migrations, we lose history in git for why the migrations
> were added in the first place.

I'm not sure I understand? Git still has the source code for the old migrations. All you'd need to do is checkout an old version of master or look at the stable/essex, stable/diablo branches right?

>
> JE
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

johannes at erdfelt

Apr 27, 2012, 9:13 AM

Post #10 of 20 (318 views)
Permalink

Re: database migration cleanup [In reply to]

On Fri, Apr 27, 2012, Dan Prince <dprince [at] redhat> wrote:
> > Mirations don't appear to be particularly slow right now, and it
> > doesn't
> > appear that merging migrations will make them significantly faster.
> >
> > What exactly is the benefit of doing this?
>
> Speed wasn't the primary motivation here I suppose. Do we really want
> to continue to maintain 100+ migrations in our codebase over the
> lifetime of the project? As we add more and more to our pep8/hacking
> tools this could become an annoying burden right?

Has that been a problem?

I'm not sure I see where pep8/hacking changes would require changes to
the unmerged migrations but the merged migrations wouldn't require them.
Wouldn't that affect either?

Looking at the logs for the migrations most of the changes (outside of
PEP8 changes) appear to be sqlalchemy related changes, which will exist
regardless if they are merged or not as well.

> I mean why require end users to run migrations that add and drop the
> same table repeatedly?

It's hard to just grep for which migrations drop a table/column added in
a previous migration. Did you see how many are like that when you merged
all of the migrations together?

> I'm not sure I understand? Git still has the source code for the old
> migrations. All you'd need to do is checkout an old version of master
> or look at the stable/essex, stable/diablo branches right?

True, you would just need to go out of your way to look at the history
instead.

I guess it's a matter of pros vs cons. Right now, I'd prefer to not
merge them simply because I haven't seen what the benefit is.

JE

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

mdragon at RACKSPACE

Apr 27, 2012, 10:46 AM

Post #11 of 20 (316 views)
Permalink

Re: database migration cleanup [In reply to]

Even better, what would it take to try using Alembic? (http://alembic.readthedocs.org/en/latest/front.html#project-homepage)

It's a big improvement over sqlalchemy. Amongst other things, migrations are not numbered, they are linked by dependancy, and run in topological-sort order. That there eliminates alot of "my migration number got taken... again..." problems.
On Apr 27, 2012, at 10:47 AM, Dan Prince wrote:

>
>
> ----- Original Message -----
>> From: "Sean Dague" <sdague [at] linux>
>> To: openstack [at] lists
>> Sent: Friday, April 27, 2012 10:21:17 AM
>> Subject: Re: [Openstack] database migration cleanup
>>
>> On 04/26/2012 03:24 PM, Dan Prince wrote:
>> <snip>
>>> I think this scheme would support users who follow stable releases
>>> as well as users who follow trunk very closely.
>>>
>>> We talked about this at the conference but I thought this issue
>>> might be near and dear to some of our end users so it was worth
>>> discussing on the list.
>>>
>>> What are general thoughts on this approach?
>>
>> Is there any support in sqlalchemy, or related tools, to handle
>> migrations the way rails does, where a schema file is created at the
>> end
>> of every migration? It would be ideal if we both had a full migration
>> history, as well as a short cut at any snap shot to get to the end.
>
> Ah. Yes, the Rails schema.rb. I looked around for just this sort of thing and didn't find much. Python-migrate has some "experimental" support for generating models and I did make use of that initially. See 'create_model' below:
>
>
> [root [at] nova migrate_repo]# python ./manage.py --repository=./ --url=mysql://nova:password [at] localhos/nova
> Usage: manage.py COMMAND ...
>
> Available commands:
> compare_model_to_db - compare MetaData against the current database state
> create - create an empty repository at the specified path
> create_model - dump the current database as a Python model to stdout
> db_version - show the current version of the repository under version control
> downgrade - downgrade a database to an earlier version
> drop_version_control - removes version control from a database
> help - displays help on a given command
> make_update_script_for_model - create a script changing the old MetaData to the new (current) MetaData
> manage - creates a Python script that runs Migrate with a set of default values
> script - create an empty change Python script
> script_sql - create empty change SQL scripts for given database
> source - display the Python code for a particular version in this repository
> test - performs the upgrade and downgrade command on the given database
> update_db_from_model - modify the database to match the structure of the current MetaData
> upgrade - upgrade a database to a later version
> version - display the latest version available in a repository
> version_control - mark a database as under this repository's version control
>
> ----
>
> python-migrate's 'create_model' does not however give you something that exactly matches the schema you'd get by running the all the migrations. So auto generation doesn't appear to be an option right now. It would be nice to contribute python-migrate in this regard and get better support for model generation, etc. Maybe a good long term goal?
>
> Dan
>
>
>>
>> -Sean
>>
>> --
>> Sean Dague
>> IBM Linux Technology Center
>> email: sldague [at] us
>> alt-email: sdague [at] linux
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack [at] lists
>> Unsubscribe : https://launchpad.net/~openstack
>> More help : https://help.launchpad.net/ListHelp
>>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp

--
Monsyne M. Dragon
OpenStack/Nova
cell 210-441-0965
work x 5014190

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

johannes at erdfelt

Apr 27, 2012, 12:57 PM

Post #12 of 20 (327 views)
Permalink

Re: database migration cleanup [In reply to]

On Fri, Apr 27, 2012, Monsyne Dragon <mdragon [at] RACKSPACE> wrote:
> Even better, what would it take to try using Alembic?
> (http://alembic.readthedocs.org/en/latest/front.html#project-homepage)
>
> It's a big improvement over sqlalchemy. Amongst other things,
> migrations are not numbered, they are linked by dependancy, and run in
> topological-sort order. That there eliminates alot of "my migration
> number got taken... again..." problems.

I've mentioned in on the list before, but here's a partial switch over
to alembic:

https://github.com/jerdfelt/nova/tree/alembic

However, it's orthogonal to merging migrations. It wouldn't be any
better or worse in that regard.

JE

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

dprince at redhat

Apr 30, 2012, 8:58 AM

Post #13 of 20 (310 views)
Permalink

Re: database migration cleanup [In reply to]

----- Original Message -----
> From: "Monsyne Dragon" <mdragon [at] RACKSPACE>
> To: "Dan Prince" <dprince [at] redhat>
> Cc: "Sean Dague" <sdague [at] linux>, "<openstack [at] lists>" <openstack [at] lists>
> Sent: Friday, April 27, 2012 1:46:03 PM
> Subject: Re: [Openstack] database migration cleanup
>
> Even better, what would it take to try using Alembic?
> (http://alembic.readthedocs.org/en/latest/front.html#project-homepage)

>From the Alembic docs site: "Note that Alembic is still in alpha status."

I would guess we'd want the project to be in at least a beta (stable API) state before we committed to using it, etc.

>
> It's a big improvement over sqlalchemy. Amongst other things,
> migrations are not numbered, they are linked by dependancy, and run
> in topological-sort order. That there eliminates alot of "my
> migration number got taken... again..." problems.

To be clear Alembic would be a potential replacement for python-migrate. Not Sqlalchemy right? Looks like an interesting project but I'm not convinced it is worth switching over to as our default at this time.

> On Apr 27, 2012, at 10:47 AM, Dan Prince wrote:
>
> >
> >
> > ----- Original Message -----
> >> From: "Sean Dague" <sdague [at] linux>
> >> To: openstack [at] lists
> >> Sent: Friday, April 27, 2012 10:21:17 AM
> >> Subject: Re: [Openstack] database migration cleanup
> >>
> >> On 04/26/2012 03:24 PM, Dan Prince wrote:
> >> <snip>
> >>> I think this scheme would support users who follow stable
> >>> releases
> >>> as well as users who follow trunk very closely.
> >>>
> >>> We talked about this at the conference but I thought this issue
> >>> might be near and dear to some of our end users so it was worth
> >>> discussing on the list.
> >>>
> >>> What are general thoughts on this approach?
> >>
> >> Is there any support in sqlalchemy, or related tools, to handle
> >> migrations the way rails does, where a schema file is created at
> >> the
> >> end
> >> of every migration? It would be ideal if we both had a full
> >> migration
> >> history, as well as a short cut at any snap shot to get to the
> >> end.
> >
> > Ah. Yes, the Rails schema.rb. I looked around for just this sort of
> > thing and didn't find much. Python-migrate has some "experimental"
> > support for generating models and I did make use of that
> > initially. See 'create_model' below:
> >
> >
> > [root [at] nova migrate_repo]# python ./manage.py --repository=./
> > --url=mysql://nova:password [at] localhos/nova
> > Usage: manage.py COMMAND ...
> >
> > Available commands:
> > compare_model_to_db - compare MetaData against the
> > current database state
> > create - create an empty repository at the
> > specified path
> > create_model - dump the current database as a
> > Python model to stdout
> > db_version - show the current version of the
> > repository under version control
> > downgrade - downgrade a database to an earlier
> > version
> > drop_version_control - removes version control from a
> > database
> > help - displays help on a given command
> > make_update_script_for_model - create a script changing the old
> > MetaData to the new (current) MetaData
> > manage - creates a Python script that runs
> > Migrate with a set of default values
> > script - create an empty change Python
> > script
> > script_sql - create empty change SQL scripts for
> > given database
> > source - display the Python code for a
> > particular version in this repository
> > test - performs the upgrade and downgrade
> > command on the given database
> > update_db_from_model - modify the database to match the
> > structure of the current MetaData
> > upgrade - upgrade a database to a later
> > version
> > version - display the latest version
> > available in a repository
> > version_control - mark a database as under this
> > repository's version control
> >
> > ----
> >
> > python-migrate's 'create_model' does not however give you something
> > that exactly matches the schema you'd get by running the all the
> > migrations. So auto generation doesn't appear to be an option
> > right now. It would be nice to contribute python-migrate in this
> > regard and get better support for model generation, etc. Maybe a
> > good long term goal?
> >
> > Dan
> >
> >
> >>
> >> -Sean
> >>
> >> --
> >> Sean Dague
> >> IBM Linux Technology Center
> >> email: sldague [at] us
> >> alt-email: sdague [at] linux
> >>
> >>
> >> _______________________________________________
> >> Mailing list: https://launchpad.net/~openstack
> >> Post to : openstack [at] lists
> >> Unsubscribe : https://launchpad.net/~openstack
> >> More help : https://help.launchpad.net/ListHelp
> >>
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to : openstack [at] lists
> > Unsubscribe : https://launchpad.net/~openstack
> > More help : https://help.launchpad.net/ListHelp
>
> --
> Monsyne M. Dragon
> OpenStack/Nova
> cell 210-441-0965
> work x 5014190
>
>

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

dprince at redhat

Apr 30, 2012, 9:13 AM

Post #14 of 20 (306 views)
Permalink

Re: database migration cleanup [In reply to]

----- Original Message -----
> From: "Johannes Erdfelt" <johannes [at] erdfelt>
> To: openstack [at] lists
> Sent: Friday, April 27, 2012 12:13:47 PM
> Subject: Re: [Openstack] database migration cleanup
>
> On Fri, Apr 27, 2012, Dan Prince <dprince [at] redhat> wrote:
> > > Mirations don't appear to be particularly slow right now, and it
> > > doesn't
> > > appear that merging migrations will make them significantly
> > > faster.
> > >
> > > What exactly is the benefit of doing this?
> >
> > Speed wasn't the primary motivation here I suppose. Do we really
> > want
> > to continue to maintain 100+ migrations in our codebase over the
> > lifetime of the project? As we add more and more to our
> > pep8/hacking
> > tools this could become an annoying burden right?
>
> Has that been a problem?
>
> I'm not sure I see where pep8/hacking changes would require changes
> to
> the unmerged migrations but the merged migrations wouldn't require
> them.
> Wouldn't that affect either?

The primary benefit here is it is simply less code to maintain:

The old migrations scripts for Essex are around 6200 lines of code.

The new compacted migration for Essex is around 950 lines of code.

>
> Looking at the logs for the migrations most of the changes (outside
> of
> PEP8 changes) appear to be sqlalchemy related changes, which will
> exist
> regardless if they are merged or not as well.
>
> > I mean why require end users to run migrations that add and drop
> > the
> > same table repeatedly?
>
> It's hard to just grep for which migrations drop a table/column added
> in
> a previous migration. Did you see how many are like that when you
> merged
> all of the migrations together?
>
> > I'm not sure I understand? Git still has the source code for the
> > old
> > migrations. All you'd need to do is checkout an old version of
> > master
> > or look at the stable/essex, stable/diablo branches right?
>
> True, you would just need to go out of your way to look at the
> history
> instead.
>
> I guess it's a matter of pros vs cons. Right now, I'd prefer to not
> merge them simply because I haven't seen what the benefit is.

The primary benefit here is just cleaning up our code base. If that isn't a good reason or if you'd rather keep the 82 Essex migrations in the code base for the long term please feel free to comment accordingly on the merge proposal.

Dan

>
> JE
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

johannes at erdfelt

Apr 30, 2012, 9:31 AM

Post #15 of 20 (308 views)
Permalink

Re: database migration cleanup [In reply to]

On Mon, Apr 30, 2012, Dan Prince <dprince [at] redhat> wrote:
> The primary benefit here is it is simply less code to maintain:
>
> The old migrations scripts for Essex are around 6200 lines of code.
>
> The new compacted migration for Essex is around 950 lines of code.

It seems like you're counting raw lines, which isn't a very useful
number.

I don't think anyone will worry about maintaining the copyright header
in each file for instance.

> The primary benefit here is just cleaning up our code base. If that
> isn't a good reason or if you'd rather keep the 82 Essex migrations
> in the code base for the long term please feel free to comment
> accordingly on the merge proposal.

Ultimately, experience with sqlalchemy-migrate leaves me scared that
making non-obvious changes will result in even more non-obvious
behavior.

For instance, your change deletes all of the mysql and sqlite specific
.sql files with no replacement. Do we know for sure that it will result
in identical migrations?

I don't think I'd have a problem with this sort of migration merge if it
was using some other code that I could trust more or we had more
extensive tests to ensure this change was safe.

JE

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

thierry at openstack

May 2, 2012, 3:08 AM

Post #16 of 20 (287 views)
Permalink

Re: database migration cleanup [In reply to]

Dan Prince wrote:
>>> * Migrations added during Folsom release cycle could be compacted
>>> during "E" release cycle. TBD if/when we do the next compaction.
>>
>> An alternative idea would be to do the compaction *prior* to the
>> Folsom relase instead of after, so that the cleanest possible
>> migration path is presented to non-trunk-chasing users. It could for
>> example be a task that's part of spinning up the first Folsom-RC.
>>
>> Its unlikely that after new migrations are added after the release
>> candidate goes out the door (as these are generally associated with
>> non-trivial new features, which would have missed the boat at that
>> late stage). But if there are any, these would have to be adde to
>> the squashed aggregate migration from the get-go.
>
> I thought about this... but that still leaves only a couple weeks to catch any issues that might come up in the release candidate phase. Also, using the RC makes the compaction point a bit more fuzzy for end users who are following trunk more closely. I do like that it would keep the release tree cleaner however.
>
> Performing the compaction after release is sort of a middle ground approach which should allow us to clean house from time to time but also keep things stable around release time.

Yes, I've seen a few hard-to-spot regressions caused by migrations in
the past, mostly corner cases that affect specific databases, so
compacting migrations as the very last step before RC1 generation sounds
a bit dangerous to me. It clearly falls in the "clean-up to make code
more tidy but can introduce gratuitous regressions without fixing any
bug" category, which I prefer to happen early in the cycle rather than late.

--
Thierry Carrez (ttx)
Release Manager, OpenStack

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

dprince at redhat

May 2, 2012, 1:20 PM

Post #17 of 20 (287 views)
Permalink

Re: database migration cleanup [In reply to]

----- Original Message -----
> From: "Vishvananda Ishaya" <vishvananda [at] gmail>
> To: "Dan Prince" <dprince [at] redhat>
> Cc: openstack [at] lists
> Sent: Thursday, April 26, 2012 4:14:25 PM
> Subject: Re: [Openstack] database migration cleanup
>
> +1. Might be nice to have some kind of test to verify that the new
> migration leaves the tables in exactly the same state as the old
> migrations.

Hey Vish,

This is an outline of what I did to test MySQL and PostgreSQL to ensure the compact migration script generates *exactly* the same schemas as before:

http://wiki.openstack.org/database_migration_testing

As things stand both MySQL and PostgreSQL are exactly the same. I have some pending changes that I've found in the schemas that need to be fixed in Folsom... but the goal here was to replicate Essex with migration 082 so that is what I did.

Sqlite has a few differences (indexes for example). How important is it that the Sqlite schema be exactly the same? Unit tests are passing.

Dan

>
> Vish
>
> On Apr 26, 2012, at 12:24 PM, Dan Prince wrote:
>
> > The OpenStack Essex release had 82 database migrations. As these
> > grow in number it seems reasonable to clean house from time to
> > time. Now seems as good a time as any.
> >
> > I came up with a first go at it here:
> >
> > https://review.openstack.org/#/c/6847/
> >
> > The idea is that we would:
> >
> > * Do this early in the release cycle to minimize risk.
> >
> > * Compact all pre-Folsom migrations into a single migration. This
> > migration would be used for new installations.
> >
> > * New migrations during the Folsom release cycle would proceed as
> > normal.
> >
> > * Migrations added during Folsom release cycle could be compacted
> > during "E" release cycle. TBD if/when we do the next compaction.
> >
> > * Users upgrading from pre-Essex would need to upgrade to Essex
> > first. Then Folsom.
> >
> > --
> >
> > I think this scheme would support users who follow stable releases
> > as well as users who follow trunk very closely.
> >
> > We talked about this at the conference but I thought this issue
> > might be near and dear to some of our end users so it was worth
> > discussing on the list.
> >
> > What are general thoughts on this approach?
> >
> > Dan (dprince)
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to : openstack [at] lists
> > Unsubscribe : https://launchpad.net/~openstack
> > More help : https://help.launchpad.net/ListHelp
>
>

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

John.Garbutt at citrix

May 3, 2012, 7:56 AM

Post #18 of 20 (280 views)
Permalink

Re: database migration cleanup [In reply to]

I may have missed this in the discussions, but does this impact on upgrade?

I am guessing you have tested Essex -> Folsom upgrade, but does this affect people upgrading from any of the Essex milestones to Folsom? I guess the deeper question is which upgrade paths do we want to maintain...

Thanks,
John

> -----Original Message-----
> From: openstack-bounces+john.garbutt=eu.citrix.com [at] lists
> [mailto:openstack-bounces+john.garbutt=eu.citrix.com [at] lists]
> On Behalf Of Dan Prince
> Sent: 02 May 2012 21:20
> To: Vishvananda Ishaya
> Cc: openstack [at] lists
> Subject: Re: [Openstack] database migration cleanup
>
>
>
> ----- Original Message -----
> > From: "Vishvananda Ishaya" <vishvananda [at] gmail>
> > To: "Dan Prince" <dprince [at] redhat>
> > Cc: openstack [at] lists
> > Sent: Thursday, April 26, 2012 4:14:25 PM
> > Subject: Re: [Openstack] database migration cleanup
> >
> > +1. Might be nice to have some kind of test to verify that the new
> > migration leaves the tables in exactly the same state as the old
> > migrations.
>
> Hey Vish,
>
> This is an outline of what I did to test MySQL and PostgreSQL to ensure the
> compact migration script generates *exactly* the same schemas as before:
>
> http://wiki.openstack.org/database_migration_testing
>
> As things stand both MySQL and PostgreSQL are exactly the same. I have
> some pending changes that I've found in the schemas that need to be fixed
> in Folsom... but the goal here was to replicate Essex with migration 082 so
> that is what I did.
>
> Sqlite has a few differences (indexes for example). How important is it that
> the Sqlite schema be exactly the same? Unit tests are passing.
>
> Dan
>
>
> >
> > Vish
> >
> > On Apr 26, 2012, at 12:24 PM, Dan Prince wrote:
> >
> > > The OpenStack Essex release had 82 database migrations. As these
> > > grow in number it seems reasonable to clean house from time to time.
> > > Now seems as good a time as any.
> > >
> > > I came up with a first go at it here:
> > >
> > > https://review.openstack.org/#/c/6847/
> > >
> > > The idea is that we would:
> > >
> > > * Do this early in the release cycle to minimize risk.
> > >
> > > * Compact all pre-Folsom migrations into a single migration. This
> > > migration would be used for new installations.
> > >
> > > * New migrations during the Folsom release cycle would proceed as
> > > normal.
> > >
> > > * Migrations added during Folsom release cycle could be compacted
> > > during "E" release cycle. TBD if/when we do the next compaction.
> > >
> > > * Users upgrading from pre-Essex would need to upgrade to Essex
> > > first. Then Folsom.
> > >
> > > --
> > >
> > > I think this scheme would support users who follow stable releases
> > > as well as users who follow trunk very closely.
> > >
> > > We talked about this at the conference but I thought this issue
> > > might be near and dear to some of our end users so it was worth
> > > discussing on the list.
> > >
> > > What are general thoughts on this approach?
> > >
> > > Dan (dprince)
> > >
> > > _______________________________________________
> > > Mailing list: https://launchpad.net/~openstack
> > > Post to : openstack [at] lists
> > > Unsubscribe : https://launchpad.net/~openstack
> > > More help : https://help.launchpad.net/ListHelp
> >
> >
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

dprince at redhat

May 3, 2012, 8:07 AM

Post #19 of 20 (286 views)
Permalink

Re: database migration cleanup [In reply to]

----- Original Message -----
> From: "John Garbutt" <John.Garbutt [at] citrix>
> To: "Dan Prince" <dprince [at] redhat>, "Vishvananda Ishaya" <vishvananda [at] gmail>
> Cc: openstack [at] lists
> Sent: Thursday, May 3, 2012 10:56:44 AM
> Subject: RE: [Openstack] database migration cleanup
>
> I may have missed this in the discussions, but does this impact on
> upgrade?
>
> I am guessing you have tested Essex -> Folsom upgrade, but does this
> affect people upgrading from any of the Essex milestones to Folsom?

What this does is compact the pre-Essex (final) migrations into a single migration. Users of any of the Essex milestones would need to first upgrade to the final Essex release and then upgrade to Folsom.

This seemed like a reasonable approach since most distributions release updates that contain the final releases anyway.

> I guess the deeper question is which upgrade paths do we want to
> maintain...
>
> Thanks,
> John
>
> > -----Original Message-----
> > From:
> > openstack-bounces+john.garbutt=eu.citrix.com [at] lists
> > [mailto:openstack-bounces+john.garbutt=eu.citrix.com [at] lists]
> > On Behalf Of Dan Prince
> > Sent: 02 May 2012 21:20
> > To: Vishvananda Ishaya
> > Cc: openstack [at] lists
> > Subject: Re: [Openstack] database migration cleanup
> >
> >
> >
> > ----- Original Message -----
> > > From: "Vishvananda Ishaya" <vishvananda [at] gmail>
> > > To: "Dan Prince" <dprince [at] redhat>
> > > Cc: openstack [at] lists
> > > Sent: Thursday, April 26, 2012 4:14:25 PM
> > > Subject: Re: [Openstack] database migration cleanup
> > >
> > > +1. Might be nice to have some kind of test to verify that the
> > > new
> > > migration leaves the tables in exactly the same state as the old
> > > migrations.
> >
> > Hey Vish,
> >
> > This is an outline of what I did to test MySQL and PostgreSQL to
> > ensure the
> > compact migration script generates *exactly* the same schemas as
> > before:
> >
> > http://wiki.openstack.org/database_migration_testing
> >
> > As things stand both MySQL and PostgreSQL are exactly the same. I
> > have
> > some pending changes that I've found in the schemas that need to be
> > fixed
> > in Folsom... but the goal here was to replicate Essex with
> > migration 082 so
> > that is what I did.
> >
> > Sqlite has a few differences (indexes for example). How important
> > is it that
> > the Sqlite schema be exactly the same? Unit tests are passing.
> >
> > Dan
> >
> >
> > >
> > > Vish
> > >
> > > On Apr 26, 2012, at 12:24 PM, Dan Prince wrote:
> > >
> > > > The OpenStack Essex release had 82 database migrations. As
> > > > these
> > > > grow in number it seems reasonable to clean house from time to
> > > > time.
> > > > Now seems as good a time as any.
> > > >
> > > > I came up with a first go at it here:
> > > >
> > > > https://review.openstack.org/#/c/6847/
> > > >
> > > > The idea is that we would:
> > > >
> > > > * Do this early in the release cycle to minimize risk.
> > > >
> > > > * Compact all pre-Folsom migrations into a single migration.
> > > > This
> > > > migration would be used for new installations.
> > > >
> > > > * New migrations during the Folsom release cycle would proceed
> > > > as
> > > > normal.
> > > >
> > > > * Migrations added during Folsom release cycle could be
> > > > compacted
> > > > during "E" release cycle. TBD if/when we do the next
> > > > compaction.
> > > >
> > > > * Users upgrading from pre-Essex would need to upgrade to Essex
> > > > first. Then Folsom.
> > > >
> > > > --
> > > >
> > > > I think this scheme would support users who follow stable
> > > > releases
> > > > as well as users who follow trunk very closely.
> > > >
> > > > We talked about this at the conference but I thought this issue
> > > > might be near and dear to some of our end users so it was worth
> > > > discussing on the list.
> > > >
> > > > What are general thoughts on this approach?
> > > >
> > > > Dan (dprince)
> > > >
> > > > _______________________________________________
> > > > Mailing list: https://launchpad.net/~openstack
> > > > Post to : openstack [at] lists
> > > > Unsubscribe : https://launchpad.net/~openstack
> > > > More help : https://help.launchpad.net/ListHelp
> > >
> > >
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to : openstack [at] lists
> > Unsubscribe : https://launchpad.net/~openstack
> > More help : https://help.launchpad.net/ListHelp
>

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

trey.morris at rackspace

May 3, 2012, 10:40 AM

Post #20 of 20 (281 views)
Permalink

Re: database migration cleanup [In reply to]

merge prop: https://review.openstack.org/#/c/6847/
now has both required +2s. I'll wait a day or two to approve just in case
there are any lingering objections.

-tr3buchet

On Thu, May 3, 2012 at 10:07 AM, Dan Prince <dprince [at] redhat> wrote:

>
>
> ----- Original Message -----
> > From: "John Garbutt" <John.Garbutt [at] citrix>
> > To: "Dan Prince" <dprince [at] redhat>, "Vishvananda Ishaya" <
> vishvananda [at] gmail>
> > Cc: openstack [at] lists
> > Sent: Thursday, May 3, 2012 10:56:44 AM
> > Subject: RE: [Openstack] database migration cleanup
> >
> > I may have missed this in the discussions, but does this impact on
> > upgrade?
> >
> > I am guessing you have tested Essex -> Folsom upgrade, but does this
> > affect people upgrading from any of the Essex milestones to Folsom?
>
> What this does is compact the pre-Essex (final) migrations into a single
> migration. Users of any of the Essex milestones would need to first upgrade
> to the final Essex release and then upgrade to Folsom.
>
> This seemed like a reasonable approach since most distributions release
> updates that contain the final releases anyway.
>
>
> > I guess the deeper question is which upgrade paths do we want to
> > maintain...
> >
> > Thanks,
> > John
> >
> > > -----Original Message-----
> > > From:
> > > openstack-bounces+john.garbutt=eu.citrix.com [at] lists
> > > [mailto:openstack-bounces+john.garbutt=
> eu.citrix.com [at] lists]
> > > On Behalf Of Dan Prince
> > > Sent: 02 May 2012 21:20
> > > To: Vishvananda Ishaya
> > > Cc: openstack [at] lists
> > > Subject: Re: [Openstack] database migration cleanup
> > >
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Vishvananda Ishaya" <vishvananda [at] gmail>
> > > > To: "Dan Prince" <dprince [at] redhat>
> > > > Cc: openstack [at] lists
> > > > Sent: Thursday, April 26, 2012 4:14:25 PM
> > > > Subject: Re: [Openstack] database migration cleanup
> > > >
> > > > +1. Might be nice to have some kind of test to verify that the
> > > > new
> > > > migration leaves the tables in exactly the same state as the old
> > > > migrations.
> > >
> > > Hey Vish,
> > >
> > > This is an outline of what I did to test MySQL and PostgreSQL to
> > > ensure the
> > > compact migration script generates *exactly* the same schemas as
> > > before:
> > >
> > > http://wiki.openstack.org/database_migration_testing
> > >
> > > As things stand both MySQL and PostgreSQL are exactly the same. I
> > > have
> > > some pending changes that I've found in the schemas that need to be
> > > fixed
> > > in Folsom... but the goal here was to replicate Essex with
> > > migration 082 so
> > > that is what I did.
> > >
> > > Sqlite has a few differences (indexes for example). How important
> > > is it that
> > > the Sqlite schema be exactly the same? Unit tests are passing.
> > >
> > > Dan
> > >
> > >
> > > >
> > > > Vish
> > > >
> > > > On Apr 26, 2012, at 12:24 PM, Dan Prince wrote:
> > > >
> > > > > The OpenStack Essex release had 82 database migrations. As
> > > > > these
> > > > > grow in number it seems reasonable to clean house from time to
> > > > > time.
> > > > > Now seems as good a time as any.
> > > > >
> > > > > I came up with a first go at it here:
> > > > >
> > > > > https://review.openstack.org/#/c/6847/
> > > > >
> > > > > The idea is that we would:
> > > > >
> > > > > * Do this early in the release cycle to minimize risk.
> > > > >
> > > > > * Compact all pre-Folsom migrations into a single migration.
> > > > > This
> > > > > migration would be used for new installations.
> > > > >
> > > > > * New migrations during the Folsom release cycle would proceed
> > > > > as
> > > > > normal.
> > > > >
> > > > > * Migrations added during Folsom release cycle could be
> > > > > compacted
> > > > > during "E" release cycle. TBD if/when we do the next
> > > > > compaction.
> > > > >
> > > > > * Users upgrading from pre-Essex would need to upgrade to Essex
> > > > > first. Then Folsom.
> > > > >
> > > > > --
> > > > >
> > > > > I think this scheme would support users who follow stable
> > > > > releases
> > > > > as well as users who follow trunk very closely.
> > > > >
> > > > > We talked about this at the conference but I thought this issue
> > > > > might be near and dear to some of our end users so it was worth
> > > > > discussing on the list.
> > > > >
> > > > > What are general thoughts on this approach?
> > > > >
> > > > > Dan (dprince)
> > > > >
> > > > > _______________________________________________
> > > > > Mailing list: https://launchpad.net/~openstack
> > > > > Post to : openstack [at] lists
> > > > > Unsubscribe : https://launchpad.net/~openstack
> > > > > More help : https://help.launchpad.net/ListHelp
> > > >
> > > >
> > >
> > > _______________________________________________
> > > Mailing list: https://launchpad.net/~openstack
> > > Post to : openstack [at] lists
> > > Unsubscribe : https://launchpad.net/~openstack
> > > More help : https://help.launchpad.net/ListHelp
> >
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads