Mailing List Archive: Zope: CMF

Best use of source numbers in GS upgrade steps?

Index | Next | Previous | View Threaded

m.van.rees at zestsoftware

Apr 16, 2009, 5:09 AM

Post #1 of 13 (1550 views)
Permalink

Best use of source numbers in GS upgrade steps?

Hi,

I wonder what the best way is of specifying the source number of a
GenericSetup upgrade step.

Use case
--------

Case in point: Products.Poi, an issue tracker for CMFPlone. Between
version 1.1 and 1.2 there were changes that needed an upgrade step,
which I registered like this:

<gs:upgradeStep
title="Issue descriptions"
description="Fix description of issues."
source="1.1"
destination="1.2"
handler="Products.Poi.migration.fix_descriptions"
sortkey="1"
profile="Products.Poi:default" />

Works fine: the upgrade step is shown in the Upgrades tab of
portal_setup when migrating from Poi 1.1 to 1.2.

Problem
-------

Problem is, the 1.1 version of Poi got a few minor releases. So
someone could be upgrading from Poi 1.1.2 to 1.2. And then the
upgrade steps do *not* get shown. Reason: 1.1.2 is larger than 1.1 so
portal_setup assumes the upgrade step has been run already. Asking
portal_setup to show old upgrades does still work as expected of
course.

I solved this by changing the registration of the upgrade step: it is
now defined as being an upgrade from source="1.1.9". That version
does not (yet) exist. The assumption is that this source will always
be higher than any possible future release from the 1.1 branch.

Even when restricting yourself to simple increasing integers in a
metadata.xml instead of a dotted number, the problem is the same. Say
the 1.1 branch is at profile revision 1100 now. If I then set the
profile revision of trunk to 1101 I deny the 1.1 branch any future
profile revision changes. So I would give trunk profile revision
1200, allowing the 1.1 branch room for 99 profile revision updates.

Probably not relevant here, but for completeness sake:

- Poi branch 1.1 has 1.1.x in its version.txt and does not have a
metadata.xml.

- Poi trunk has 1.2.x in both its version.txt and metadata.xml.

I know that it is best to use both files (or a version in setup.py
instead of version.txt) and that both versions need not be the same at
all. But this is the current situation.

Question
--------

So my question is: is this a sane way of doing this? Is it alright to
specify a version (or really a profile revision) as source when that
version does not yet exist? It works fine as far as I can tell.

Thanks for any further insight,

--
Maurits van Rees | http://maurits.vanrees.org/
Work | http://zestsoftware.nl/
"This is your day, don't let them take it away." [Barlow Girl]

_______________________________________________
Zope-CMF maillist - Zope-CMF [at] lists
http://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

y.2009 at wcm-solutions

Apr 16, 2009, 6:23 AM

Post #2 of 13 (1470 views)
Permalink

Re: Best use of source numbers in GS upgrade steps? [In reply to]

Hi Maurits!

Maurits van Rees wrote:
[...]
> Probably not relevant here, but for completeness sake:
>
> - Poi branch 1.1 has 1.1.x in its version.txt and does not have a
> metadata.xml.
>
> - Poi trunk has 1.2.x in both its version.txt and metadata.xml.
>
> I know that it is best to use both files (or a version in setup.py
> instead of version.txt) and that both versions need not be the same at
> all. But this is the current situation.

If the profile hasn't changed between 1.1 and 1.1.x, the profile version
should be the same. Your problem is caused by giving the same profile
different version numbers.

> Question
> --------
>
> So my question is: is this a sane way of doing this? Is it alright to
> specify a version (or really a profile revision) as source when that
> version does not yet exist? It works fine as far as I can tell.

AFAICS this will not work with GenericSetup trunk. Maybe you want to
show the upgrade step for *all* versions before 1.2? In that case you
don't specify any source version.

BTW: The relevant behavior is quite inconsistent in GenericSetup 1.4. I
added several tests and cleaned up the behavior on the trunk:
http://svn.zope.org/*checkout*/Products.GenericSetup/trunk/Products/GenericSetup/tests/upgrade.txt
Please let me know if I did break useful behavior.

Cheers,

Yuppie

_______________________________________________
Zope-CMF maillist - Zope-CMF [at] lists
http://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

ra at burningman

Apr 16, 2009, 4:29 PM

Post #3 of 13 (1475 views)
Permalink

Re: Best use of source numbers in GS upgrade steps? [In reply to]

Maurits van Rees wrote:
> Hi,
>
> I wonder what the best way is of specifying the source number of a
> GenericSetup upgrade step.
>
<---SNIP a bunch of stuff about how using versions to mark GS upgrade steps is
annoying--->

yes, as you discovered, using source and destination versions to mark when
upgrade steps should be run can be a bit janky. i realized this when i first
ported the upgrade stuff over from CPS, but i wasn't quite sure how else to
handle it.

thinking about it now, though, i'd say perhaps the zcml should support only
including a "source" version, with the setup tool persisting the id of each
upgrade step when it is run. then the UI would show an upgrade step as
needing to be run if a) the loaded profile version is greater than the source
version specified in the ZCML and b) the id of the upgrade step is not yet
stored in the setup tool's list of completed steps.

feel like fixing it? ;)

-r

_______________________________________________
Zope-CMF maillist - Zope-CMF [at] lists
http://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

m.van.rees at zestsoftware

Apr 16, 2009, 5:39 PM

Post #4 of 13 (1476 views)
Permalink

Re: Best use of source numbers in GS upgrade steps? [In reply to]

Hello yuppie!

yuppie, on 2009-04-16:
> Maurits van Rees wrote:
>> So my question is: is this a sane way of doing this? Is it alright to
>> specify a version (or really a profile revision) as source when that
>> version does not yet exist? It works fine as far as I can tell.
>
> AFAICS this will not work with GenericSetup trunk. Maybe you want to
> show the upgrade step for *all* versions before 1.2? In that case you
> don't specify any source version.

Yes, that is what I want. Looking at the trunk code, setting '*' as
source should have the exact behaviour that I want. Sweet!

> BTW: The relevant behavior is quite inconsistent in GenericSetup
> 1.4.

Right, in the 1.4 code I see for example that the destination number
is never actually used, apart from showing it on the upgrades tab.

> I added several tests and cleaned up the behavior on the trunk:
> http://svn.zope.org/*checkout*/Products.GenericSetup/trunk/Products/GenericSetup/tests/upgrade.txt
> Please let me know if I did break useful behavior.

Ah, that looks much saner, thanks! Nothing breaks here AFAICT.

And this tells me that my way of specifying source=1.1.9 and dest=2.0
should still work. A snippet of those tests adapted to my numbers
gives this result:

1.1.9 (source) > 1.1.2 (current) < 1.2 (dest)

>>> e.versionMatch('1.1.2')
False
>>> e.isProposed(tool, '1.1.2')
False
>>> bool(_extractStepInfo(tool, 'ID', e, '1.1.2'))
True

So the version does not match and the step is not proposed, but the
step info is extracted anyway, and as far as I can see this is what
matters in the end, as this is called in listUpgradeSteps.

BTW, do I understand correctly that when in this example we add a
checker that returns False the step will still be shown?

Specifying '*' instead of '1.1.9' as source is conceptually better and
works just as well on trunk. But on GS 1.4 this has the effect of
always listing that upgrade step, as the current version is never
compared to the destination.

To get the exact same behaviour on 1.4 as on trunk I guess I could
copy the versionMatch code from trunk and add that as a checker to my
upgrade step... Seems silly though. :-)

Okay, my conclusion: I will stick to specifying 1.1.9 as source in
this case. Alternatively I will use '*' as source and make very sure
that running those upgrade steps a second time has no adverse effects
and is fast. And for an upgrade step in a package that is meant for
GenericSetup/Plone trunk I will use source="*".

Since I made some notes while investigating, I might as well share
them. So here are some random observations for reference, with some
CMFPlone versions thrown in for good measure.

- GS 1.3.3 is used in Plone 3.0.6.

- GS 1.4.1 is used in Plone 3.1.7.

- GS 1.4.2.2 is used in Plone 3.2.2.

- GS 1.4.2.2 is used in Plone 3.3rc1.

- GS trunk (1.5) is used in Plone trunk (4.0).

- The upgrade.py file is exactly the same in GS 1.3.x and 1.4.x, so
upgrade step behaviour should be the same in Plone 3.0-3.3.

- Checkers:

- GS 1.3/1.4: if a step has a checker, then the source and
destination do not matter anymore: only the return value of the
checker matters.

- GS trunk (1.5): if a step has a checker, then its return value is
used together with checks on the source and destination numbers.

- Destination:

- GS 1.3/1.4: the version destination does not matter at all (!) as
it is not used anywhere apart from being shown in the UI...

- GS trunk (1.5): the destination matters, as it is compared to the
currently applied profile revision.

--
Maurits van Rees | http://maurits.vanrees.org/
Work | http://zestsoftware.nl/
"This is your day, don't let them take it away." [Barlow Girl]

_______________________________________________
Zope-CMF maillist - Zope-CMF [at] lists
http://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

y.2009 at wcm-solutions

Apr 16, 2009, 11:53 PM

Post #5 of 13 (1478 views)
Permalink

Re: Best use of source numbers in GS upgrade steps? [In reply to]

Hi Maurits!

Maurits van Rees wrote:
> yuppie, on 2009-04-16:
>> I added several tests and cleaned up the behavior on the trunk:
>> http://svn.zope.org/*checkout*/Products.GenericSetup/trunk/Products/GenericSetup/tests/upgrade.txt
>> Please let me know if I did break useful behavior.
>
> Ah, that looks much saner, thanks! Nothing breaks here AFAICT.
>
> And this tells me that my way of specifying source=1.1.9 and dest=2.0
> should still work. A snippet of those tests adapted to my numbers
> gives this result:
>
> 1.1.9 (source) > 1.1.2 (current) < 1.2 (dest)
>
> >>> e.versionMatch('1.1.2')
> False
> >>> e.isProposed(tool, '1.1.2')
> False
> >>> bool(_extractStepInfo(tool, 'ID', e, '1.1.2'))
> True

If you add that test to the trunk this behavior will become officially
supported...

> So the version does not match and the step is not proposed, but the
> step info is extracted anyway, and as far as I can see this is what
> matters in the end, as this is called in listUpgradeSteps.

Well. I was focused on 'isProposed'. This is the definition:

"""Check if a step can be applied.

False means already applied or does not apply.
True means can be applied.
"""

Your result is False for a step that should be applied.

> BTW, do I understand correctly that when in this example we add a
> checker that returns False the step will still be shown?

*All* unchecked steps are "not recommended, run on your own risk". The
checker is just a restriction like versionMatch. In same cases it might
still make sense to (re-)run these steps.

But I'm sure the UI can be improved. Please note that hiding these steps
better will also hide your unchecked step.

Cheers,

Yuppie

_______________________________________________
Zope-CMF maillist - Zope-CMF [at] lists
http://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

y.2009 at wcm-solutions

Apr 16, 2009, 11:55 PM

Post #6 of 13 (1478 views)
Permalink

Re: Best use of source numbers in GS upgrade steps? [In reply to]

Rob Miller wrote:
> thinking about it now, though, i'd say perhaps the zcml should support only
> including a "source" version, with the setup tool persisting the id of each
> upgrade step when it is run. then the UI would show an upgrade step as
> needing to be run if a) the loaded profile version is greater than the source
> version specified in the ZCML and b) the id of the upgrade step is not yet
> stored in the setup tool's list of completed steps.
>
> feel like fixing it? ;)

Did you have a look at the changes I made on GS trunk? Looks like your
proposal is based on the 1.4 code and moves in a different direction.

Cheers, Yuppie

_______________________________________________
Zope-CMF maillist - Zope-CMF [at] lists
http://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

wichert at wiggy

Apr 17, 2009, 12:16 AM

Post #7 of 13 (1472 views)
Permalink

Re: Best use of source numbers in GS upgrade steps? [In reply to]

Previously Maurits van Rees wrote:
> Since I made some notes while investigating, I might as well share
> them. So here are some random observations for reference, with some
> CMFPlone versions thrown in for good measure.
>
> - GS 1.3.3 is used in Plone 3.0.6.
>
> - GS 1.4.1 is used in Plone 3.1.7.
>
> - GS 1.4.2.2 is used in Plone 3.2.2.
>
> - GS 1.4.2.2 is used in Plone 3.3rc1.
>
> - GS trunk (1.5) is used in Plone trunk (4.0).

I'm planning to switch to GS 1.5 for Plone 3.4.

Wichert.

--
Wichert Akkerman <wichert [at] wiggy> It is simple to make things.
http://www.wiggy.net/ It is hard to make things simple.
_______________________________________________
Zope-CMF maillist - Zope-CMF [at] lists
http://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

wichert at wiggy

Apr 17, 2009, 12:16 AM

Post #8 of 13 (1474 views)
Permalink

Re: Best use of source numbers in GS upgrade steps? [In reply to]

wichert at wiggy

Apr 17, 2009, 12:18 AM

Post #9 of 13 (1473 views)
Permalink

Re: Best use of source numbers in GS upgrade steps? [In reply to]

Previously Rob Miller wrote:
> Maurits van Rees wrote:
> > Hi,
> >
> > I wonder what the best way is of specifying the source number of a
> > GenericSetup upgrade step.
> >
> <---SNIP a bunch of stuff about how using versions to mark GS upgrade steps is
> annoying--->
>
> yes, as you discovered, using source and destination versions to mark when
> upgrade steps should be run can be a bit janky. i realized this when i first
> ported the upgrade stuff over from CPS, but i wasn't quite sure how else to
> handle it.
>
> thinking about it now, though, i'd say perhaps the zcml should support only
> including a "source" version, with the setup tool persisting the id of each
> upgrade step when it is run. then the UI would show an upgrade step as
> needing to be run if a) the loaded profile version is greater than the source
> version specified in the ZCML and b) the id of the upgrade step is not yet
> stored in the setup tool's list of completed steps.

-1

This is a radical change in behaviour, and I'm not convinced it is
better. There may be good reasons to skip certain upgrade steps when you
move from one source to another, which is no longer possible with your
proposal.

Wichert.

--
Wichert Akkerman <wichert [at] wiggy> It is simple to make things.
http://www.wiggy.net/ It is hard to make things simple.
_______________________________________________
Zope-CMF maillist - Zope-CMF [at] lists
http://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

wichert at wiggy

Apr 17, 2009, 12:18 AM

Post #10 of 13 (1469 views)
Permalink

Re: Best use of source numbers in GS upgrade steps? [In reply to]

m.van.rees at zestsoftware

Apr 17, 2009, 5:44 AM

Post #11 of 13 (1470 views)
Permalink

Re: Best use of source numbers in GS upgrade steps? [In reply to]

yuppie, on 2009-04-17:
>> And this tells me that my way of specifying source=1.1.9 and dest=2.0
>> should still work. A snippet of those tests adapted to my numbers
>> gives this result:
>>
>> 1.1.9 (source) > 1.1.2 (current) < 1.2 (dest)
>>
>> >>> e.versionMatch('1.1.2')
>> False
>> >>> e.isProposed(tool, '1.1.2')
>> False
>> >>> bool(_extractStepInfo(tool, 'ID', e, '1.1.2'))
>> True
>
> If you add that test to the trunk this behavior will become officially
> supported...

Ah, I meant that this test is already in trunk. The versions there
are slightly different but not in a relevant way.

trunk:

1.1 > 1.0 < 1.2

me:

1.1.9 > 1.1.2 < 1.2

and those two comparisons are exactly the same in principle:

source > current < dest

So trunk works fine for me.

--
Maurits van Rees | http://maurits.vanrees.org/
Work | http://zestsoftware.nl/
"This is your day, don't let them take it away." [Barlow Girl]

_______________________________________________
Zope-CMF maillist - Zope-CMF [at] lists
http://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

m.van.rees at zestsoftware

Apr 17, 2009, 5:44 AM

Post #12 of 13 (1472 views)
Permalink

Re: Best use of source numbers in GS upgrade steps? [In reply to]

Wichert Akkerman, on 2009-04-17:
> Previously Maurits van Rees wrote:
>> Since I made some notes while investigating, I might as well share
>> them. So here are some random observations for reference, with some
>> CMFPlone versions thrown in for good measure.
>>
>> - GS 1.3.3 is used in Plone 3.0.6.
>>
>> - GS 1.4.1 is used in Plone 3.1.7.
>>
>> - GS 1.4.2.2 is used in Plone 3.2.2.
>>
>> - GS 1.4.2.2 is used in Plone 3.3rc1.
>>
>> - GS trunk (1.5) is used in Plone trunk (4.0).
>
> I'm planning to switch to GS 1.5 for Plone 3.4.

+1

--
Maurits van Rees | http://maurits.vanrees.org/
Work | http://zestsoftware.nl/
"This is your day, don't let them take it away." [Barlow Girl]

_______________________________________________
Zope-CMF maillist - Zope-CMF [at] lists
http://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

ra at burningman

Apr 17, 2009, 12:20 PM

Post #13 of 13 (1470 views)
Permalink

Re: Best use of source numbers in GS upgrade steps? [In reply to]

yuppie wrote:
> Rob Miller wrote:
>> thinking about it now, though, i'd say perhaps the zcml should support only
>> including a "source" version, with the setup tool persisting the id of each
>> upgrade step when it is run. then the UI would show an upgrade step as
>> needing to be run if a) the loaded profile version is greater than the source
>> version specified in the ZCML and b) the id of the upgrade step is not yet
>> stored in the setup tool's list of completed steps.
>>
>> feel like fixing it? ;)
>
> Did you have a look at the changes I made on GS trunk? Looks like your
> proposal is based on the 1.4 code and moves in a different direction.

nope, i didn't. and, as wichert indicated, my idea makes it hard to avoid
running a step, if you don't want to run it. i'm happy to let things progress
as they are. :)

-r

_______________________________________________
Zope-CMF maillist - Zope-CMF [at] lists
http://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads