Mailing List Archive: Zope: CMF

Test setup problems with Zope 2.13

Index | Next | Previous | View Threaded

hanno at hannosch

Jul 11, 2010, 4:54 AM

Post #1 of 15 (1807 views)
Permalink

Test setup problems with Zope 2.13

Hi.

Maybe you have noticed from my commits today, but we have some test
setup problems in Zope 2.13. I tried to solve those, but failed so
far. Unfortunately we don't have nightly test runs for any CMF + Zope
2.13 combination for a while, so I'm not sure when all the problems
started.

One recent problem is due to my refactoring of Products.OFSP / OFS
though. I moved the registrations of all standard OFS types from the
initialize of OFSP into an initialize in OFS itself. In order for this
to be picked up, I added the five:registerPackage statement, including
a hint at the initialize function to OFS' configure.zcml.

During normal Zope startup this works fine and the initialize is
called. But we have a problem with ZopeTestCase based tests. So far it
did an installProduct('OFSP') call unconditionally at module scope. As
soon as someone imported it, the OFS types got registered. If the
ZopeLite layer was active, the call was deferred to its layer setup
time.

In Zope 2.13 I removed the installProduct('OFSP') call, as it doesn't
do anything anymore (apart from registering help system stuff which
nobody needs in tests). But I cannot simply put an
installPackage('OFS') in there as a replacement. In order for that to
work, one needs to load the OFS.configure.zcml first (the
five:registerPackage to be exact). Since ZopeTestCase doesn't know
anything about ZCML setup so far, I don't want to introduce any such
handling. Consumers of ZopeTestCase have all dealt with ZCML loading
in various different ways and I expect even bigger problems when
changing anything there.

But this means any testcase that needs the OFS types (to show up in
all_meta_types for example) will need to set up the ZCML and call
installPackage itself. I tried to do this in CMFCore by adding a new
layer, but cannot get it to work. There's always some side-effects in
other tests. It doesn't help that installPackage internally does a
transaction.commit() in 2.12 but not in 2.13.

If anyone more familiar with the multitude of CMF test layers could
help, that would be much appreciated.

Hanno
_______________________________________________
Zope-CMF maillist - Zope-CMF [at] zope
https://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

charlie.clark at clark-consulting

Jul 11, 2010, 10:31 AM

Post #2 of 15 (1759 views)
Permalink

Re: Test setup problems with Zope 2.13 [In reply to]

Am 11.07.2010, 13:54 Uhr, schrieb Hanno Schlichting <hanno [at] hannosch>:

> Hi.
> Maybe you have noticed from my commits today, but we have some test
> setup problems in Zope 2.13. I tried to solve those, but failed so
> far. Unfortunately we don't have nightly test runs for any CMF + Zope
> 2.13 combination for a while, so I'm not sure when all the problems
> started.

Hi Hanno,

I don't think I can be much help on the testing side as I don't understand
the CMFCore internals that well myself. However, as things stand it's not
just the tests that are broken:

2010-07-11 19:23:38 ERROR Application Couldn't install BTreeFolder2
Traceback (most recent call last):
File
"/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/src/Zope2/src/OFS/Application.py",
line 691, in install_product
initmethod(context)
File
"/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/src/Zope2/src/Products/BTreeFolder2/__init__.py",
line 26, in initialize
File
"/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/src/Zope2/src/App/ProductContext.py",
line 224, in registerClass
icon = ImageFile(icon, self.__pack.__dict__)
File
"/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/src/Zope2/src/App/ImageFile.py",
line 77, in __init__
stat_info = os.stat(path)
OSError: [Errno 2] No such file or directory:
'/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/src/Zope2/src/Products/BTreeFolder2/btreefolder2.gif'
Traceback (most recent call last):
File "/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/bin/runzope",
line 92, in <module>
Zope2.Startup.run.run()
File
"/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/src/Zope2/src/Zope2/Startup/run.py",
line 21, in run
starter.prepare()
File
"/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/src/Zope2/src/Zope2/Startup/__init__.py",
line 86, in prepare
self.startZope()
File
"/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/src/Zope2/src/Zope2/Startup/__init__.py",
line 259, in startZope
Zope2.startup()
File
"/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/src/Zope2/src/Zope2/__init__.py",
line 47, in startup
_startup()
File
"/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/src/Zope2/src/Zope2/App/startup.py",
line 125, in startup
OFS.Application.initialize(application)
File
"/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/src/Zope2/src/OFS/Application.py",
line 253, in initialize
initializer.initialize()
File
"/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/src/Zope2/src/OFS/Application.py",
line 279, in initialize
self.install_products()
File
"/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/src/Zope2/src/OFS/Application.py",
line 506, in install_products
return install_products(app)
File
"/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/src/Zope2/src/OFS/Application.py",
line 537, in install_products
folder_permissions, raise_exc=debug_mode)
File
"/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/src/Zope2/src/OFS/Application.py",
line 691, in install_product
initmethod(context)
File
"/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/src/Zope2/src/Products/BTreeFolder2/__init__.py",
line 26, in initialize

File
"/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/src/Zope2/src/App/ProductContext.py",
line 224, in registerClass
icon = ImageFile(icon, self.__pack.__dict__)
File
"/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/src/Zope2/src/App/ImageFile.py",
line 77, in __init__
stat_info = os.stat(path)
OSError: [Errno 2] No such file or directory:
'/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/src/Zope2/src/Products/BTreeFolder2/btreefolder2.gif'

I see from the updates that Products/BTreeFolder2 has been deleted /
factored out. The question is - where's it gone to? Or what bits of my
buildout do I need to reinstall?

Charlie
--
Charlie Clark
Managing Director
Clark Consulting & Research
German Office
Helmholtzstr. 20
Düsseldorf
D- 40215
Tel: +49-211-600-3657
Mobile: +49-178-782-6226
_______________________________________________
Zope-CMF maillist - Zope-CMF [at] zope
https://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

hanno at hannosch

Jul 11, 2010, 10:39 AM

Post #3 of 15 (1766 views)
Permalink

Re: Test setup problems with Zope 2.13 [In reply to]

Hi Charlie,

On Sun, Jul 11, 2010 at 7:31 PM, Charlie Clark
<charlie.clark [at] clark-consulting> wrote:
> 2010-07-11 19:23:38 ERROR Application Couldn't install BTreeFolder2
> Traceback (most recent call last):
...
> stat_info = os.stat(path)
> OSError: [Errno 2] No such file or directory:
> '/Users/charlieclark/Sites/cmf-svn/CMF.buildout/trunk/src/Zope2/src/Products/BTreeFolder2/btreefolder2.gif'
>
> I see from the updates that Products/BTreeFolder2 has been deleted /
> factored out. The question is - where's it gone to? Or what bits of my
> buildout do I need to reinstall?

When you work against an SVN checkout, please make sure to always run
both SVN update and bin/buildout. Also check svn status for orphaned
directories (those with a question mark) and delete them.

Products.BTreeFolder2 doesn't live inside the Zope2/src tree anymore
but is a standalone distribution. In Zope 2.13 this distribution is
still a dependency of Zope2 and installed when running bin/buildout.
Only in Zope 2.14 will it be a complete optional dependency.

If you follow my above instructions, you should get a working buildout again.

Hanno
_______________________________________________
Zope-CMF maillist - Zope-CMF [at] zope
https://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

charlie.clark at clark-consulting

Jul 11, 2010, 11:11 AM

Post #4 of 15 (1760 views)
Permalink

Re: Test setup problems with Zope 2.13 [In reply to]

Am 11.07.2010, 19:39 Uhr, schrieb Hanno Schlichting <hanno [at] hannosch>:

> When you work against an SVN checkout, please make sure to always run
> both SVN update and bin/buildout. Also check svn status for orphaned
> directories (those with a question mark) and delete them.

Hi Hanno,

thanks for patiently pointing out the obvious! I blame the heat.

Affected are: Products.BTreeFolder2, Products.ExternalMethods,
Products.MIMETools, Products.OFSP, Products.PythonScripts,
Products.ZSQLMethods, Shared.DC.ZRDB

Charlie
--
Charlie Clark
Managing Director
Clark Consulting & Research
German Office
Helmholtzstr. 20
Düsseldorf
D- 40215
Tel: +49-211-600-3657
Mobile: +49-178-782-6226
_______________________________________________
Zope-CMF maillist - Zope-CMF [at] zope
https://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

hanno at hannosch

Jul 11, 2010, 11:24 AM

Post #5 of 15 (1757 views)
Permalink

Re: Test setup problems with Zope 2.13 [In reply to]

On Sun, Jul 11, 2010 at 8:11 PM, Charlie Clark
<charlie.clark [at] clark-consulting> wrote:
> thanks for patiently pointing out the obvious! I blame the heat.

No problem. This happens rarely enough. I just happen to take apart
Zope 2 at the moment :)

> Affected are: Products.BTreeFolder2, Products.ExternalMethods,
> Products.MIMETools, Products.OFSP, Products.PythonScripts,
> Products.ZSQLMethods, Shared.DC.ZRDB

And by now it's also zLOG and Products.StandardCacheManagers, more to come.

Hanno
_______________________________________________
Zope-CMF maillist - Zope-CMF [at] zope
https://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

hanno at hannosch

Jul 19, 2010, 11:53 AM

Post #6 of 15 (1706 views)
Permalink

Re: Test setup problems with Zope 2.13 [In reply to]

Hi.

Looks like nobody feels like jumping into the test layers waters.

Unless someone does in the next days, I'll revert the OFS / OFSP
refactoring on Zope trunk.

Hanno

On Sun, Jul 11, 2010 at 1:54 PM, Hanno Schlichting <hanno [at] hannosch> wrote:
> Maybe you have noticed from my commits today, but we have some test
> setup problems in Zope 2.13. I tried to solve those, but failed so
> far. Unfortunately we don't have nightly test runs for any CMF + Zope
> 2.13 combination for a while, so I'm not sure when all the problems
> started.
>
> One recent problem is due to my refactoring of Products.OFSP / OFS
> though. I moved the registrations of all standard OFS types from the
> initialize of OFSP into an initialize in OFS itself. In order for this
> to be picked up, I added the five:registerPackage statement, including
> a hint at the initialize function to OFS' configure.zcml.
>
> During normal Zope startup this works fine and the initialize is
> called. But we have a problem with ZopeTestCase based tests. So far it
> did an installProduct('OFSP') call unconditionally at module scope. As
> soon as someone imported it, the OFS types got registered. If the
> ZopeLite layer was active, the call was deferred to its layer setup
> time.
>
> In Zope 2.13 I removed the installProduct('OFSP') call, as it doesn't
> do anything anymore (apart from registering help system stuff which
> nobody needs in tests). But I cannot simply put an
> installPackage('OFS') in there as a replacement. In order for that to
> work, one needs to load the OFS.configure.zcml first (the
> five:registerPackage to be exact). Since ZopeTestCase doesn't know
> anything about ZCML setup so far, I don't want to introduce any such
> handling. Consumers of ZopeTestCase have all dealt with ZCML loading
> in various different ways and I expect even bigger problems when
> changing anything there.
>
> But this means any testcase that needs the OFS types (to show up in
> all_meta_types for example) will need to set up the ZCML and call
> installPackage itself. I tried to do this in CMFCore by adding a new
> layer, but cannot get it to work. There's always some side-effects in
> other tests. It doesn't help that installPackage internally does a
> transaction.commit() in 2.12 but not in 2.13.
>
> If anyone more familiar with the multitude of CMF test layers could
> help, that would be much appreciated.
_______________________________________________
Zope-CMF maillist - Zope-CMF [at] zope
https://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

y.2010 at wcm-solutions

Jul 22, 2010, 4:37 AM

Post #7 of 15 (1677 views)
Permalink

Re: Test setup problems with Zope 2.13 [In reply to]

Hi Hanno!

Hanno Schlichting wrote:
> Looks like nobody feels like jumping into the test layers waters.

Finally found some time to look into this.

> Unless someone does in the next days, I'll revert the OFS / OFSP
> refactoring on Zope trunk.

AFAICS this is the point where you went astray:

http://svn.zope.org/?rev=114576&view=rev

If you revert that revision and fix installPackage instead of trying to
work around the bug everything seems to work fine. I'm not in the mood
to write tests for installPackage and to figure out what else needs to
be reverted, so I just attached my patch.

Cheers,

Yuppie

Index: Zope2/src/Testing/ZopeTestCase/ZopeLite.py
===================================================================
--- Zope2/src/Testing/ZopeTestCase/ZopeLite.py (revision 114921)
+++ Zope2/src/Testing/ZopeTestCase/ZopeLite.py (working copy)
@@ -194,6 +194,7 @@
from OFS.metaconfigure import has_package
return has_package(name)

+@layer.onsetup
def installPackage(name, quiet=0):
'''Installs a registered Python package.'''
quiet = 1 # Ignore argument
@@ -218,6 +219,7 @@
if not quiet: _print('Installing %s ... NOT FOUND\n' % name)

installProduct('PluginIndexes', 1) # Must install first
+installPackage('OFS', 1)

# So people can use ZopeLite.app()
app = Zope2.app
_______________________________________________
Zope-CMF maillist - Zope-CMF [at] zope
https://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

y.2010 at wcm-solutions

Jul 22, 2010, 5:00 AM

Post #8 of 15 (1688 views)
Permalink

Re: Test setup problems with Zope 2.13 [In reply to]

yuppie wrote:
> +@layer.onsetup
> def installPackage(name, quiet=0):
> '''Installs a registered Python package.'''

Oops!

For some unknown reason @layer.onsetup was explicitly removed here:

http://svn.zope.org/?view=rev&rev=89898

???, Yuppie
_______________________________________________
Zope-CMF maillist - Zope-CMF [at] zope
https://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

hanno at hannosch

Jul 24, 2010, 6:26 AM

Post #9 of 15 (1667 views)
Permalink

Re: Test setup problems with Zope 2.13 [In reply to]

On Thu, Jul 22, 2010 at 2:00 PM, yuppie <y.2010 [at] wcm-solutions> wrote:
> yuppie wrote:
>> +@layer.onsetup
>> def installPackage(name, quiet=0):
>> '''Installs a registered Python package.'''
>
> Oops!
>
> For some unknown reason @layer.onsetup was explicitly removed here:
>
> http://svn.zope.org/?view=rev&rev=89898

Hhm. This all looks like a can of worms to me. I think I'll still
revert my changes for 2.13 and see if I find the proper motivation to
fix this for 2.14 instead.

Hanno
_______________________________________________
Zope-CMF maillist - Zope-CMF [at] zope
https://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

hanno at hannosch

Jul 25, 2010, 3:17 AM

Post #10 of 15 (1658 views)
Permalink

Re: Test setup problems with Zope 2.13 [In reply to]

On Sat, Jul 24, 2010 at 3:26 PM, Hanno Schlichting <hanno [at] hannosch> wrote:
> Hhm. This all looks like a can of worms to me. I think I'll still
> revert my changes for 2.13 and see if I find the proper motivation to
> fix this for 2.14 instead.

I reverted this now and all tests pass again. Both CMF 2.2 and CMF
trunk against Zope 2.13.

Since I do want to use the CMF 2.2 + Zope 2.13 combination for Plone
4.1, I'd prefer that to be stable. I need to tell that to myself more
often ;)

Hanno
_______________________________________________
Zope-CMF maillist - Zope-CMF [at] zope
https://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

y.2010 at wcm-solutions

Jul 27, 2010, 4:09 AM

Post #11 of 15 (1637 views)
Permalink

Re: Test setup problems with Zope 2.13 [In reply to]

Hi!

Hanno Schlichting wrote:
> On Sat, Jul 24, 2010 at 3:26 PM, Hanno Schlichting<hanno-+xqiSR9SqHvsq35pWSNszA [at] public> wrote:
>> Hhm. This all looks like a can of worms to me. I think I'll still
>> revert my changes for 2.13 and see if I find the proper motivation to
>> fix this for 2.14 instead.

The whole registerPackage concept is a can of worms. We need real ZCML
directives for *all* the stuff currently done in initialization
functions. And to deprecate the usage of initialize instead of expanding
that old concept to non-product packages.

> I reverted this now and all tests pass again. Both CMF 2.2 and CMF
> trunk against Zope 2.13.

I guess some more changes should be reverted. There is at least a
ZopeTestCase.installPackage('OFS') in the FunctionalLayer of CMFCalendar.

Cheers,

Yuppie
_______________________________________________
Zope-CMF maillist - Zope-CMF [at] zope
https://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

hanno at hannosch

Jul 28, 2010, 2:45 AM

Post #12 of 15 (1626 views)
Permalink

Re: Test setup problems with Zope 2.13 [In reply to]

Morning.

On Tue, Jul 27, 2010 at 1:09 PM, yuppie <y.2010 [at] wcm-solutions> wrote:
> The whole registerPackage concept is a can of worms. We need real ZCML
> directives for *all* the stuff currently done in initialization
> functions. And to deprecate the usage of initialize instead of expanding
> that old concept to non-product packages.

Right. I think it's not all that many things left there.

Zope2 has:

registerClass
registerIcon
HelpSys (effectively deprecated)

CMF brings in:

ToolInit
ContentInit

I'd love to see someone tackle registerClass and at the same time
decouple ZMI add menu visibility and the OFS.CopySupport stuff. Well,
in general the CopySupport stuff could use some work and merging with
zope.copy / zope.copypastemove to correctly deal with persistent
parent pointers. But that's another story that also applies to the
ZEXP mechanism.

>> I reverted this now and all tests pass again. Both CMF 2.2 and CMF
>> trunk against Zope 2.13.
>
> I guess some more changes should be reverted. There is at least a
> ZopeTestCase.installPackage('OFS') in the FunctionalLayer of CMFCalendar.

I reverted the corresponding changeset and looked over my other
commits. I think that one was the only other change.

Hanno
_______________________________________________
Zope-CMF maillist - Zope-CMF [at] zope
https://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

y.2010 at wcm-solutions

Jul 28, 2010, 4:17 AM

Post #13 of 15 (1623 views)
Permalink

Re: Test setup problems with Zope 2.13 [In reply to]

Hi!

Hanno Schlichting wrote:
> Zope2 has:
>
> registerClass
> registerIcon

registerIcon is defined in CMFCore, not in Zope2.

> HelpSys (effectively deprecated)
>
> CMF brings in:
>
> ToolInit
> ContentInit

ContentInit is only used for BBB. It registers oldstyle constructors.

> I'd love to see someone tackle registerClass and at the same time
> decouple ZMI add menu visibility and the OFS.CopySupport stuff.

You mean the CopySupport issue I fixed 5 years ago? Or something
different? http://svn.zope.org/?view=rev&rev=40552

five:registerClass doesn't require to specify an 'addview'. CopySupport
works for all CMF content types without adding them to the ZMI add menu.

We just can't use five:registerClass in OFS/OFSP because it doesn't
support oldstyle constructors and legacy methods.

Cheers,

Yuppie
_______________________________________________
Zope-CMF maillist - Zope-CMF [at] zope
https://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

hanno at hannosch

Jul 28, 2010, 5:28 AM

Post #14 of 15 (1624 views)
Permalink

Re: Test setup problems with Zope 2.13 [In reply to]

On Wed, Jul 28, 2010 at 1:17 PM, yuppie <y.2010 [at] wcm-solutions> wrote:
> Hanno Schlichting wrote:
>> ContentInit
>
> ContentInit is only used for BBB. It registers oldstyle constructors.

Well, right. Except in Plone / Archetypes this and variants based on
it are still the only option. Archetypes doesn't know about add views
or any of that "new" IFactory pattern.

>> I'd love to see someone tackle registerClass and at the same time
>> decouple ZMI add menu visibility and the OFS.CopySupport stuff.
>
> You mean the CopySupport issue I fixed 5 years ago? Or something
> different? http://svn.zope.org/?view=rev&rev=40552
>
> five:registerClass doesn't require to specify an 'addview'. CopySupport
> works for all CMF content types without adding them to the ZMI add menu.
>
> We just can't use five:registerClass in OFS/OFSP because it doesn't
> support oldstyle constructors and legacy methods.

Ah, yeah! This was exactly what I meant. I had no idea this was already done.

So a belated thank you :)
Hanno
_______________________________________________
Zope-CMF maillist - Zope-CMF [at] zope
https://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

charlie.clark at clark-consulting

Jul 28, 2010, 5:32 AM

Post #15 of 15 (1622 views)
Permalink

Re: Test setup problems with Zope 2.13 [In reply to]

Am 28.07.2010, 14:28 Uhr, schrieb Hanno Schlichting <hanno [at] hannosch>:

> Well, right. Except in Plone / Archetypes this and variants based on
> it are still the only option. Archetypes doesn't know about add views
> or any of that "new" IFactory pattern.

Hi,

thanks for reverting the changes and all the other fine work you're doing.
I'm not sure about the scope of these changes but I'm happy to chip in but
not sure what you want done with ContentInit. Can't Archetypes be updated
to work with this stuff?

Charlie
--
Charlie Clark
Managing Director
Clark Consulting & Research
German Office
Helmholtzstr. 20
Düsseldorf
D- 40215
Tel: +49-211-600-3657
Mobile: +49-178-782-6226
_______________________________________________
Zope-CMF maillist - Zope-CMF [at] zope
https://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads