Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Zope: CMF

CMFCatalogAware.dispatchToOpaqueItems madness

 

 

Zope cmf RSS feed   Index | Next | Previous | View Threaded


matth at netsight

Dec 18, 2008, 1:54 PM

Post #1 of 6 (580 views)
Permalink
CMFCatalogAware.dispatchToOpaqueItems madness

I really don't understand this code, so maybe I'm missing something here, but
anyone know wtf the opaqueItem code does in CMFCore.CMFCatalogAware?!

Why does the opaqueItems method iterate through *every* attribute in the root of
the site causing all objects (including, at least in plone) all sub-objects of
folders to be woken up too):

for name in self_base.__dict__.keys():
obj = getattr(self, name)
if ICallableOpaqueItem.providedBy(obj) \
or z2ICallableOpaqueItem.isImplementedBy(obj):
items.append((obj.getId(), obj))

If it is looking for opaque items (I still don't know what these really are)
then why does it look in content objects too?!

How about the changes below?

objectids = dict([(x,1) for x in self.objectIds()])

for name in self_base.__dict__.keys():
if name not in objectids:

obj = getattr(self, name)
if ICallableOpaqueItem.providedBy(obj) \
or z2ICallableOpaqueItem.isImplementedBy(obj):
items.append((obj.getId(), obj))

(that was probably all wrapped to hell)

Basically, I've got a production site which loads up several tens of thousands
of objects from the ZODB just to display the front page. Madness.

-Matt

--
Matt Hamilton matth[at]netsight.co.uk
Netsight Internet Solutions, Ltd. Understand. Develop. Deliver
http://www.netsight.co.uk +44 (0)117 9090901
Web Design | Zope/Plone Development & Consulting | Co-location | Hosting

_______________________________________________
Zope-CMF maillist - Zope-CMF[at]lists.zope.org
http://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests


matth at netsight

Dec 19, 2008, 1:48 AM

Post #2 of 6 (549 views)
Permalink
Re: CMFCatalogAware.dispatchToOpaqueItems madness [In reply to]

Matt Hamilton <matth[at]...> writes:

>
> I really don't understand this code, so maybe I'm missing something here, but
> anyone know wtf the opaqueItem code does in CMFCore.CMFCatalogAware?!
>
> Why does the opaqueItems method iterate through *every* attribute in the root
> of the site causing all objects (including, at least in plone) all sub-objects
> of folders to be woken up too):

Of course as soon as you find the problem, you then know how to ask the
question... and in looking for opaqueitems, found that Helge has already got
there and started waving some performance pixie dust over it:

http://pypi.python.org/pypi/experimental.opaquespeedup

Still... I don't quite understand why CMF is doing what it is doing in the first
place.

-Matt

--
Matt Hamilton matth[at]netsight.co.uk
Netsight Internet Solutions, Ltd. Understand. Develop. Deliver
http://www.netsight.co.uk +44 (0)117 9090901
Web Design | Zope/Plone Development & Consulting | Co-location | Hosting



_______________________________________________
Zope-CMF maillist - Zope-CMF[at]lists.zope.org
http://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests


charlie at begeistert

Dec 19, 2008, 2:34 AM

Post #3 of 6 (542 views)
Permalink
Re: CMFCatalogAware.dispatchToOpaqueItems madness [In reply to]

Am 19.12.2008 um 10:48 schrieb Matt Hamilton:

> Of course as soon as you find the problem, you then know how to ask
> the
> question...

I think that sense of embarassment is an essential part of the
solution! ;-)

> and in looking for opaqueitems, found that Helge has already got
> there and started waving some performance pixie dust over it:
>
> http://pypi.python.org/pypi/experimental.opaquespeedup
>
> Still... I don't quite understand why CMF is doing what it is doing
> in the first
> place.


They are containers which won't be picked up by the normal methods. I
agree that the current practice of checking every attribute could be a
little expensive if you have lots of child objects stored in
attributes. I think the solution is probably to see if the problem
that they were introduced to address can't be solved in a different
manner. The discussion a couple of weeks ago about CMFCatalogAware
suggested that this class does indeed need refactoring for more
predictable behaviour.

Charlie
--
Charlie Clark
Helmholtzstr. 20
Düsseldorf
D- 40215
Tel: +49-211-938-5360
GSM: +49-178-782-6226



_______________________________________________
Zope-CMF maillist - Zope-CMF[at]lists.zope.org
http://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests


matth at netsight

Dec 19, 2008, 3:53 AM

Post #4 of 6 (548 views)
Permalink
Re: CMFCatalogAware.dispatchToOpaqueItems madness [In reply to]

Charlie Clark <charlie[at]...> writes:

>
>
> Am 19.12.2008 um 10:48 schrieb Matt Hamilton:
>
> > Of course as soon as you find the problem, you then know how to ask
> > the
> > question...
>
> I think that sense of embarassment is an essential part of the
> solution!

Indeed ;)

> > and in looking for opaqueitems, found that Helge has already got
> > there and started waving some performance pixie dust over it:
> >
> > http://pypi.python.org/pypi/experimental.opaquespeedup
> >
> > Still... I don't quite understand why CMF is doing what it is doing
> > in the first
> > place.
>
> They are containers which won't be picked up by the normal methods. I
> agree that the current practice of checking every attribute could be a
> little expensive if you have lots of child objects stored in
> attributes. I think the solution is probably to see if the problem
> that they were introduced to address can't be solved in a different
> manner. The discussion a couple of weeks ago about CMFCatalogAware
> suggested that this class does indeed need refactoring for more
> predictable behaviour.

The issue is it's not just finding the opaque objects but waking up all the
'normal' objects too. Hence it is looking in folders (which would already get
the event no doubt). I suppose the bit that really confuses me is why is a
BeforeTraverse event being handled and dispatched to these opaque objects by
code in CMFCatalogAware? My first thoughts when looking at the tracebacks in
pdb went something like 'OK I'm traversing, so a traversal event is fired...
wait a sec, why is CMFCatalogAware code being executed... I'm not indexing
anything... wtf?!'.

-Matt

--
Matt Hamilton matth[at]netsight.co.uk
Netsight Internet Solutions, Ltd. Understand. Develop. Deliver
http://www.netsight.co.uk +44 (0)117 9090901
Web Design | Zope/Plone Development & Consulting | Co-location | Hosting



_______________________________________________
Zope-CMF maillist - Zope-CMF[at]lists.zope.org
http://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests


tseaver at palladion

Dec 20, 2008, 1:09 AM

Post #5 of 6 (533 views)
Permalink
Re: CMFCatalogAware.dispatchToOpaqueItems madness [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Matt Hamilton wrote:
> I really don't understand this code, so maybe I'm missing something here, but
> anyone know wtf the opaqueItem code does in CMFCore.CMFCatalogAware?!

It allows propagating IObjectEvents to nested objects not findable via
the normal "content items" interface: the one example in the CMF is the
'talkback' attribute which holds DiscussionItems for an object.

> Why does the opaqueItems method iterate through *every* attribute in the root of
> the site causing all objects (including, at least in plone) all sub-objects of
> folders to be woken up too):
>
> for name in self_base.__dict__.keys():
> obj = getattr(self, name)
> if ICallableOpaqueItem.providedBy(obj) \
> or z2ICallableOpaqueItem.isImplementedBy(obj):
> items.append((obj.getId(), obj))
>
> If it is looking for opaque items (I still don't know what these really are)
> then why does it look in content objects too?!

The 'opaqueItems' method in CMFCatalogAware is called only by the
'opaqueIds' and 'opaqueValues' methods in the same class. The
'opaqueIds' method is never called in stock CMF at all. The
'opaqueValues' method is called by the event subscriber which dispatches
IObjectEvents.

> How about the changes below?
>
> objectids = dict([(x,1) for x in self.objectIds()])
>
> for name in self_base.__dict__.keys():
> if name not in objectids:
>
> obj = getattr(self, name)
> if ICallableOpaqueItem.providedBy(obj) \
> or z2ICallableOpaqueItem.isImplementedBy(obj):
> items.append((obj.getId(), obj))
>
> (that was probably all wrapped to hell)
>
> Basically, I've got a production site which loads up several tens of thousands
> of objects from the ZODB just to display the front page. Madness.

AFAIK, that stuff should have no impact on rendering the front page of
your site: the event handler should only be triggered on IObjectEvents
emitted for an object, which will typically only be on writes. You
wouldn't be doing a write-on-read for every rendering of the homepage,
would you? If so, you have way worse problems than this bit of code.



Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 tseaver[at]palladion.com
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJTLZZ+gerLs4ltQ4RAqNzAJwL2O6RpIkyKxTBKoGWQHlEnf4CaQCgngOG
GrgKYfmdvTC/Jlz4oxuQ40s=
=66GV
-----END PGP SIGNATURE-----

_______________________________________________
Zope-CMF maillist - Zope-CMF[at]lists.zope.org
http://mail.zope.org/mailman/listinfo/zope-cmf

See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests


tseaver at palladion

Dec 20, 2008, 1:33 AM

Post #6 of 6 (535 views)
Permalink
Re: CMFCatalogAware.dispatchToOpaqueItems madness [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tres Seaver wrote:
> Matt Hamilton wrote:
>> I really don't understand this code, so maybe I'm missing something here, but
>> anyone know wtf the opaqueItem code does in CMFCore.CMFCatalogAware?!
>
> It allows propagating IObjectEvents to nested objects not findable via
> the normal "content items" interface: the one example in the CMF is the
> 'talkback' attribute which holds DiscussionItems for an object.
>
>> Why does the opaqueItems method iterate through *every* attribute in the root of
>> the site causing all objects (including, at least in plone) all sub-objects of
>> folders to be woken up too):
>
>> for name in self_base.__dict__.keys():
>> obj = getattr(self, name)
>> if ICallableOpaqueItem.providedBy(obj) \
>> or z2ICallableOpaqueItem.isImplementedBy(obj):
>> items.append((obj.getId(), obj))
>
>> If it is looking for opaque items (I still don't know what these really are)
>> then why does it look in content objects too?!
>
> The 'opaqueItems' method in CMFCatalogAware is called only by the
> 'opaqueIds' and 'opaqueValues' methods in the same class. The
> 'opaqueIds' method is never called in stock CMF at all. The
> 'opaqueValues' method is called by the event subscriber which dispatches
> IObjectEvents.
>
>> How about the changes below?
>
>> objectids = dict([(x,1) for x in self.objectIds()])
>
>> for name in self_base.__dict__.keys():
>> if name not in objectids:
>
>> obj = getattr(self, name)
>> if ICallableOpaqueItem.providedBy(obj) \
>> or z2ICallableOpaqueItem.isImplementedBy(obj):
>> items.append((obj.getId(), obj))
>
>> (that was probably all wrapped to hell)
>
>> Basically, I've got a production site which loads up several tens of thousands
>> of objects from the ZODB just to display the front page. Madness.
>
> AFAIK, that stuff should have no impact on rendering the front page of
> your site: the event handler should only be triggered on IObjectEvents
> emitted for an object, which will typically only be on writes. You
> wouldn't be doing a write-on-read for every rendering of the homepage,
> would you? If so, you have way worse problems than this bit of code.

OK, my bad here: the *real* problem is that the event handler is
registered promiscuously for IObjectEvent, which turns out to get fired
a *lot*: BeforeTraversal event is the one which is affecting you.

The *only* events of real interest for opaque subobjects are the "added"
/ "cloned" / "removed" events, which already have their own event
handler ('handleOpaqueItemEvent'). The BeforeTraversal event should not
be propagaged to subitems at all.

The attached patch registers that handler for only the objects it should
properly care about. Can you verify that it makes your pain abate?

In any case, we need to review *all* of the event handlers registered
for IObjectEvent by the CMF.



Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 tseaver[at]palladion.com
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJTLvP+gerLs4ltQ4RAnnNAJ4sqG5LmmABuBWtNOA72O5FT5smTACdH/8Z
oUc8kMVUPVvt1wIhIH0h4OA=
=FkC7
-----END PGP SIGNATURE-----
Attachments: right_opaque_events.diff (1.10 KB)

Zope cmf RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact lists@gossamer-threads.com
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.