Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Zope: Dev

zope.intid and zope.keyreference.interfaces.NotYet exception (patch)

 

 

Zope dev RSS feed   Index | Next | Previous | View Threaded


cykooz at googlemail

Jan 10, 2012, 2:28 PM

Post #1 of 11 (703 views)
Permalink
zope.intid and zope.keyreference.interfaces.NotYet exception (patch)

Hi,

I created a patch for a package zope.intid that almost completely
solves the problem with the zope.keyreference.interfaces.NotYet
exception.

I implemented a deferred indexing of objects added to the container
which is not connected to the ZODB.
As soon as the container is added to the ZODB - all deferred objects
will be indexed.

I also added a marker IIntIdsDisabled for marking objects that must
not be indexed.


I would be grateful if someone could look at my changes and merge them
into the main branch.
My fork of zope.intid - https://bitbucket.org/cykooz/zope.intid/src


PS: Sorry for my English.

-------------------
Cykooz (Kirill Kuzminykh)
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


cykooz at googlemail

Jan 23, 2012, 2:20 PM

Post #2 of 11 (591 views)
Permalink
Re: zope.intid and zope.keyreference.interfaces.NotYet exception (patch) [In reply to]

Oh ...
Or there is no one who is engaged in package zope.intid, or no one
gets an NotYet exception on the fault this package.

Who can give me write access into SVN for the package zope.intid? And
also the right to add release into pypi.python.org?

PS: Sorry for my English.
-------------------
Cykooz (Kirill Kuzminykh)


2012/1/11 Cykooz <cykooz [at] googlemail>:
> Hi,
>
> I created a patch for a package zope.intid that almost completely
> solves the problem with the zope.keyreference.interfaces.NotYet
> exception.
>
> I implemented a deferred indexing of objects added to the container
> which is not connected to the ZODB.
> As soon as the container is added to the ZODB - all deferred objects
> will be indexed.
>
> I also added a marker IIntIdsDisabled for marking objects that must
> not be indexed.
>
>
> I would be grateful if someone could look at my changes and merge them
> into the main branch.
> My fork of zope.intid - https://bitbucket.org/cykooz/zope.intid/src
>
>
> PS: Sorry for my English.
>
> -------------------
> Cykooz (Kirill Kuzminykh)
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


charlie.clark at clark-consulting

Jan 24, 2012, 6:10 AM

Post #3 of 11 (589 views)
Permalink
Re: zope.intid and zope.keyreference.interfaces.NotYet exception (patch) [In reply to]

Hiya,

Am 23.01.2012, 23:20 Uhr, schrieb Cykooz <cykooz [at] googlemail>:

> Oh ...
> Or there is no one who is engaged in package zope.intid, or no one
> gets an NotYet exception on the fault this package.
> Who can give me write access into SVN for the package zope.intid?

You must apply to the Zope Foundation for access to the repository.

Charlie
--
Charlie Clark
Managing Director
Clark Consulting & Research
German Office
Kronenstr. 27a
Düsseldorf
D- 40217
Tel: +49-211-600-3657
Mobile: +49-178-782-6226
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


marius at gedmin

Jan 24, 2012, 2:33 PM

Post #4 of 11 (583 views)
Permalink
Re: zope.intid and zope.keyreference.interfaces.NotYet exception (patch) [In reply to]

On Tue, Jan 24, 2012 at 03:10:15PM +0100, Charlie Clark wrote:
> Am 23.01.2012, 23:20 Uhr, schrieb Cykooz <cykooz [at] googlemail>:
>
> >Oh ...
> >Or there is no one who is engaged in package zope.intid, or no one
> >gets an NotYet exception on the fault this package.
> >Who can give me write access into SVN for the package zope.intid?
>
> You must apply to the Zope Foundation for access to the repository.

http://foundation.zope.org/agreements is where you can find the
committer agreement form.

Marius Gedminas
--
http://pov.lt/ -- Zope 3/BlueBream consulting and development
Attachments: signature.asc (0.19 KB)


cykooz at googlemail

Jan 24, 2012, 2:49 PM

Post #5 of 11 (583 views)
Permalink
Re: zope.intid and zope.keyreference.interfaces.NotYet exception (patch) [In reply to]

2012/1/25 Marius Gedminas <marius [at] gedmin>:
>
> http://foundation.zope.org/agreements is where you can find the
> committer agreement form.

Thanks.
But I have one problem. I do not know any of the existing Committer
who can vouch for me.
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


janwijbrand at gmail

Jul 3, 2012, 2:47 AM

Post #6 of 11 (470 views)
Permalink
Re: zope.intid and zope.keyreference.interfaces.NotYet exception (patch) [In reply to]

On 1/24/12 23:49 , Cykooz wrote:
> 2012/1/25 Marius Gedminas <marius [at] gedmin>:
>>
>> http://foundation.zope.org/agreements is where you can find the
>> committer agreement form.
>
> Thanks.
> But I have one problem. I do not know any of the existing Committer
> who can vouch for me.

Bringing up a rather old thread:

Cykooz, was there any kind of follow up on this discussion? Are you
basically using a forked zope.keyreference for the time being?

Kind regards, jw


_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


cykooz at gmail

Jul 3, 2012, 4:26 AM

Post #7 of 11 (469 views)
Permalink
Re: zope.intid and zope.keyreference.interfaces.NotYet exception (patch) [In reply to]

2012/7/3 Jan-Wijbrand Kolman <janwijbrand [at] gmail>

> Are you basically using a forked zope.keyreference for the time being?
>

No, I do not use a forked zope.keyreference. I used my fork of the
zope.intid.


janwijbrand at gmail

Jul 3, 2012, 4:56 AM

Post #8 of 11 (470 views)
Permalink
Re: zope.intid and zope.keyreference.interfaces.NotYet exception (patch) [In reply to]

On 7/3/12 13:26 , Cykooz wrote:
> 2012/7/3 Jan-Wijbrand Kolman <janwijbrand [at] gmail
> <mailto:janwijbrand [at] gmail>>
>
> Are you basically using a forked zope.keyreference for the time being?
>
> No, I do not use a forked zope.keyreference. I used my fork of the
> zope.intid.

Ah, yes of course, that's what I meant: zope.intid.

I would not consider myself in a position currently to vouch for you to
have comitter rights. I do however wonder if anyone else would like to
comment on your proposed change.

If the comments a favorable I could try to help apply the patch to
zope.intid. This would probably help me and you as you then would not
have to use a forked package anymore.

At the end of this post, I pasted the diff from the current zope.intid
trunk against your "fork" on bitbucket. Maybe this would make it easier
for others to comment on it?

regards, jw


Proposed patch:
===============

diff -u zope.intid/trunk/src/zope/intid//__init__.py
zope.intid-cykooz/src/zope/intid//__init__.py
--- zope.intid/trunk/src/zope/intid//__init__.py 2012-07-03
11:56:11.576511518 +0200
+++ zope.intid-cykooz/src/zope/intid//__init__.py 2012-07-03
11:55:17.261865415 +0200
@@ -19,12 +19,14 @@
This functionality can be used in cataloging.
"""
import random
+import threading
+from weakref import WeakKeyDictionary, WeakSet

import BTrees
from persistent import Persistent
from zope.component import adapter, getAllUtilitiesRegisteredFor,
subscribers
from zope.event import notify
-from zope.interface import implementer
+from zope.interface import implements
from zope.keyreference.interfaces import IKeyReference, NotYet
from zope.lifecycleevent.interfaces import IObjectAddedEvent
from zope.lifecycleevent.interfaces import IObjectRemovedEvent
@@ -32,16 +34,16 @@
from zope.location.interfaces import IContained
from zope.security.proxy import removeSecurityProxy

-from zope.intid.interfaces import IIntIds, IIntIdEvent
+from zope.intid.interfaces import IIntIds, IIntIdEvent, IIntIdsDisabled
from zope.intid.interfaces import IntIdAddedEvent, IntIdRemovedEvent

- [at] implemente(IIntIds, IContained)
class IntIds(Persistent):
"""This utility provides a two way mapping between objects and
integer ids.

IKeyReferences to objects are stored in the indexes.
"""
+ implements(IIntIds, IContained)

__parent__ = __name__ = None

@@ -136,6 +138,10 @@
del self.ids[key]


+thread_data = threading.local()
+thread_data.deferred_objects = WeakKeyDictionary()
+
+
@adapter(ILocation, IObjectRemovedEvent)
def removeIntIdSubscriber(ob, event):
"""A subscriber to ObjectRemovedEvent
@@ -143,9 +149,22 @@
Removes the unique ids registered for the object in all the unique
id utilities.
"""
+
utilities = tuple(getAllUtilitiesRegisteredFor(IIntIds))
if utilities:
- key = IKeyReference(ob, None)
+ try:
+ key = IKeyReference(ob, None)
+ except NotYet:
+ deferred_objects = thread_data.deferred_objects
+ if ob in deferred_objects:
+ del deferred_objects[ob]
+ parent = getattr(ob, '__parent__', None)
+ if parent in deferred_objects and ob in
deferred_objects[parent]:
+ deferred_objects[parent].remove(ob)
+ if len(deferred_objects[parent]) == 0:
+ del deferred_objects[parent]
+ return
+
# Register only objects that adapt to key reference
if key is not None:
# Notify the catalogs that this object is about to be removed.
@@ -156,6 +175,7 @@
except KeyError:
pass

+
@adapter(ILocation, IObjectAddedEvent)
def addIntIdSubscriber(ob, event):
"""A subscriber to ObjectAddedEvent
@@ -163,16 +183,46 @@
Registers the object added in all unique id utilities and fires
an event for the catalogs.
"""
+
utilities = tuple(getAllUtilitiesRegisteredFor(IIntIds))
if utilities: # assert that there are any utilites
+ register_object(ob, utilities, event)
+
+
+def register_object(ob, utilities, event):
+ deferred_objects = thread_data.deferred_objects
+ intids_enabled = not IIntIdsDisabled.providedBy(ob)
+ try:
key = IKeyReference(ob, None)
- # Register only objects that adapt to key reference
- if key is not None:
- idmap = {}
- for utility in utilities:
- idmap[utility] = utility.register(key)
- # Notify the catalogs that this object was added.
- notify(IntIdAddedEvent(ob, event, idmap))
+ except NotYet:
+ if intids_enabled:
+ parent = getattr(ob, '__parent__', None)
+ if parent is None:
+ raise
+ if parent not in deferred_objects:
+ deferred_objects[parent] = WeakSet()
+ deferred_objects[parent].add(ob)
+ return
+
+ # Register only objects that adapt to key reference
+ if key is None:
+ return
+
+ # Register the current object if it is enabled
+ if intids_enabled:
+ idmap = {}
+ for utility in utilities:
+ idmap[utility] = utility.register(key)
+ # Notify the catalogs that this object was added.
+ notify(IntIdAddedEvent(ob, event, idmap))
+
+ # Register the deferred children of the current object
+ if ob in deferred_objects:
+ children = deferred_objects.pop(ob)
+ for child in children:
+ if child.__parent__ is ob:
+ register_object(child, utilities, event)
+

@adapter(IIntIdEvent)
def intIdEventNotify(event):
diff -u zope.intid/trunk/src/zope/intid//interfaces.py
zope.intid-cykooz/src/zope/intid//interfaces.py
--- zope.intid/trunk/src/zope/intid//interfaces.py 2012-07-03
11:56:11.576511518 +0200
+++ zope.intid-cykooz/src/zope/intid//interfaces.py 2012-07-03
11:55:17.261865415 +0200
@@ -1,6 +1,6 @@
"""Interfaces for the unique id utility.
"""
-from zope.interface import Interface, Attribute, implementer
+from zope.interface import Interface, Attribute, implements


class IIntIdsQuery(Interface):
@@ -61,6 +61,10 @@
"""


+class IIntIdsDisabled(Interface):
+ """ Marker for objects that should not be indexed. """
+
+
class IIntIdEvent(Interface):
"""Generic base interface for IntId-related events"""

@@ -77,12 +81,13 @@
"""


- [at] implemente(IIntIdRemovedEvent)
class IntIdRemovedEvent(object):
"""The event which is published before the unique id is removed
from the utility so that the catalogs can unindex the object.
"""

+ implements(IIntIdRemovedEvent)
+
def __init__(self, object, event):
self.object = object
self.original_event = event
@@ -98,12 +103,13 @@
idmap = Attribute("The dictionary that holds an (utility -> id)
mapping of created ids")


- [at] implemente(IIntIdAddedEvent)
class IntIdAddedEvent(object):
"""The event which gets sent when an object is registered in a
unique id utility.
"""

+ implements(IIntIdAddedEvent)
+
def __init__(self, object, event, idmap=None):
self.object = object
self.original_event = event
Only in zope.intid/trunk/src/zope/intid/: .svn
diff -u zope.intid/trunk/src/zope/intid//tests.py
zope.intid-cykooz/src/zope/intid//tests.py
--- zope.intid/trunk/src/zope/intid//tests.py 2012-07-03
11:56:11.576511518 +0200
+++ zope.intid-cykooz/src/zope/intid//tests.py 2012-07-03
11:55:17.261865415 +0200
@@ -13,6 +13,7 @@
##############################################################################
"""Tests for the unique id utility.
"""
+import gc
import random
import unittest

@@ -25,23 +26,25 @@
from zope.component import provideHandler
from zope.component import testing, eventtesting
from zope.component.interfaces import ISite, IComponentLookup
-from zope.interface import implementer, Interface
+from zope.container.interfaces import ISimpleReadContainer
+from zope.container.traversal import ContainerTraversable
+from zope.interface import implements, Interface
+from zope.interface.declarations import directlyProvides
from zope.interface.verify import verifyObject
from zope.keyreference.persistent import KeyReferenceToPersistent
from zope.keyreference.persistent import connectionOfPersistent
from zope.keyreference.interfaces import IKeyReference
+from zope.lifecycleevent import ObjectAddedEvent
from zope.location.interfaces import ILocation
from zope.site.hooks import setSite, setHooks, resetHooks
-from zope.site.folder import rootFolder
+from zope.site.folder import rootFolder, Folder
from zope.site.site import SiteManagerAdapter, LocalSiteManager
from zope.traversing import api
from zope.traversing.testing import setUp as traversingSetUp
from zope.traversing.interfaces import ITraversable
-from zope.container.traversal import ContainerTraversable
-from zope.container.interfaces import ISimpleReadContainer

-from zope.intid import IntIds, intIdEventNotify
-from zope.intid.interfaces import IIntIds
+from zope.intid import IntIds, intIdEventNotify, addIntIdSubscriber
+from zope.intid.interfaces import IIntIds, IIntIdsDisabled


# Local Utility Addition
@@ -67,9 +70,8 @@
return api.traverse(folder, "++etc++site")


- [at] implemente(ILocation)
class P(Persistent):
- pass
+ implements(ILocation)


class ConnectionStub(object):
@@ -101,9 +103,9 @@
self.root = rootFolder()
createSiteManager(self.root, setsite=True)

- provideAdapter(connectionOfPersistent, (IPersistent, ),
IConnection)
+ provideAdapter(connectionOfPersistent, (IPersistent,), IConnection)
provideAdapter(
- KeyReferenceToPersistent, (IPersistent, ), IKeyReference)
+ KeyReferenceToPersistent, (IPersistent,), IKeyReference)

def tearDown(self):
resetHooks()
@@ -159,14 +161,14 @@
# behaviour of the _generateId method
u = self.createIntIds()
maxint = u.family.maxint
- u._randrange = lambda x,y: maxint-1
+ u._randrange = lambda x, y: maxint - 1

conn = ConnectionStub()

obj = P()
conn.add(obj)
uid = u.register(obj)
- self.assertEquals(maxint-1, uid)
+ self.assertEquals(maxint - 1, uid)
self.assertEquals(maxint, u._v_nextid)

# The next chosen int is exactly the largest number possible
that is
@@ -243,7 +245,6 @@
class TestSubscribers(ReferenceSetupMixin, unittest.TestCase):

def setUp(self):
- from zope.site.folder import Folder

ReferenceSetupMixin.setUp(self)

@@ -298,8 +299,6 @@
self.assertEquals(objevents[0][1].original_event.object,
parent_folder)

def test_addIntIdSubscriber(self):
- from zope.lifecycleevent import ObjectAddedEvent
- from zope.intid import addIntIdSubscriber
from zope.intid.interfaces import IIntIdAddedEvent
from zope.site.interfaces import IFolder
parent_folder = self.root['folder1']['folder1_1']
@@ -344,11 +343,63 @@
return IntIds(family=BTrees.family64)


+class TestNotYetFix(ReferenceSetupMixin, unittest.TestCase):
+
+ def setUp(self):
+ ReferenceSetupMixin.setUp(self)
+ sm = getSiteManager(self.root)
+ self.utility = addUtility(sm, '1', IIntIds, IntIds())
+ provideHandler(intIdEventNotify)
+ self.root._p_jar = ConnectionStub()
+ setSite(self.root)
+
+ def test_deferred_indexes(self):
+ folder1 = Folder()
+ folder2 = Folder()
+
+ folder1['folder2'] = folder2
+ addIntIdSubscriber(folder2, ObjectAddedEvent(folder1))
+ self.assertRaises(KeyError, self.utility.getId, folder2)
+
+ self.root['folder1'] = folder1
+ addIntIdSubscriber(folder1, ObjectAddedEvent(self.root))
+ self.assertIsNotNone(self.utility.getId(folder1))
+ self.assertIsNotNone(self.utility.getId(folder2))
+
+ def test_clear_deferred_objects(self):
+ from zope.intid import thread_data
+
+ folder1 = Folder()
+ folder2 = Folder()
+ folder1['folder2'] = folder2
+
+ addIntIdSubscriber(folder2, ObjectAddedEvent(folder1))
+ eventtesting.clearEvents()
+
+ self.assertRaises(KeyError, self.utility.getId, folder2)
+ self.assertTrue(len(thread_data.deferred_objects) > 0)
+
+ del folder2
+ del folder1
+
+ gc.collect()
+
+ self.assertTrue(len(thread_data.deferred_objects) == 0)
+
+ def test_IIntIdsDisabled(self):
+ folder = Folder()
+ directlyProvides(folder, IIntIdsDisabled)
+ self.root['folder'] = folder
+ addIntIdSubscriber(folder, ObjectAddedEvent(self.root))
+ self.assertRaises(KeyError, self.utility.getId, folder)
+
+
def test_suite():
suite = unittest.TestSuite()
suite.addTest(unittest.makeSuite(TestIntIds))
suite.addTest(unittest.makeSuite(TestIntIds64))
suite.addTest(unittest.makeSuite(TestSubscribers))
+ suite.addTest(unittest.makeSuite(TestNotYetFix))
return suite

if __name__ == '__main__':





_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


hanno at hannosch

Jul 3, 2012, 6:06 AM

Post #9 of 11 (471 views)
Permalink
Re: zope.intid and zope.keyreference.interfaces.NotYet exception (patch) [In reply to]

On Tue, Jul 3, 2012 at 1:56 PM, Jan-Wijbrand Kolman
<janwijbrand [at] gmail> wrote:
> At the end of this post, I pasted the diff from the current zope.intid
> trunk against your "fork" on bitbucket. Maybe this would make it easier
> for others to comment on it?

It would be easier to read if you did the diff against the base
version the fork was started from. Or otherwise update the fork with
the changes done in the meantime.

I see a bunch of unrelated changes in there, like the
implements/implementer changes. I'm not sure what else is unrelated to
the proposed change.

Hanno
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


janwijbrand at gmail

Jul 4, 2012, 2:47 AM

Post #10 of 11 (463 views)
Permalink
Re: zope.intid and zope.keyreference.interfaces.NotYet exception (patch) [In reply to]

On 7/3/12 15:06 , Hanno Schlichting wrote:
> On Tue, Jul 3, 2012 at 1:56 PM, Jan-Wijbrand Kolman
> <janwijbrand [at] gmail> wrote:
>> At the end of this post, I pasted the diff from the current zope.intid
>> trunk against your "fork" on bitbucket. Maybe this would make it easier
>> for others to comment on it?
>
> It would be easier to read if you did the diff against the base
> version the fork was started from. Or otherwise update the fork with
> the changes done in the meantime.
>
> I see a bunch of unrelated changes in there, like the
> implements/implementer changes. I'm not sure what else is unrelated to
> the proposed change.

Ow, well, yes you are right. I didn't investigate too deeply either and
I do not know the revision the fork was based on. Maybe I just didn't
look hard enough.

I just wanted to see if we could get a little feedback by posting the
diff over here, instead for others to have to go through this just see
what the proposed changes are. Being the messenger so to say.

Did you stop looking at the diff once you noticed it was created
suboptimally? Would it help to find out the base revision and post a
diff again? I guess Cykooz would know the revision of the common base?

regards, jw



_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


cykooz at gmail

Jul 4, 2012, 3:03 AM

Post #11 of 11 (466 views)
Permalink
Re: zope.intid and zope.keyreference.interfaces.NotYet exception (patch) [In reply to]

2012/7/4 Jan-Wijbrand Kolman <janwijbrand [at] gmail>

> I guess Cykooz would know the revision of the common base?
>
> regards, jw
>

I do not remember the revision. But it certainly was from trunk branch. You
can see the old revisions from SVN and my new here -
https://bitbucket.org/cykooz/zope.intid/changesets

Zope dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.