Mailing List Archive: Zope: Dev

ZODB Competing read/writes: How to find out which attribute?

Index | Next | Previous | View Threaded

dusty at qwer

Oct 9, 2009, 9:10 AM

Post #1 of 9 (1153 views)
Permalink

ZODB Competing read/writes: How to find out which attribute?

Hi,
I once in the while get the following warning in my Zope 3 log, which I'd like
to resolve:

2009-10-07T14:35:41 WARNING ZopePublication Competing writes/reads
at /BSPSite/act/++vh++http:zis.act.at:80/bankneu/++/c/acc/booklist/index.html:
database conflict error (oid 0x7a5d, class BTrees.OOBTree.OOBTree, serial
this txn started with 0x0381388ca6804966 2009-10-07 12:28:39.023700, serial
currently committed 0x03813893b087d0cc 2009-10-07 12:35:41.374343)

I wonder how I can find out, to which attribute this applies, I do know the
view and the class, probably BTrees.OOBTree.OOBTree, but don't know the
attribute itself.

Any clues?

Best Regards,
Hermann

--
hermann [at] qwer
GPG key ID: 299893C7 (on keyservers)
FP: 0124 2584 8809 EF2A DBF9 4902 64B4 D16B 2998 93C7
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )

shane at hathawaymix

Oct 9, 2009, 10:12 AM

Post #2 of 9 (1101 views)
Permalink

Re: ZODB Competing read/writes: How to find out which attribute? [In reply to]

Hermann Himmelbauer wrote:
> Hi,
> I once in the while get the following warning in my Zope 3 log, which I'd like
> to resolve:
>
> 2009-10-07T14:35:41 WARNING ZopePublication Competing writes/reads
> at /BSPSite/act/++vh++http:zis.act.at:80/bankneu/++/c/acc/booklist/index.html:
> database conflict error (oid 0x7a5d, class BTrees.OOBTree.OOBTree, serial
> this txn started with 0x0381388ca6804966 2009-10-07 12:28:39.023700, serial
> currently committed 0x03813893b087d0cc 2009-10-07 12:35:41.374343)

That URL looks like it should only be reading the database, but the fact
that you get a conflict error suggests that the code behind that URL is
writing to the database. If I were you, I would first ensure that read
operations never write to the database.

Shane

_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )

dev at projekt01

Oct 9, 2009, 12:44 PM

Post #3 of 9 (1089 views)
Permalink

Re: ZODB Competing read/writes: How to find out which attribute? [In reply to]

Hi Hermann

> Betreff: Re: [Zope-dev] ZODB Competing read/writes: How to
> find out which attribute?
>
> Hermann Himmelbauer wrote:
> > Hi,
> > I once in the while get the following warning in my Zope 3
> log, which
> > I'd like to resolve:
> >
> > 2009-10-07T14:35:41 WARNING ZopePublication Competing
> writes/reads at
> >
> /BSPSite/act/++vh++http:zis.act.at:80/bankneu/++/c/acc/booklis
t/index.html:
> > database conflict error (oid 0x7a5d, class BTrees.OOBTree.OOBTree,
> > serial this txn started with 0x0381388ca6804966 2009-10-07
> > 12:28:39.023700, serial currently committed 0x03813893b087d0cc
> > 2009-10-07 12:35:41.374343)
>
> That URL looks like it should only be reading the database,
> but the fact that you get a conflict error suggests that the
> code behind that URL is writing to the database. If I were
> you, I would first ensure that read operations never write to
> the database.

just a hint, sometimes session get accessed with session[key]
wich will force a write operation. If so, you can prevent creating
a new session by using session.get(key, someDefaultValue)

Regards
Roger Ineichen

> Shane
>
> _______________________________________________
> Zope-Dev maillist - Zope-Dev [at] zope
> https://mail.zope.org/mailman/listinfo/zope-dev
> ** No cross posts or HTML encoding! ** (Related lists -
> https://mail.zope.org/mailman/listinfo/zope-announce
> https://mail.zope.org/mailman/listinfo/zope )
>

_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )

dusty at qwer

Oct 10, 2009, 1:48 AM

Post #4 of 9 (1088 views)
Permalink

Re: ZODB Competing read/writes: How to find out which attribute? [In reply to]

Am Freitag 09 Oktober 2009 21:44:12 schrieb Roger Ineichen:
> Hi Hermann
>
> > Betreff: Re: [Zope-dev] ZODB Competing read/writes: How to
> > find out which attribute?
> >
> > Hermann Himmelbauer wrote:
> > > Hi,
> > > I once in the while get the following warning in my Zope 3
> >
> > log, which
> >
> > > I'd like to resolve:
> > >
> > > 2009-10-07T14:35:41 WARNING ZopePublication Competing
> >
> > writes/reads at
> >
> > /BSPSite/act/++vh++http:zis.act.at:80/bankneu/++/c/acc/booklis
>
> t/index.html:
> > > database conflict error (oid 0x7a5d, class BTrees.OOBTree.OOBTree,
> > > serial this txn started with 0x0381388ca6804966 2009-10-07
> > > 12:28:39.023700, serial currently committed 0x03813893b087d0cc
> > > 2009-10-07 12:35:41.374343)
> >
> > That URL looks like it should only be reading the database,
> > but the fact that you get a conflict error suggests that the
> > code behind that URL is writing to the database. If I were
> > you, I would first ensure that read operations never write to
> > the database.
>
> just a hint, sometimes session get accessed with session[key]
> wich will force a write operation. If so, you can prevent creating
> a new session by using session.get(key, someDefaultValue)

Thanks for the info - but why is that? I always had the impression, that
mydict['mykey'] does the same as mydict.get('mykey')?

What exactly is written during session['mykey']?

Moreover, does that concurrent read/write problem with sessions apply to one
user only or to anyone who accesses the session?

Best Regards,
Hermann

--
hermann [at] qwer
GPG key ID: 299893C7 (on keyservers)
FP: 0124 2584 8809 EF2A DBF9 4902 64B4 D16B 2998 93C7
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )

dusty at qwer

Oct 10, 2009, 1:56 AM

Post #5 of 9 (1086 views)
Permalink

Re: ZODB Competing read/writes: How to find out which attribute? [In reply to]

Am Freitag 09 Oktober 2009 19:12:09 schrieb Shane Hathaway:
> Hermann Himmelbauer wrote:
> > Hi,
> > I once in the while get the following warning in my Zope 3 log, which I'd
> > like to resolve:
> >
> > 2009-10-07T14:35:41 WARNING ZopePublication Competing writes/reads
> > at
> > /BSPSite/act/++vh++http:zis.act.at:80/bankneu/++/c/acc/booklist/index.htm
> >l: database conflict error (oid 0x7a5d, class BTrees.OOBTree.OOBTree,
> > serial this txn started with 0x0381388ca6804966 2009-10-07
> > 12:28:39.023700, serial currently committed 0x03813893b087d0cc 2009-10-07
> > 12:35:41.374343)
>
> That URL looks like it should only be reading the database, but the fact
> that you get a conflict error suggests that the code behind that URL is
> writing to the database. If I were you, I would first ensure that read
> operations never write to the database.

That's exactly the problem - it's a read operation and there should not be any
write operation involved. However, it's hard to find out where the write
operation in my code occurs, I can't find it in the view, maybe there's
something in the authentication code, but I'd have to dig deep through my
source. So It would be extremely helpful to know a faster way to find out
what code triggered this warning.

Best Regards,
Hermann

--
hermann [at] qwer
GPG key ID: 299893C7 (on keyservers)
FP: 0124 2584 8809 EF2A DBF9 4902 64B4 D16B 2998 93C7
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )

shane at hathawaymix

Oct 10, 2009, 7:00 PM

Post #6 of 9 (1078 views)
Permalink

Re: ZODB Competing read/writes: How to find out which attribute? [In reply to]

Hermann Himmelbauer wrote:
> That's exactly the problem - it's a read operation and there should not be any
> write operation involved. However, it's hard to find out where the write
> operation in my code occurs, I can't find it in the view, maybe there's
> something in the authentication code, but I'd have to dig deep through my
> source. So It would be extremely helpful to know a faster way to find out
> what code triggered this warning.

The code that modifies something will indirectly call the register()
method of the ZODB.Connection class. If you arrange for
Connection.register() to raise an exception when the transaction is not
supposed to modify anything, you should get a traceback that pinpoints
what is doing the modification. Unfortunately, there is no API for
that, so the easiest thing to do is modify the source code of the
register() method in ZODB.Connection. Fortunately, this is Python, so
modifying someone else's code is usually quite easy. :-)

Shane

_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )

jim at zope

Oct 11, 2009, 6:16 AM

Post #7 of 9 (1077 views)
Permalink

Re: ZODB Competing read/writes: How to find out which attribute? [In reply to]

On Sat, Oct 10, 2009 at 10:00 PM, Shane Hathaway <shane [at] hathawaymix> wrote:
> Hermann Himmelbauer wrote:
>> That's exactly the problem - it's a read operation and there should not be any
>> write operation involved. However, it's hard to find out where the write
>> operation in my code occurs, I can't find it in the view, maybe there's
>> something in the authentication code, but I'd have to dig deep through my
>> source. So It would be extremely helpful to know a faster way to find out
>> what code triggered this warning.
>
> The code that modifies something will indirectly call the register()
> method of the ZODB.Connection class. �If you arrange for
> Connection.register() to raise an exception when the transaction is not
> supposed to modify anything, you should get a traceback that pinpoints
> what is doing the modification. �Unfortunately, there is no API for
> that, so the easiest thing to do is modify the source code of the
> register() method in ZODB.Connection. �Fortunately, this is Python, so
> modifying someone else's code is usually quite easy. :-)

Easier yet, in a development environment, add a breakpoint in register
and make a read-only request to see what's causing it to be called and
why.

Jim

--
Jim Fulton
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )

marius at gedmin

Oct 11, 2009, 12:27 PM

Post #8 of 9 (1074 views)
Permalink

Re: ZODB Competing read/writes: How to find out which attribute? [In reply to]

On Sat, Oct 10, 2009 at 10:56:09AM +0200, Hermann Himmelbauer wrote:
> Am Freitag 09 Oktober 2009 19:12:09 schrieb Shane Hathaway:
> > Hermann Himmelbauer wrote:
> > > Hi,
> > > I once in the while get the following warning in my Zope 3 log, which I'd
> > > like to resolve:
> > >
> > > 2009-10-07T14:35:41 WARNING ZopePublication Competing writes/reads
> > > at
> > > /BSPSite/act/++vh++http:zis.act.at:80/bankneu/++/c/acc/booklist/index.htm
> > >l: database conflict error (oid 0x7a5d, class BTrees.OOBTree.OOBTree,
> > > serial this txn started with 0x0381388ca6804966 2009-10-07
> > > 12:28:39.023700, serial currently committed 0x03813893b087d0cc 2009-10-07
> > > 12:35:41.374343)
> >
> > That URL looks like it should only be reading the database, but the fact
> > that you get a conflict error suggests that the code behind that URL is
> > writing to the database. If I were you, I would first ensure that read
> > operations never write to the database.
>
> That's exactly the problem - it's a read operation and there should not be any
> write operation involved. However, it's hard to find out where the write
> operation in my code occurs, I can't find it in the view, maybe there's
> something in the authentication code, but I'd have to dig deep through my
> source. So It would be extremely helpful to know a faster way to find out
> what code triggered this warning.

Sometimes looking at the object in question can provide some hints. You
have the OID, start up zodbbrowser (http://pypi.python.org/pypi/zodbbrowser),
go to @@zodbbrowser?oid=0x7a5d and see if the data sheds some link.
Some things, e.g. annotations, stand out pretty clearly (and the default
adapter for IAttributeAnnotatable is one of the things that has
write-on-read semantics.)

Sadly, the ZODB doesn't track backreferences, so there only way to find
out for sure what points to your object is to do a full graph traversal.

Cheers!
Marius Gedminas
--
http://pov.lt/ -- Zope 3 consulting and development

Attachments:

signature.asc (0.19 KB)

jim at zope

Oct 12, 2009, 3:31 AM

Post #9 of 9 (1068 views)
Permalink

Re: ZODB Competing read/writes: How to find out which attribute? [In reply to]

On Sun, Oct 11, 2009 at 3:27 PM, Marius Gedminas <marius [at] gedmin> wrote:
...
> Sadly, the ZODB doesn't track backreferences, so there only way to find
> out for sure what points to your object is to do a full graph traversal.

The zc.zodbdgc package has a database verification scripts that
optionally creates a backreference database.

Jim

--
Jim Fulton
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads