Mailing List Archive: Zope: Users

zopectl does not terminate

Index | Next | Previous | View Threaded

hedley at upfrontsystems

Mar 17, 2009, 12:45 AM

Post #1 of 6 (1544 views)
Permalink

zopectl does not terminate

Hi all

I run my script foo.zctl with "zopectl run foo.ctl param1 param2".
This script operates on a large ZODB and catches ConflictErrors
accordingly. It iterates over a set, updates data and commits the
transaction every 100 iterations. But I've noticed two things:

1. ConflictErrors are never fully caught. The show up in the console
(this is acceptable I suppose), but my script stops executing on the
conflict and does not continue. The zope process stays alive.
2. In the event of no conflict errors my script executes its last line
(print 'done') but the process does not always terminate.

If I instruct my script to not update the ZODB at all it terminates
without problems. I'm running it on a live site with 7 ZEO clients.
I've stopped a client (say client 2) so it is not accessed
concurrently and run my script with client2/zopectl. It is in fact a
Plone site but that should be irrelevant.

Thanks for any help
Hedley
_______________________________________________
Zope maillist - Zope [at] zope
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

tseaver at palladion

Mar 17, 2009, 11:11 AM

Post #2 of 6 (1446 views)
Permalink

Re: zopectl does not terminate [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hedley Roos wrote:
> Hi all
>
> I run my script foo.zctl with "zopectl run foo.ctl param1 param2".
> This script operates on a large ZODB and catches ConflictErrors
> accordingly. It iterates over a set, updates data and commits the
> transaction every 100 iterations. But I've noticed two things:
>
> 1. ConflictErrors are never fully caught. The show up in the console
> (this is acceptable I suppose), but my script stops executing on the
> conflict and does not continue. The zope process stays alive.

Your script has to do the following when interrupted by a ConflictError:

- abort the current transaction
- sync its database connection (to pick up the changed objects)
- re-do the work it was trying to commit (that last 100 updates).

> 2. In the event of no conflict errors my script executes its last line
> (print 'done') but the process does not always terminate.

You may need to wait longer: a commit can get blocked for a period.

> If I instruct my script to not update the ZODB at all it terminates
> without problems. I'm running it on a live site with 7 ZEO clients.
> I've stopped a client (say client 2) so it is not accessed
> concurrently and run my script with client2/zopectl. It is in fact a
> Plone site but that should be irrelevant.

Long-running ZODB-updating scripts are pretty hard to get right. I hope
the dance outlined above helps.

Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 tseaver [at] palladion
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJv+fO+gerLs4ltQ4RAsUKAJ95OJvb/pH1hqn2DlyJNlxrP/KuEgCgvOsa
TaUIrLs0Vm7jRWjMST0zh+E=
=3kvF
-----END PGP SIGNATURE-----

_______________________________________________
Zope maillist - Zope [at] zope
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

hedleyroos at gmail

Mar 17, 2009, 12:41 PM

Post #3 of 6 (1444 views)
Permalink

Re: zopectl does not terminate [In reply to]

Hi Tres!

Thanks for the tips. I managed to get my script running in batches and
with manual intervention. When in future I encounter the same problems
I'll report back to this thread.

Hedley
_______________________________________________
Zope maillist - Zope [at] zope
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

chris at simplistix

Mar 18, 2009, 4:31 AM

Post #4 of 6 (1429 views)
Permalink

Re: zopectl does not terminate [In reply to]

Tres Seaver wrote:
> - abort the current transaction
> - sync its database connection (to pick up the changed objects)

I'm pretty sure the abort has done the sync for quite some time now...

Chris

--
Simplistix - Content Management, Zope & Python Consulting
- http://www.simplistix.co.uk
_______________________________________________
Zope maillist - Zope [at] zope
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

tseaver at palladion

Mar 18, 2009, 4:37 AM

Post #5 of 6 (1435 views)
Permalink

Re: zopectl does not terminate [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Chris Withers wrote:
> Tres Seaver wrote:
>> - abort the current transaction
>> - sync its database connection (to pick up the changed objects)
>
> I'm pretty sure the abort has done the sync for quite some time now...

Nope, see:

http://svn.zope.org/ZODB/trunk/src/ZODB/Connection.py

'sync()' calls 'self.transaction_manager.abort()', actually.

Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 tseaver [at] palladion
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJwNz7+gerLs4ltQ4RArhUAJ0fyhrGauOor35ailUIcjfRQNd9zACeOt+5
f/7g2A2lUZmxo6tyom/OFO8=
=c63W
-----END PGP SIGNATURE-----
_______________________________________________
Zope maillist - Zope [at] zope
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

chris at simplistix

Mar 23, 2009, 5:48 AM

Post #6 of 6 (1350 views)
Permalink

Re: zopectl does not terminate [In reply to]

Tres Seaver wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Chris Withers wrote:
>> Tres Seaver wrote:
>>> - abort the current transaction
>>> - sync its database connection (to pick up the changed objects)
>> I'm pretty sure the abort has done the sync for quite some time now...
>
> Nope, see:
>
> http://svn.zope.org/ZODB/trunk/src/ZODB/Connection.py
>
> 'sync()' calls 'self.transaction_manager.abort()', actually.

Well, I've never needed to call sync() in addition to abort() for a good
3 or 4 years now, an that includes when another thread changes the data
being fiddled with...

Chris

--
Simplistix - Content Management, Zope & Python Consulting
- http://www.simplistix.co.uk
_______________________________________________
Zope maillist - Zope [at] zope
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads