Mailing List Archive: Python: Bugs

[issue7060] test_multiprocessing dictionary changed size errors and hang

Index | Next | Previous | View Threaded

report at bugs

Nov 12, 2009, 6:37 PM

Post #1 of 20 (587 views)
Permalink

[issue7060] test_multiprocessing dictionary changed size errors and hang

Benjamin Peterson <benjamin [at] python> added the comment:

If this is not ported by the time we get to a 2.7 or 3.2 release, I'd
like to back it out.

----------
priority: normal -> release blocker

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7060>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 13, 2009, 1:16 AM

Post #2 of 20 (562 views)
Permalink

[issue7060] test_multiprocessing dictionary changed size errors and hang [In reply to]

Kristján Valur Jónsson <kristjan [at] ccpgames> added the comment:

I can easily port it again, and remove the traceback from the exception.
But won't we be sweeping that GC problem under the carpet then?

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7060>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 13, 2009, 7:04 AM

Post #3 of 20 (558 views)
Permalink

[issue7060] test_multiprocessing dictionary changed size errors and hang [In reply to]

Antoine Pitrou <pitrou [at] free> added the comment:

Please note there's a patch in issue7105 to allow safer iteration of
weak dicts.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7060>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 13, 2009, 8:11 AM

Post #4 of 20 (552 views)
Permalink

[issue7060] test_multiprocessing dictionary changed size errors and hang [In reply to]

Kristján Valur Jónsson <kristjan [at] ccpgames> added the comment:

Ok, I ported the change again in revision 76238, removing the traceback
from the stored exception object.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7060>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 13, 2009, 8:12 AM

Post #5 of 20 (554 views)
Permalink

[issue7060] test_multiprocessing dictionary changed size errors and hang [In reply to]

Antoine Pitrou <pitrou [at] free> added the comment:

> Kristján Valur Jónsson <kristjan [at] ccpgames> added the comment:
>
> Ok, I ported the change again in revision 76238, removing the traceback
> from the stored exception object.

"versionchanged" should be 3.2, not 3.1.
Thanks!

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7060>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 13, 2009, 8:16 AM

Post #6 of 20 (564 views)
Permalink

[issue7060] test_multiprocessing dictionary changed size errors and hang [In reply to]

Kristján Valur Jónsson <kristjan [at] ccpgames> added the comment:

Thanks. Fixed in revision 76239.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7060>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 13, 2009, 9:04 AM

Post #7 of 20 (551 views)
Permalink

[issue7060] test_multiprocessing dictionary changed size errors and hang [In reply to]

Antoine Pitrou <pitrou [at] free> added the comment:

I don't know if it's your last change, but test_multiprocessing is now
hanging on all buildbots.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7060>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 13, 2009, 9:26 AM

Post #8 of 20 (553 views)
Permalink

[issue7060] test_multiprocessing dictionary changed size errors and hang [In reply to]

R. David Murray <rdmurray [at] bitdance> added the comment:

It is. TestZZZNumberOfObjects hangs at r76238, and does not hang at
r76239 on my Gentoo Linux box.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7060>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 13, 2009, 9:34 AM

Post #9 of 20 (552 views)
Permalink

[issue7060] test_multiprocessing dictionary changed size errors and hang [In reply to]

Changes by R. David Murray <rdmurray [at] bitdance>:

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7060>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 13, 2009, 9:34 AM

Post #10 of 20 (551 views)
Permalink

[issue7060] test_multiprocessing dictionary changed size errors and hang [In reply to]

R. David Murray <rdmurray [at] bitdance> added the comment:

It is. TestZZZNumberOfObjects hangs at r76237, and does not hang at
r76238 on my Gentoo Linux box.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7060>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 13, 2009, 9:35 AM

Post #11 of 20 (552 views)
Permalink

[issue7060] test_multiprocessing dictionary changed size errors and hang [In reply to]

report at bugs

Nov 13, 2009, 9:35 AM

Post #12 of 20 (552 views)
Permalink

[issue7060] test_multiprocessing dictionary changed size errors and hang [In reply to]

R. David Murray <rdmurray [at] bitdance> added the comment:

It is. TestZZZNumberOfObjects hangs at r76238, and does not hang at
r76237 on my Gentoo Linux box.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7060>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 13, 2009, 10:04 AM

Post #13 of 20 (554 views)
Permalink

[issue7060] test_multiprocessing dictionary changed size errors and hang [In reply to]

Antoine Pitrou <pitrou [at] free> added the comment:

It seems the error is different, though. This is what I get after Ctrl-C
(I omit the tracebacks from the other processes):

Traceback (most recent call last):
File "/home/antoine/py3k/__svn__/Lib/test/test_multiprocessing.py",
line 1071, in test_number_of_objects
refs = self.manager._number_of_objects()
File "Z:\py3k\__svn__\lib\multiprocessing\managers.py", line 569, in
_number_of_objects
conn = self._Client(self._address, authkey=self._authkey)
File "Z:\py3k\__svn__\lib\multiprocessing\connection.py", line 134, in
Client
c = SocketClient(address)
File "Z:\py3k\__svn__\lib\multiprocessing\connection.py", line 253, in
SocketClient
s.connect(address)
socket.error: [Errno 2] No such file or directory

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7060>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 13, 2009, 11:04 AM

Post #14 of 20 (550 views)
Permalink

[issue7060] test_multiprocessing dictionary changed size errors and hang [In reply to]

Antoine Pitrou <pitrou [at] free> added the comment:

Ok, the problem is that a finalizer calls _decref which tries to open an
XMLRPC connection (!) to an address on which nobody listens anymore.
Then stupid SocketClient (in multiprocessing.connection) loops endlessly
because it is designed to do so when it gets ECONNREFUSED.

(when you Ctrl-C, you don't see all this because it happens in a
finalizer, and only a terse message is displayed)

So there are two problems:
- why does test_multiprocessing think it should open an XMLRPC
connection to a closed endpoint
- why doesn't SocketClient have some kind of timeout rather than looping
stupidly

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7060>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 13, 2009, 11:29 AM

Post #15 of 20 (548 views)
Permalink

[issue7060] test_multiprocessing dictionary changed size errors and hang [In reply to]

Changes by Antoine Pitrou <pitrou [at] free>:

----------
dependencies: +multiprocessing should not wait endlessly

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7060>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 13, 2009, 1:39 PM

Post #16 of 20 (538 views)
Permalink

[issue7060] test_multiprocessing dictionary changed size errors and hang [In reply to]

Kristján Valur Jónsson <kristjan [at] ccpgames> added the comment:

Yes, this is quite different.
After seeing this, I created issue 7314.
I didn't realize that it had anything to do with my patch but rather
blamed it on my brand new windows 7.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7060>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 13, 2009, 1:41 PM

Post #17 of 20 (539 views)
Permalink

[issue7060] test_multiprocessing dictionary changed size errors and hang [In reply to]

Antoine Pitrou <pitrou [at] free> added the comment:

> Yes, this is quite different.
> After seeing this, I created issue 7314.
> I didn't realize that it had anything to do with my patch but rather
> blamed it on my brand new windows 7.

I don't think it has anything to do with your patch. While diagnosing
the problem, I noticed that adding some random print() lines would
sometimes make the problem disappear. So it is really timing-dependent
and your patch happens to trigger the right (or wrong ;-)) timing.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7060>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 13, 2009, 1:42 PM

Post #18 of 20 (542 views)
Permalink

[issue7060] test_multiprocessing dictionary changed size errors and hang [In reply to]

Kristján Valur Jónsson <kristjan [at] ccpgames> added the comment:

See my analysis from issue 7314:
The other processes are all waiting to ReadFile(). Maybe they have yet
to run to the point where they open the socket to which the master
process is trying to connect to.

The mystery is also: Why does the harmless act of setting an instance
attribute on a context manager (and without any traceback) trigger this
behaviour?

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7060>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 13, 2009, 1:44 PM

Post #19 of 20 (540 views)
Permalink

[issue7060] test_multiprocessing dictionary changed size errors and hang [In reply to]

Kristján Valur Jónsson <kristjan [at] ccpgames> added the comment:

Ah, okay then. Let's focus on solving 7414 then.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7060>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 17, 2009, 2:07 AM

Post #20 of 20 (486 views)
Permalink

[issue7060] test_multiprocessing dictionary changed size errors and hang [In reply to]

Antoine Pitrou <pitrou [at] free> added the comment:

I don't think this is a blocker anymore. Kristján's patch has been
ported and the test_multiprocessing freezes have been "fixed" by adding
a timeout.
As for the original purpose of this ticket, issue7105 is still (part
of?) the answer.

----------
priority: release blocker -> normal

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7060>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads