Mailing List Archive: Python: Bugs

[issue15881] multiprocessing 'NoneType' object is not callable

Index | Next | Previous | View Threaded

report at bugs

Sep 7, 2012, 11:56 AM

Post #1 of 18 (351 views)
Permalink

[issue15881] multiprocessing 'NoneType' object is not callable

New submission from Chris McDonough:

The symptom is an exact duplicate of the symptom reported in the following (closed) issue:

http://bugs.python.org/issue9775

The issue is also related to the following other issues:

http://bugs.python.org/issue4106
http://bugs.python.org/issue9205
http://bugs.python.org/issue9207

To reproduce the symptom driving the patches that will be attached to this issue:

git clone git://github.com/pypa/pip.git
cd pip
/any/python setup.py dev
/any/python setup.py test

You can either wait for the entire test suite to finish or you can press ctrl-C at any time (the tests take a long time). In either case, a traceback like the following will be printed to the console.

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/home/chrism/opt/Python-2.7.3/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/home/chrism/opt/Python-2.7.3/lib/python2.7/multiprocessing/util.py", line 284, in _exit_function
info('process shutting down')
TypeError: 'NoneType' object is not callable
Error in sys.exitfunc:
Traceback (most recent call last):
File "/home/chrism/opt/Python-2.7.3/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/home/chrism/opt/Python-2.7.3/lib/python2.7/multiprocessing/util.py", line 284, in _exit_function
info('process shutting down')
TypeError: 'NoneType' object is not callable

>From what I understand in other issues, multiprocessing.util._exit_function shouldn't actually be called *after* the containing module's globals are destroyed (it should be called before), but this does indeed happen.

Patches will be attached that anticipate the symptom and prevent a shutdown error. One will be attached for Python 2.7 branch, the other for the Python tip. Each causes functions that are called at shutdown time to keep a reference around to other functions and globals used within the function, and each does some checks for the insane state and prevents an error from happening in this insane state.

----------
components: Library (Lib)
messages: 170003
nosy: mcdonc
priority: normal
severity: normal
status: open
title: multiprocessing 'NoneType' object is not callable
type: behavior
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15881>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Sep 7, 2012, 12:13 PM

Post #2 of 18 (322 views)
Permalink

[issue15881] multiprocessing 'NoneType' object is not callable [In reply to]

Changes by Richard Oudkerk <shibturn [at] gmail>:

----------
nosy: +sbt

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15881>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Sep 7, 2012, 12:15 PM

Post #3 of 18 (327 views)
Permalink

[issue15881] multiprocessing 'NoneType' object is not callable [In reply to]

Chris McDonough added the comment:

Patch for tip.

----------
keywords: +patch
Added file: http://bugs.python.org/file27142/shutdown_typeerror-tip.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15881>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Sep 7, 2012, 12:23 PM

Post #4 of 18 (318 views)
Permalink

[issue15881] multiprocessing 'NoneType' object is not callable [In reply to]

Chris McDonough added the comment:

2.7 branch patch.

----------
Added file: http://bugs.python.org/file27143/shutdown_typeerror-27.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15881>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Sep 7, 2012, 12:40 PM

Post #5 of 18 (315 views)
Permalink

[issue15881] multiprocessing 'NoneType' object is not callable [In reply to]

Changes by Mark Dickinson <dickinsm [at] gmail>:

----------
stage: -> patch review
versions: -Python 2.6, Python 3.1

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15881>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Sep 7, 2012, 9:24 PM

Post #6 of 18 (316 views)
Permalink

[issue15881] multiprocessing 'NoneType' object is not callable [In reply to]

Alexander Belopolsky added the comment:

The patch makes sense. I'll take another look over the weekend, but it seems to be ready to be applied.

----------
assignee: -> belopolsky
nosy: +belopolsky
stage: patch review -> commit review

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15881>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Sep 7, 2012, 10:21 PM

Post #7 of 18 (313 views)
Permalink

[issue15881] multiprocessing 'NoneType' object is not callable [In reply to]

Chris Jerdonek added the comment:

+ # NB: we hold on to references to functions in the arglist due to the

This is a nit, but I think adding "NB:", "Note:", etc. to the beginning of a comment is redundant because by being a comment it is already implicit that it should be noted.

----------
nosy: +cjerdonek

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15881>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Sep 9, 2012, 10:28 AM

Post #8 of 18 (315 views)
Permalink

[issue15881] multiprocessing 'NoneType' object is not callable [In reply to]

Roundup Robot added the comment:

New changeset 27d410dd5431 by Alexander Belopolsky in branch '3.2':
Issue #15881: Fixed atexit hook in multiprocessing.
http://hg.python.org/cpython/rev/27d410dd5431

New changeset 08c680918ff8 by Alexander Belopolsky in branch 'default':
Issue #15881: Fixed atexit hook in multiprocessing.
http://hg.python.org/cpython/rev/08c680918ff8

----------
nosy: +python-dev

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15881>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Sep 9, 2012, 10:31 AM

Post #9 of 18 (308 views)
Permalink

[issue15881] multiprocessing 'NoneType' object is not callable [In reply to]

Roundup Robot added the comment:

New changeset db67b848ddc3 by Alexander Belopolsky in branch '3.2':
Issue #15881: Fixed 3.2 backport.
http://hg.python.org/cpython/rev/db67b848ddc3

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15881>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Sep 9, 2012, 10:55 AM

Post #10 of 18 (307 views)
Permalink

[issue15881] multiprocessing 'NoneType' object is not callable [In reply to]

Alexander Belopolsky added the comment:

Applied to 3.2 and 3.3. Thanks for the patch!

Leaving it open pending 2.7 commit.

----------
resolution: -> fixed
stage: commit review -> committed/rejected

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15881>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Sep 9, 2012, 11:12 AM

Post #11 of 18 (308 views)
Permalink

[issue15881] multiprocessing 'NoneType' object is not callable [In reply to]

Roundup Robot added the comment:

New changeset b05547e8ff92 by Alexander Belopolsky in branch '3.2':
Issue #15881: Added NEWS entry and proper credit.
http://hg.python.org/cpython/rev/b05547e8ff92

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15881>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Sep 10, 2012, 6:07 AM

Post #12 of 18 (308 views)
Permalink

[issue15881] multiprocessing 'NoneType' object is not callable [In reply to]

Richard Oudkerk added the comment:

I see the same error on Windows (when pressing ^C), but on Linux I get

Error in sys.exitfunc:
Traceback (most recent call last):
File "/usr/lib/python2.7/atexit.py", line 28, in _run_exitfuncs
import traceback
File "/usr/lib/python2.7/traceback.py", line 3, in <module>
import linecache
File "/usr/lib/python2.7/linecache.py", line 9, in <module>
import os
File "/usr/lib/python2.7/os.py", line 119, in <module>
sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'

This also suggests that module teardown has begun before/while sys.exitfunc is running.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15881>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Sep 10, 2012, 11:41 AM

Post #13 of 18 (309 views)
Permalink

[issue15881] multiprocessing 'NoneType' object is not callable [In reply to]

Richard Oudkerk added the comment:

I suspect the problem is caused by nose's isolate plugin.

With this enabled, a copy of sys.modules is saved before each test and then restored after the test. This causes garbage collection of newly imported modules. The destructor for the module type causes all globals to be replaced by None.

This will break the atexit function registered by multiprocessing since it depends on globals.

PS. A simple work-around (which does not require people to upgrade to a bugfixed version of Python) is to put

try:
import multiprocessing
except ImportError:
pass

near the beginning of setup.py. After this change I don't get the error when running "python setup.py test".

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15881>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Sep 10, 2012, 12:23 PM

Post #14 of 18 (301 views)
Permalink

[issue15881] multiprocessing 'NoneType' object is not callable [In reply to]

Richard Oudkerk added the comment:

Actually, I am not so sure it is the isolate plugin. But I do think that sys.modules is being manipulated somewhere before shutdown.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15881>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Sep 10, 2012, 1:22 PM

Post #15 of 18 (303 views)
Permalink

[issue15881] multiprocessing 'NoneType' object is not callable [In reply to]

Richard Oudkerk added the comment:

Actually it is test.with_project_on_sys_path() in setuptools/commands/test.py that does the save/restore of sys.modules. See

http://www.eby-sarna.com/pipermail/peak/2010-May/003357.html

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15881>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Sep 10, 2012, 3:41 PM

Post #16 of 18 (301 views)
Permalink

[issue15881] multiprocessing 'NoneType' object is not callable [In reply to]

Changes by Arfrever Frehtes Taifersar Arahesis <Arfrever.FTA [at] GMail>:

----------
nosy: +Arfrever

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15881>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Sep 12, 2012, 1:48 AM

Post #17 of 18 (300 views)
Permalink

[issue15881] multiprocessing 'NoneType' object is not callable [In reply to]

Changes by Mitar <mmitar [at] gmail>:

----------
nosy: +mitar

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15881>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Sep 13, 2012, 9:28 AM

Post #18 of 18 (299 views)
Permalink

[issue15881] multiprocessing 'NoneType' object is not callable [In reply to]

Roundup Robot added the comment:

New changeset 2b79b4848f44 by Richard Oudkerk in branch 'default':
Issue #15881: Clarify comment in exit function
http://hg.python.org/cpython/rev/2b79b4848f44

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15881>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads