Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Bugs

[issue9260] A finer grained import lock

 

 

First page Previous page 1 2 Next page Last page  View All Python bugs RSS feed   Index | Next | Previous | View Threaded


report at bugs

Apr 28, 2012, 1:57 PM

Post #1 of 27 (284 views)
Permalink
[issue9260] A finer grained import lock

Antoine Pitrou <pitrou [at] free> added the comment:

Ok, here is a draft patch for the new importlib.
Several issues with this patch:
- introduces a pure Python function (_lock_unlock_module) on the fast import path
- synchronization issues due to interruptibility of pure Python code (see _ModuleLock.acquire)
- afterfork fix-up necessary
- relies on _thread.RLock for bootstrapping reasons
- module locks are immortal

----------
Added file: http://bugs.python.org/file25392/module_locks.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Apr 28, 2012, 4:17 PM

Post #2 of 27 (283 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Antoine Pitrou <pitrou [at] free> added the comment:

New patch gets rid of the reliance on _thread.RLock (uses non-recursive locks instead), and should solve the synchronization issue. Other issues remain.

----------
Added file: http://bugs.python.org/file25394/module_locks2.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Apr 28, 2012, 5:55 PM

Post #3 of 27 (280 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Antoine Pitrou <pitrou [at] free> added the comment:

Updated patch fixes the performance issue and disposes of module locks when they aren't used anymore.
Only the afterfork question remains. Should I hook in threading's own facility? Should we wait for an atfork module? Something else.

----------
Added file: http://bugs.python.org/file25398/module_locks3.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Apr 28, 2012, 5:57 PM

Post #4 of 27 (280 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Changes by Antoine Pitrou <pitrou [at] free>:


Removed file: http://bugs.python.org/file25398/module_locks3.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Apr 28, 2012, 6:00 PM

Post #5 of 27 (281 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Changes by Antoine Pitrou <pitrou [at] free>:


Added file: http://bugs.python.org/file25399/module_locks3.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

May 5, 2012, 11:08 AM

Post #6 of 27 (277 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Antoine Pitrou <pitrou [at] free> added the comment:

Updated patch also makes PyImport_ImportModuleNoBlock a simple alias of PyImport_ImportModule.

----------
Added file: http://bugs.python.org/file25466/module_locks4.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

May 5, 2012, 11:20 AM

Post #7 of 27 (276 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Changes by Antoine Pitrou <pitrou [at] free>:


Removed file: http://bugs.python.org/file25466/module_locks4.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

May 5, 2012, 11:21 AM

Post #8 of 27 (277 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Changes by Antoine Pitrou <pitrou [at] free>:


Added file: http://bugs.python.org/file25469/module_locks4.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

May 5, 2012, 11:36 AM

Post #9 of 27 (277 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Changes by Eric Snow <ericsnowcurrently [at] gmail>:


----------
nosy: +eric.snow

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

May 5, 2012, 12:03 PM

Post #10 of 27 (276 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Antoine Pitrou <pitrou [at] free> added the comment:

Updated patch also adds unit tests for the module locks and the deadlock avoidance algorithm.

----------
Added file: http://bugs.python.org/file25470/module_locks5.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

May 5, 2012, 12:16 PM

Post #11 of 27 (277 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Antoine Pitrou <pitrou [at] free> added the comment:

Updated patch with a couple new tests.

----------
stage: -> patch review
Added file: http://bugs.python.org/file25471/module_locks6.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

May 5, 2012, 12:19 PM

Post #12 of 27 (276 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Martin v. Löwis <martin [at] v> added the comment:

I still wonder whether Graham Dumpleton's observation has merits.

Suppose we have these modules

# a.py
time.sleep(10)
import b

# b.py
time.sleep(10)
import a

# main.py
def x():
import a
def y():
import b

Now, if x and y are executed in separate threads - won't it deadlock?

----------
nosy: +loewis

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

May 5, 2012, 12:21 PM

Post #13 of 27 (277 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Antoine Pitrou <pitrou [at] free> added the comment:

> Now, if x and y are executed in separate threads - won't it deadlock?

Well, the patch has a deadlock avoidance mechanism, and it includes unit
tests for precisely this situation.
I cannot promise the algorithm is perfect (although there *are* a bunch
of tests), but it looks correct from here. :)

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

May 5, 2012, 12:33 PM

Post #14 of 27 (277 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Martin v. Löwis <martin [at] v> added the comment:

Can you please elaborate in the patch what the deadlock avoidance does? AFAICT, the comment explains that it is able to detect deadlocks, but nowhere says what it does when it has detected a deadlock.

Also, please submit patches against default's head, or stop using git-style diffs, to enable Rietveld review.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

May 5, 2012, 12:50 PM

Post #15 of 27 (277 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Antoine Pitrou <pitrou [at] free> added the comment:

Updated patch against tip, and with a comment of what deadlock avoidance does (in _ModuleLock.acquire's docstring).

----------
Added file: http://bugs.python.org/file25475/module_locks7.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

May 5, 2012, 1:12 PM

Post #16 of 27 (277 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Martin v. Löwis <martin [at] v> added the comment:

The patch parser of Rietveld actually choked on the git binary diff. It now skips over these chunks.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

May 8, 2012, 6:21 AM

Post #17 of 27 (260 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Changes by Andrew Svetlov <andrew.svetlov [at] gmail>:


----------
nosy: +asvetlov

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

May 8, 2012, 6:39 AM

Post #18 of 27 (261 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Antoine Pitrou <pitrou [at] free> added the comment:

Updated patch against tip. I also changed the internal API of module locks a bit (acquire() raises _DeadlockError instead of returning False, and deadlock detection is not optional anymore).

----------
Added file: http://bugs.python.org/file25496/module_locks8.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

May 10, 2012, 9:02 AM

Post #19 of 27 (258 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Antoine Pitrou <pitrou [at] free> added the comment:

I had forgotten to tackle threadless builds, this patch fixes it.

----------
Added file: http://bugs.python.org/file25521/module_locks9.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

May 13, 2012, 1:31 PM

Post #20 of 27 (252 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Antoine Pitrou <pitrou [at] free> added the comment:

Does anyone else want to review this patch?

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

May 13, 2012, 1:52 PM

Post #21 of 27 (252 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Brett Cannon <brett [at] python> added the comment:

I don't feel the need to, but I can in a few days if you want me to (just let me know if you do).

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

May 17, 2012, 10:03 AM

Post #22 of 27 (251 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Roundup Robot <devnull [at] psf> added the comment:

New changeset edb9ce3a6c2e by Antoine Pitrou in branch 'default':
Issue #9260: A finer-grained import lock.
http://hg.python.org/cpython/rev/edb9ce3a6c2e

----------
nosy: +python-dev

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

May 17, 2012, 10:07 AM

Post #23 of 27 (252 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Changes by Arfrever Frehtes Taifersar Arahesis <Arfrever.FTA [at] GMail>:


----------
nosy: +Arfrever

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

May 17, 2012, 10:10 AM

Post #24 of 27 (252 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Antoine Pitrou <pitrou [at] free> added the comment:

I have now pushed the patch.

----------
resolution: -> fixed
status: open -> pending

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

May 17, 2012, 10:10 AM

Post #25 of 27 (254 views)
Permalink
[issue9260] A finer grained import lock [In reply to]

Changes by Antoine Pitrou <pitrou [at] free>:


----------
stage: patch review -> committed/rejected
status: pending -> open

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue9260>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

First page Previous page 1 2 Next page Last page  View All Python bugs RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.