Mailing List Archive: Python: Dev

Add an optional timeout to lock operations

Index | Next | Previous | View Threaded

solipsis at pitrou

Nov 17, 2009, 3:46 AM

Post #1 of 9 (1253 views)
Permalink

Add an optional timeout to lock operations

Hello,

I've submitted a patch (*) to add an optional timeout to locking operations
(Lock.acquire() etc.). Since it's a pretty basic functionality, I would like to
know if there was any good reason for not doing it.

(*) http://bugs.python.org/issue7316

Thank you

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

guido at python

Nov 17, 2009, 8:01 AM

Post #2 of 9 (1199 views)
Permalink

Re: Add an optional timeout to lock operations [In reply to]

I think I can answer the "why" question: thread.c is *very* old code,
in fact it predates the posix threads standard. When we (actually
Sjoerd Mullender) wrote it, we had a number of OS-specific locking
APIs to work with and the API was designed to fit all of them. I don't
even recall the initial set, but I think it included SGI Irix and and
old version of SunOS. So then over time new platforms were added, but
adding a new method or parameter to the API was nearly impossible
because someone would have to find out how to implement the new
feature on all supported platforms. I think the number of platforms
has dwindled to two or three (Posix, Windows, and maybe one minority
OS?), so now's the time to do it. (IOW I think the idea of the patch
is fine.)

Will locks be interruptible with ^C? That is an oft-requested feature
which also wasn't supported at that time; what's the situation
nowadays?

--Guido

On Tue, Nov 17, 2009 at 3:46 AM, Antoine Pitrou <solipsis [at] pitrou> wrote:
>
> Hello,
>
> I've submitted a patch (*) to add an optional timeout to locking operations
> (Lock.acquire() etc.). Since it's a pretty basic functionality, I would like to
> know if there was any good reason for not doing it.
>
> (*) http://bugs.python.org/issue7316
>
> Thank you
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev [at] python
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

solipsis at pitrou

Nov 17, 2009, 8:21 AM

Post #3 of 9 (1183 views)
Permalink

Re: Add an optional timeout to lock operations [In reply to]

Guido van Rossum <guido <at> python.org> writes:
>
> I think the number of platforms
> has dwindled to two or three (Posix, Windows, and maybe one minority
> OS?), so now's the time to do it. (IOW I think the idea of the patch
> is fine.)

Thanks. (the minority OS would be OS/2, I think)

> Will locks be interruptible with ^C? That is an oft-requested feature
> which also wasn't supported at that time; what's the situation
> nowadays?

They still aren't interruptible. From what I can read it may be possible to make
them interruptible in the POSIX semaphore-based implementation, not in the POSIX
condition variable-based implementation (which is used as a fallback when POSIX
semaphores are not available, but I don't know whether this fallback is still
useful).
As for Windows I have absolutely no idea.

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ncoghlan at gmail

Nov 18, 2009, 2:38 AM

Post #4 of 9 (1178 views)
Permalink

Re: Add an optional timeout to lock operations [In reply to]

Antoine Pitrou wrote:
> Guido van Rossum <guido <at> python.org> writes:
>> Will locks be interruptible with ^C? That is an oft-requested feature
>> which also wasn't supported at that time; what's the situation
>> nowadays?
>
> They still aren't interruptible. From what I can read it may be possible to make
> them interruptible in the POSIX semaphore-based implementation, not in the POSIX
> condition variable-based implementation (which is used as a fallback when POSIX
> semaphores are not available, but I don't know whether this fallback is still
> useful).

I'm pretty sure at least some variants of *BSD still don't have OS level
semaphores - their lack is the reason multiprocessing doesn't
necessarily work everywhere that the threading module works (since mp
needs semaphores in order to work its magic).

Jesse would probably know the gory details.

Cheers,
Nick.

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

jnoller at gmail

Nov 18, 2009, 4:45 AM

Post #5 of 9 (1177 views)
Permalink

Re: Add an optional timeout to lock operations [In reply to]

On Nov 18, 2009, at 5:38 AM, Nick Coghlan <ncoghlan [at] gmail> wrote:

> Antoine Pitrou wrote:
>> Guido van Rossum <guido <at> python.org> writes:
>>> Will locks be interruptible with ^C? That is an oft-requested
>>> feature
>>> which also wasn't supported at that time; what's the situation
>>> nowadays?
>>
>> They still aren't interruptible. From what I can read it may be
>> possible to make
>> them interruptible in the POSIX semaphore-based implementation, not
>> in the POSIX
>> condition variable-based implementation (which is used as a
>> fallback when POSIX
>> semaphores are not available, but I don't know whether this
>> fallback is still
>> useful).
>
> I'm pretty sure at least some variants of *BSD still don't have OS
> level
> semaphores - their lack is the reason multiprocessing doesn't
> necessarily work everywhere that the threading module works (since mp
> needs semaphores in order to work its magic).
>
> Jesse would probably know the gory details.
>
> Cheers,
> Nick.
>

Nick is right, many of the BSDs and FreeBSD up until fairly recently
did not have named shared semaphore support. Still yet, the behavior
is broken on some OSes such as OS X which you have to work around.

I wouldn't recommend using them for threading right now, there's an
assumption that threading "just works" unlike multiprocessing, which
people understand has caveats.

Jesse
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

solipsis at pitrou

Nov 18, 2009, 5:50 AM

Post #6 of 9 (1173 views)
Permalink

Re: Add an optional timeout to lock operations [In reply to]

Jesse Noller <jnoller <at> gmail.com> writes:
>
> Nick is right, many of the BSDs and FreeBSD up until fairly recently
> did not have named shared semaphore support. Still yet, the behavior
> is broken on some OSes such as OS X which you have to work around.

The core locking support only uses anonymous semaphores (they don't need to be
shared accross processes obviously).

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

jnoller at gmail

Nov 18, 2009, 6:27 AM

Post #7 of 9 (1166 views)
Permalink

Re: Add an optional timeout to lock operations [In reply to]

On Wed, Nov 18, 2009 at 8:50 AM, Antoine Pitrou <solipsis [at] pitrou> wrote:
> Jesse Noller <jnoller <at> gmail.com> writes:
>>
>> Nick is right, many of the BSDs and FreeBSD up until fairly recently
>> did not have named shared semaphore support. Still yet, the behavior
>> is broken on some OSes such as OS X which you have to work around.
>
> The core locking support only uses anonymous semaphores (they don't need to be
> shared accross processes obviously).
>
> Regards
>
> Antoine.

That's what I get for replying from an iphone over morning coffee. See
also http://semanchuk.com/philip/sysv_ipc/ for Philip Semanchuk
semaphore/IPC work as well.

Given both multiprocessing and Philip's work are concerned about the
shared aspects, I think you're right - FBSD and others support the
posix semaphores for non-ipc stuff ok AFAIK (recent benchmark
http://www.ioremap.net/node/216).

jesse
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

db3l.net at gmail

Nov 18, 2009, 1:28 PM

Post #8 of 9 (1166 views)
Permalink

Re: Add an optional timeout to lock operations [In reply to]

Antoine Pitrou <solipsis [at] pitrou> writes:

> I've submitted a patch (*) to add an optional timeout to locking
> operations (Lock.acquire() etc.). Since it's a pretty basic
> functionality, I would like to know if there was any good reason for
> not doing it.

I always assumed it was because as a least-common-denominator set of
functionality, some platforms didn't have the necessary support.

Providing the discussion on this ends up with the an implementation
being accepted, I'd absolutely love to see this then leveraged by
threading.Condition.wait() rather rather than the current
poll-with-timed-sleep approach.

The overhead (CPU, but most notably latency) of that approach (which
also directly impacts Queues) has historically been my top reason in
various applications on Windows to have to implement my own Queue or
Condition structures using native Windows calls.

-- David

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

solipsis at pitrou

Nov 18, 2009, 2:20 PM

Post #9 of 9 (1165 views)
Permalink

Re: Add an optional timeout to lock operations [In reply to]

Hello,

> > I've submitted a patch (*) to add an optional timeout to locking
> > operations (Lock.acquire() etc.). Since it's a pretty basic
> > functionality, I would like to know if there was any good reason for
> > not doing it.
>
> I always assumed it was because as a least-common-denominator set of
> functionality, some platforms didn't have the necessary support.

Guido's answer says so indeed. Now py3k only needs to support POSIX and Windows
(and, provided Andrew MacIntyre maintains the port, OS/2), which both have
standard support for waiting-with-timeout.

> Providing the discussion on this ends up with the an implementation
> being accepted, I'd absolutely love to see this then leveraged by
> threading.Condition.wait() rather rather than the current
> poll-with-timed-sleep approach.

Agreed. The current patch (as proposed on http://bugs.python.org/issue7316)
includes exactly that.
Poll-with-timed-sleep is especially sub-optimal on laptops where short but
frequent wakeups can cause a significant decrease in battery life.
(the Linux community has been chasing this using powertop:
http://www.lesswatts.org/projects/powertop/)

Feel free to test or review if you're interested.

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads