Mailing List Archive: Python: Dev

pthread sem PyThread_acquire_lock

Index | Next | Previous | View Threaded

flormayer at aim

Jun 27, 2009, 10:20 AM

Post #1 of 9 (925 views)
Permalink

pthread sem PyThread_acquire_lock

Hello.
As this is my first post I will try to introduce myself as requested in
the welcome email. If you aren't interested in my person, just continue
reading at the next paragraph. I'm a student from Vienna/Austria. I
attend what would match high school in the United States. I have been
writing Python for about 3 years now and have just began to dig into the
implementation of CPython.

Now to my real question. I have noticed that PyThread_acquire_lock
swallows all signals with pthread using sems. Looking at the source code
it appears that this is intended, but I cannot see the reason for that.
It seems the pthread sem implementation is the only one doing so. Can
any of you tell me the reason why it swallows signals?

I have already prepared a patch that introduces a new _noswallow
function if sem are used and uses this in threadmodule.c. But if there
isn't any real reason for it to swallow exceptions, I think it would be
wisest to change that.

Best regards,
Florian Mayer
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

martin at v

Jun 27, 2009, 1:45 PM

Post #2 of 9 (869 views)
Permalink

Re: pthread sem PyThread_acquire_lock [In reply to]

> Now to my real question. I have noticed that PyThread_acquire_lock
> swallows all signals with pthread using sems. Looking at the source code
> it appears that this is intended, but I cannot see the reason for that.
> It seems the pthread sem implementation is the only one doing so. Can
> any of you tell me the reason why it swallows signals?

Can you please explain what you mean by "swallowing"?
What is the sequence of actions triggering the case you are talking
about, what happens, and what do you expect to happen instead?
Also, how would you fix this (in principle, not what the specific
patch would look like)?

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

flormayer at aim

Jun 27, 2009, 2:20 PM

Post #3 of 9 (867 views)
Permalink

Re: pthread sem PyThread_acquire_lock [In reply to]

Martin v. Löwis wrote:
> Can you please explain what you mean by "swallowing"?
The signals don't get through when acquiring the lock.
> What is the sequence of actions triggering the case you are talking
> about, what happens, and what do you expect to happen instead?
Create a Lock object, call .acquire and try to press Ctrl-C, the
keyboardinterrupt is swallowed
> Also, how would you fix this (in principle, not what the specific
> patch would look like)?
Remove the loop that explicitly does this in a new function and use that
one in threadmodule.c for lock.acquire.
>
> Regards,
> Martin
Best regards,
Florian

PS: Sorry for sending this message twice.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

flormayer at aim

Jun 27, 2009, 2:20 PM

Post #4 of 9 (871 views)
Permalink

Re: pthread sem PyThread_acquire_lock [In reply to]

Martin v. Löwis wrote:
> Can you please explain what you mean by "swallowing"?
The signals don't get through when acquiring the lock.
> What is the sequence of actions triggering the case you are talking
> about, what happens, and what do you expect to happen instead?
Easiest way is to create a Queue.Queue and call .get() on it and try to
Ctrl-C it.
> Also, how would you fix this (in principle, not what the specific
> patch would look like)?
Remove the loop that explicitly does this in a new function and use that
one in threadmodule.c for lock.acquire.
>
> Regards,
> Martin
Best regards,
Florian

PS: Excuse me, I messed up things once again.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

phillip.sitbon+python-dev at gmail

Jun 29, 2009, 12:33 PM

Post #5 of 9 (855 views)
Permalink

Re: pthread sem PyThread_acquire_lock [In reply to]

I'll do my best to try and explain/contribute, but please feel free to
correct anything I get wrong.

I believe the "swallowing" he's referring to is the ignoring of errno
EINTR. I don't think that's the correct place to handle signals to
begin with- why not just use the signal module to deal with such a
scenario?

http://docs.python.org/dev/library/signal.html#module-signal

AFAIK, ignoring EINTR doesn't preclude the calling of signal handlers.
There are many functions that don't return this value anymore, making
them reentrant. I remember a number of years ago when it wasn't part
of any standard to return EINTR or not, and so the only way to provide
consistent behavior was to ignore it and loop. I'm not sure if that is
still the case.

A great example is reading from a socket. Whether or not it can be
interrupted depends on the platform, so catching Ctrl+C often requires
a timeout loop.

Also, remember that signals are asynchronous in the sense that they
are handled outside the normal execution flow of a program. Checking
for EINTR probably isn't the best way to determine if a signal has
been sent to the program.

Cheers,

Phillip

On Sat, Jun 27, 2009 at 3:58 PM, Florian Mayer <flormayer [at] aim> wrote:
>
> Martin v. Löwis wrote:
>>
>> Can you please explain what you mean by "swallowing"?
>
> The signals don't get through when acquiring the lock.
>>
>> What is the sequence of actions triggering the case you are talking
>> about, what happens, and what do you expect to happen instead?
>
> Easiest way is to create a Queue.Queue and call .get() on it and try to Ctrl-C it.
>>
>> Also, how would you fix this (in principle, not what the specific
>> patch would look like)?
>
> Remove the loop that explicitly does this in a new function and use that
> one in threadmodule.c for lock.acquire.
>>
>> Regards,
>> Martin
>
> Best regards,
> Florian
>
> PS: Excuse me, I messed up things once again.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev [at] python
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/phillip.sitbon%2Bpython-dev%40gmail.com
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

martin at v

Jun 29, 2009, 2:28 PM

Post #6 of 9 (856 views)
Permalink

Re: pthread sem PyThread_acquire_lock [In reply to]

> AFAIK, ignoring EINTR doesn't preclude the calling of signal handlers.

This is my understanding as well - so I don't think Python actually
"swallows" the signal.

> A great example is reading from a socket. Whether or not it can be
> interrupted depends on the platform, so catching Ctrl+C often requires
> a timeout loop.
>
> Also, remember that signals are asynchronous in the sense that they
> are handled outside the normal execution flow of a program. Checking
> for EINTR probably isn't the best way to determine if a signal has
> been sent to the program.

I think it would be reasonable to support "asynchronous" exceptions,
and Python supports SIGINT fairly well most of the time.

It might be possible to support keyboard interrupts throughout
the system, but changing Python to do so could also cause
incompatibilities. So any change must be done with greatest care,
but simultaneously, should also try to arrange to cover all cases.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

greg at krypto

Jul 2, 2009, 3:47 PM

Post #7 of 9 (793 views)
Permalink

Re: pthread sem PyThread_acquire_lock [In reply to]

On Mon, Jun 29, 2009 at 2:28 PM, "Martin v. Löwis"<martin [at] v> wrote:
>> AFAIK, ignoring EINTR doesn't preclude the calling of signal handlers.
>
> This is my understanding as well - so I don't think Python actually
> "swallows" the signal.
>
>> A great example is reading from a socket. Whether or not it can be
>> interrupted depends on the platform, so catching Ctrl+C often requires
>> a timeout loop.
>>
>> Also, remember that signals are asynchronous in the sense that they
>> are handled outside the normal execution flow of a program. Checking
>> for EINTR probably isn't the best way to determine if a signal has
>> been sent to the program.
>
> I think it would be reasonable to support "asynchronous" exceptions,
> and Python supports SIGINT fairly well most of the time.
>
> It might be possible to support keyboard interrupts throughout
> the system, but changing Python to do so could also cause
> incompatibilities. So any change must be done with greatest care,
> but simultaneously, should also try to arrange to cover all cases.
>
> Regards,
> Martin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev [at] python
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/greg%40krypto.org

If you want signals to actually be handled in a timely manner, its
best to leave the main thread of the program alone as a signal
handling thread that just spends its time in a loop of time.sleep(123)
calls rather than blocking on any sort of lock. Spawn other threads
to do the actual work in your program. Signals are delivered
indirectly in the existing CPython implementation by setting an
internal flag that the main interpreter thread polls on occasion so
blocking calls that do not interrupt and return early being called
from the main thread will effectively block signals.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

exarkun at divmod

Jul 2, 2009, 4:04 PM

Post #8 of 9 (799 views)
Permalink

Re: pthread sem PyThread_acquire_lock [In reply to]

On Thu, 2 Jul 2009 15:47:48 -0700, "Gregory P. Smith" <greg [at] krypto> wrote:
>On Mon, Jun 29, 2009 at 2:28 PM, "Martin v. LÃ¶wis"<martin [at] v> wrote:
>>> AFAIK, ignoring EINTR doesn't preclude the calling of signal handlers.
>>
>> This is my understanding as well - so I don't think Python actually
>> "swallows" the signal.
>>
>>> A great example is reading from a socket. Whether or not it can be
>>> interrupted depends on the platform, so catching Ctrl+C often requires
>>> a timeout loop.
>>>
>>> Also, remember that signals are asynchronous in the sense that they
>>> are handled outside the normal execution flow of a program. Checking
>>> for EINTR probably isn't the best way to determine if a signal has
>>> been sent to the program.
>>
>> I think it would be reasonable to support "asynchronous" exceptions,
>> and Python supports SIGINT fairly well most of the time.
>>
>> It might be possible to support keyboard interrupts throughout
>> the system, but changing Python to do so could also cause
>> incompatibilities. So any change must be done with greatest care,
>> but simultaneously, should also try to arrange to cover all cases.
>>
>> Regards,
>> Martin
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev [at] python
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
>If you want signals to actually be handled in a timely manner, its
>best to leave the main thread of the program alone as a signal
>handling thread that just spends its time in a loop of time.sleep(123)
>calls rather than blocking on any sort of lock. Spawn other threads
>to do the actual work in your program. Signals are delivered
>indirectly in the existing CPython implementation by setting an
>internal flag that the main interpreter thread polls on occasion so
>blocking calls that do not interrupt and return early being called
>from the main thread will effectively block signals.

Yes, this is all true now. The question is why the implementation
works that way, and whether it is desirable to keep it working that
way.

Considering *some* of the lock implementations make themselves not
interruptable by threads while others don't bother, it seems like
*some* change to the status quo is desirable.

Jean-Paul
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

guido at python

Jul 2, 2009, 8:49 PM

Post #9 of 9 (832 views)
Permalink

Re: pthread sem PyThread_acquire_lock [In reply to]

On Thu, Jul 2, 2009 at 4:04 PM, Jean-Paul Calderone<exarkun [at] divmod> wrote:
> On Thu, 2 Jul 2009 15:47:48 -0700, "Gregory P. Smith" <greg [at] krypto>
>> If you want signals to actually be handled in a timely manner, its
>> best to leave the main thread of the program alone as a signal
>> handling thread that just spends its time in a loop of time.sleep(123)
>> calls rather than blocking on any sort of lock. Spawn other threads
>> to do the actual work in your program. Signals are delivered
>> indirectly in the existing CPython implementation by setting an
>> internal flag that the main interpreter thread polls on occasion so
>> blocking calls that do not interrupt and return early being called
>> from the main thread will effectively block signals.
>
> Yes, this is all true now. The question is why the implementation
> works that way, and whether it is desirable to keep it working that
> way.
>
> Considering *some* of the lock implementations make themselves not
> interruptable by threads while others don't bother, it seems like
> *some* change to the status quo is desirable.

I'm probably to blame for this, but IIRC when we did it, lock
acquisitions were not interruptible by signals. However, that was
before posix threads were even standardized. Nowadays I believe we use
pthreads everywhere except on Windows. But I know nearly nothing about
pthreads. If pthreads mutex acquisition is (or can easily be made)
interruptible by signals I'd be all for adding signal handling to
them, assuming it doesn't slow things down much. Also, if there are
ways to do the "try to acquire a lock or wait until a given timeout"
natively and while handling signals, that would be a great
improvement.

I'll probably regret saying this within an hour though,

--
--Guido van Rossum (home page: http://www.python.org/~guido/)
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads