Mailing List Archive: Python: Bugs

[issue3139] bytearrays are not thread safe

1 2 3

View All

Index | Next | Previous | View Threaded

report at bugs

Jun 24, 2008, 5:06 AM

Post #1 of 68 (595 views)
Permalink

[issue3139] bytearrays are not thread safe

Changes by Amaury Forgeot d'Arc <amauryfa[at]gmail.com>:

----------
title: print is not thread safe -> bytearrays are not thread safe

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 3, 2008, 11:08 AM

Post #2 of 68 (571 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

Wow...
Isn't this kind of code supposed to ask for a buffer on the bytearray
object, together with an optional lock on it (or something like that)?

----------
nosy: +pitrou

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 3, 2008, 11:29 PM

Post #3 of 68 (570 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Amaury Forgeot d'Arc <amauryfa[at]gmail.com> added the comment:

Probably, but this affects all PyArg_ParseTuple("s#") calls that release
the GIL afterwards. How many of them are there?

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 5, 2008, 11:35 AM

Post #4 of 68 (561 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

If I try to follow the chain the consequences:
- all PyArg_ParseTuple("s#") calls that release the GIL afterwards
should be re-written to use another API (which one I don't know exactly,
but hopefully the appropriate functions are already provided by the
buffer API); this applies to third-party extension modules as well
- consequently, forward compatibility is broken in an important way
(but it would probably be ok for py3k)
- perhaps the bytearray type should not have been backported to 2.6, or
perhaps it should carry a big warning in the documentation

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 5, 2008, 11:39 AM

Post #5 of 68 (561 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

By the way, here's a more reliable way to make it crash (on my Linux
machine):

import bz2, threading

bz2c = bz2.BZ2Compressor()
# Span at least a whole arena (256KB long)
junk_len = 512 * 1024
junk = b"a" * junk_len
buf = bytearray()

for x in range(50):
buf[:] = junk
t = threading.Thread(target=bz2c.compress, args=[buf])
t.start()
buf[:] = b""
t.join()

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 5, 2008, 11:48 AM

Post #6 of 68 (560 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

Now I've just discovered there is the same problem with the
array.array() type (see following code).

import bz2, threading, array

bz2c = bz2.BZ2Compressor()
# Span at least a whole arena (256KB long)
junk_len = 512 * 1024
junk = array.array("c", b"a") * junk_len
empty = array.array("c")
buf = empty[:]

for x in range(50):
buf[:] = junk
t = threading.Thread(target=bz2c.compress, args=[buf])
t.start()
buf[:] = empty
t.join()

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 5, 2008, 3:20 PM

Post #7 of 68 (557 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Martin v. Löwis <martin[at]v.loewis.de> added the comment:

I believe the 2.6 s# processing works correctly; the error is in the
bytearray object. This either shouldn't support the buffer interface, or
it shouldn't reallocate the buffer after having returned a pointer to it.

For 3k, convertbuffer shouldn't call bf_releasebuffer; instead, the
caller of ParseTuple somehow needs to release the buffer. As a
consequence, s# should refuse buffer objects who have a bf_releasebuffer
operation, and another code might get added to fill out a Py_buffer - or
s# can get changed to always produce a Py_buffer (which would affect the
code significantly).

----------
nosy: +loewis

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 5, 2008, 3:39 PM

Post #8 of 68 (557 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

Le samedi 05 juillet 2008 à 22:20 +0000, Martin v. Löwis a écrit :
> Martin v. Löwis <martin[at]v.loewis.de> added the comment:
>
> I believe the 2.6 s# processing works correctly; the error is in the
> bytearray object. This either shouldn't support the buffer interface, or
> it shouldn't reallocate the buffer after having returned a pointer to it.

getbuffer and releasebuffer exist in both 2.6 and 3.0, and bytearray
supports those methods in both.

As for array.array, it only implements old buffer API.

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 5, 2008, 3:47 PM

Post #9 of 68 (558 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

For reference, here is a proof-of-concept patch which shows how to fix
the bytearray crasher above (it raises a BufferError instead).

Added file: http://bugs.python.org/file10822/bzcrash.py

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 5, 2008, 3:47 PM

Post #10 of 68 (557 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Changes by Antoine Pitrou <pitrou[at]free.fr>:

Removed file: http://bugs.python.org/file10822/bzcrash.py

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 5, 2008, 3:48 PM

Post #11 of 68 (557 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Changes by Antoine Pitrou <pitrou[at]free.fr>:

Added file: http://bugs.python.org/file10823/bzcrash.patch

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 5, 2008, 11:28 PM

Post #12 of 68 (547 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Martin v. Löwis <martin[at]v.loewis.de> added the comment:

Fixing this in the methods themselves has the disadvantage that the
error reporting is not that good anymore.

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 6, 2008, 12:32 AM

Post #13 of 68 (546 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Changes by Ismail Donmez <ismail[at]namtrac.org>:

----------
nosy: +cartman

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 6, 2008, 10:25 AM

Post #14 of 68 (537 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Changes by Martin v. Löwis <martin[at]v.loewis.de>:

----------
priority: -> release blocker

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 16, 2008, 5:33 AM

Post #15 of 68 (511 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Barry A. Warsaw <barry[at]python.org> added the comment:

Not blocking beta 2 because there's not enough time for the fix, but
this will definitely block beta 3.

----------
nosy: +barry
priority: release blocker -> deferred blocker

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 17, 2008, 8:46 PM

Post #16 of 68 (499 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Changes by Barry A. Warsaw <barry[at]python.org>:

----------
priority: deferred blocker -> release blocker

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 24, 2008, 9:04 AM

Post #17 of 68 (471 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Martin v. Löwis <martin[at]v.loewis.de> added the comment:

Here is a patch adding the s* format, and changing files, sockets, and
fileio to use it. For bz2, the immediate effect is that you get a type
error (as an object providing bf_releasebuffer cannot be converted
through s#/w# anymore); it would be possible to fix bz2 also to use
s*/w* instead.

I'd like reviewers to focus on flaws in the approach and bugs in the
implementation; existing code should be converted to the new API
afterwards (or not converted at all for 2.6/3.0, but only as patches get
contributed).

If this is accepted in principle, I'll forward-port it to 3.0.

Added file: http://bugs.python.org/file10969/s_star.diff

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 25, 2008, 2:16 AM

Post #18 of 68 (466 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

I haven't yet studied the patch in detail but I have a few questions:

(1) are you sure it is safe not to INCREF the obj pointer in the
Py_buffer? in the cases handled by your patch we still hold a reference
to the original args tuple and dict before calling PyBuffer_Release(),
but some functions may want to store the Py_buffer object somewhere to
re-use it later. It would seem more logical for PyBuffer_FillInfo to
INCREF the obj, and for PyBuffer_Release to DECREF it and set it to NULL.

(2) is it necessary to call directly bf_getbuffer & the like or is there
a higher-level API to do it?

(3) didn't you forget to convert "PyArg_ParseTuple(args, "s#iO:sendto",
[...])" in sock_sendto?

(4) is it really necessary to do a special case with PyString_Check()
rather than rely on the string type's getbuffer method?

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 25, 2008, 2:46 AM

Post #19 of 68 (466 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

Travis, it would be really nice to have your input on this.

----------
nosy: +teoliphant

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 26, 2008, 6:07 AM

Post #20 of 68 (461 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Martin v. Löwis <martin[at]v.loewis.de> added the comment:

> (1) are you sure it is safe not to INCREF the obj pointer in the
> Py_buffer?

Yes, that's the semantics of the current buffer interface, and I cannot
see a flaw in it. When you call getbuffer, you certainly hold a
reference, and it is your obligation to hold onto this reference
somehow. So it is definitely safe (if properly documented).

> It would seem more logical for PyBuffer_FillInfo to
> INCREF the obj, and for PyBuffer_Release to DECREF it and set it to NULL.

Perhaps. I cannot quite see what undesirable consequences that might
have - feel free to develop and test an alternative patch that takes
that approach.

> (2) is it necessary to call directly bf_getbuffer & the like or is there
> a higher-level API to do it?

There is PyObject_GetBuffer and PyObject_ReleaseBuffer, but it is not
higher-level. I need to check the function pointers, anyway (to
determine whether the object supports getbuffer and requires
releasebuffer or not), so calling them directly is the right level
of abstraction (IMO).

> (3) didn't you forget to convert "PyArg_ParseTuple(args, "s#iO:sendto",
> [...])" in sock_sendto?

True.

> (4) is it really necessary to do a special case with PyString_Check()
> rather than rely on the string type's getbuffer method?

That's what the code always did (for s#). It would be possible to
eliminate that case, yes.

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 27, 2008, 1:38 AM

Post #21 of 68 (448 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Amaury Forgeot d'Arc <amauryfa[at]gmail.com> added the comment:

With the patch the following script crashes the 2.6 interpreter on Windows:

import sys, io, threading
stdout2 = io.open(sys.stdout.fileno(), mode="w")
def f(i):
for i in range(10):
stdout2.write(unicode((x, i)) + '\n')
for x in range(10):
t = threading.Thread(target=f, args=(x,))
t.start()

(with py3k, replace "stdout2.write" with a simple "print")

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 30, 2008, 9:56 AM

Post #22 of 68 (427 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

The problem is that the fix for #3295 was committed in the py3k branch
(in r64751) rather thank on the trunk!
Once PyExc_BufferError is defined properly the crash disappears and
exceptions are printed instead.

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 30, 2008, 10:45 AM

Post #23 of 68 (425 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Benjamin Peterson <musiccomposition[at]gmail.com> added the comment:

Sorry, that was my oversight! I've backported the fix.

----------
nosy: +benjamin.peterson

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 30, 2008, 4:03 PM

Post #24 of 68 (424 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Amaury Forgeot d'Arc <amauryfa[at]gmail.com> added the comment:

It's indeed better.
Now with when running my previous script I can see the exception ;-)

Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\dev\python\trunk1\lib\threading.py", line 523, in
__bootstrap_inner
self.run()
File "C:\dev\python\trunk1\lib\threading.py", line 478, in run
self.__target(*self.__args, **self.__kwargs)
File "<stdin>", line 3, in f
File "C:\dev\python\trunk1\lib\io.py", line 1473, in write
self.buffer.write(b)
File "C:\dev\python\trunk1\lib\io.py", line 1041, in write
self._write_buf.extend(b)
BufferError: Existing exports of data: object cannot be re-sized

Again, I think this is unfortunate for a simple script that prints from
several threads.

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 30, 2008, 4:41 PM

Post #25 of 68 (422 views)
Permalink

[issue3139] bytearrays are not thread safe [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

Le mercredi 30 juillet 2008 à 23:03 +0000, Amaury Forgeot d'Arc a
écrit :
> Again, I think this is unfortunate for a simple script that prints from
> several threads.

Yes, but it's an issue with the BufferedWriter implementation, since it
changes the length of its underlying bytearray object. If it was
rewritten to use a fixed-size bytearray, the problem would probably
disappear.

(in the middle term BufferedReader and BufferedWriter should perhaps be
both rewritten in C)

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue3139>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

1 2 3

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com