Mailing List Archive: Python: Bugs

[issue7211] select module - kevent ident field 64 bit issue

Index | Next | Previous | View Threaded

report at bugs

Oct 26, 2009, 12:54 PM

Post #1 of 14 (138 views)
Permalink

[issue7211] select module - kevent ident field 64 bit issue

New submission from Michael Broghton <mbroughton[at]advanis.ca>:

On FreeBSD and MacOS 64-bit systems the ident field of a kevent is big
enough to hold a 64-bit integer (uintptr_t). Looks like Python is
casting it to an unsigned 32-bit integer.

This is inconvenient for implementing kqueue timers, where id(timer_obj)
is a natural choice for an ident.

----------
components: Library (Lib)
messages: 94502
nosy: mbroughton
severity: normal
status: open
title: select module - kevent ident field 64 bit issue
type: behavior
versions: Python 3.1

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue7211>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Oct 26, 2009, 1:01 PM

Post #2 of 14 (131 views)
Permalink

[issue7211] select module - kevent ident field 64 bit issue [In reply to]

Changes by Antoine Pitrou <pitrou[at]free.fr>:

----------
nosy: +christian.heimes, therve

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue7211>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Oct 26, 2009, 1:01 PM

Post #3 of 14 (131 views)
Permalink

[issue7211] select module - kevent ident field 64 bit issue [In reply to]

Changes by Antoine Pitrou <pitrou[at]free.fr>:

----------
priority: -> normal
type: behavior -> feature request
versions: +Python 2.7, Python 3.2 -Python 3.1

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue7211>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Oct 26, 2009, 1:28 PM

Post #4 of 14 (131 views)
Permalink

[issue7211] select module - kevent ident field 64 bit issue [In reply to]

Martin v. Löwis <martin[at]v.loewis.de> added the comment:

Would you like to propose a patch?

----------
nosy: +loewis

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue7211>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Oct 26, 2009, 3:34 PM

Post #5 of 14 (128 views)
Permalink

[issue7211] select module - kevent ident field 64 bit issue [In reply to]

Michael Broghton <mbroughton[at]advanis.ca> added the comment:

I'm not sure how to patch this so that it will work on both 32 and 64
bit systems. Issues:

1. What would be an appropriate member type for ident in
kqueue_event_members? It seems like T_PYSSIZET might work. Otherwise, I
am guessing that this will involve some #if's.

2. I think the format spec in kqueue_event_repr needs to change. It
seems like this will also require some #if's.

3. kqueue_event_init uses PyObject_AsFileDescriptor to set the ident
field. This should be doing a PyLong_Check first to see if
PyLong_AsSomething would be more appropriate.

4. I think the type of the result variable in kqueue_event_richcompare
needs to be changed to long long int.

----------

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue7211>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Oct 26, 2009, 3:48 PM

Post #6 of 14 (128 views)
Permalink

[issue7211] select module - kevent ident field 64 bit issue [In reply to]

Martin v. Löwis <martin[at]v.loewis.de> added the comment:

> 1. What would be an appropriate member type for ident in
> kqueue_event_members? It seems like T_PYSSIZET might work. Otherwise, I
> am guessing that this will involve some #if's.

IIUC, it needs to match *exactly* the field size inside struct kevent.
So you'll have to use #ifs, possible along with autoconf tests (to
find out the sizes of the fields that typically vary across
implementations).

> 2. I think the format spec in kqueue_event_repr needs to change. It
> seems like this will also require some #if's.

Here, I would widen the fields to size_t, and use the size_t formatter
(assuming that all systems supporting kqueue also know how to print
size_t, or know to print long long).

> 3. kqueue_event_init uses PyObject_AsFileDescriptor to set the ident
> field. This should be doing a PyLong_Check first to see if
> PyLong_AsSomething would be more appropriate.

Hmm. I think I need to understand the use case better. Can you post
some sample code where this all matters?

If this *is* a regular file-like object, then surely int is enough, no?

> 4. I think the type of the result variable in kqueue_event_richcompare
> needs to be changed to long long int.

Fine with me. The question then is whether long long is available on
all systems that support kqueue.

----------

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue7211>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Oct 26, 2009, 5:52 PM

Post #7 of 14 (123 views)
Permalink

[issue7211] select module - kevent ident field 64 bit issue [In reply to]

Michael Broghton <mbroughton[at]advanis.ca> added the comment:

Martin, thanks for your responses. In regards to point three:

Kqueue's are not just used for file descriptors. I believe this is the
reason why the ident field is a uintptr_t and not an int.

The example I gave was for kqueue timers. Since the operating system
does not allocate 'timer descriptors' for you, I decided to use the
return value from the id function.

Here is some code that demonstrates:

import select
a = 1
b = id(a)
c = select.kevent(a)
d = select.kevent(b)
assert a == c.ident
assert b == d.ident
assert b & (1<<32) - 1 == d.ident

The second assert will fail on 64 bit systems if 'b' is too big.

Anyway, I will try to come up with a patch for this.

----------

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue7211>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Oct 27, 2009, 8:35 AM

Post #8 of 14 (110 views)
Permalink

[issue7211] select module - kevent ident field 64 bit issue [In reply to]

Michael Broghton <mbroughton[at]advanis.ca> added the comment:

This is against release31-maint, if it matters.

----------
keywords: +patch
Added file: http://bugs.python.org/file15212/kevent.patch

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue7211>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Oct 29, 2009, 6:29 AM

Post #9 of 14 (102 views)
Permalink

[issue7211] select module - kevent ident field 64 bit issue [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

According to man pages on the Web, the kevent structure is:

struct kevent {
uintptr_t ident; /* identifier for this event */
short filter; /* filter for event */
u_short flags; /* action flags for kqueue */
u_int fflags; /* filter flag value */
intptr_t data; /* filter data value */
void *udata; /* opaque user data identifier */
};

So the `ident` field is indeed an uintptr_t.
However the patch is slightly wrong IMO:

- you should not blindly use `long long`. Python already defines a
"Py_uintptr_t" type. If you need more information you should conditional
compilation such as in (for the off_t type)
http://svn.python.org/view/python/trunk/Modules/_io/_iomodule.h?view=markup

- in tests, you should use sys.maxsize (which gives you the max value of
a ssize_t integer) rather than choosing based on platform.architecture():

>>> sys.maxsize
9223372036854775807
>>> 2**64*1
18446744073709551616L
>>> sys.maxsize*2 + 1
18446744073709551615L

PS : a patch against trunk or py3k is preferred, but in this case it
would probably apply cleanly anyway

----------
nosy: +pitrou

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue7211>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Oct 29, 2009, 6:31 AM

Post #10 of 14 (102 views)
Permalink

[issue7211] select module - kevent ident field 64 bit issue [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

Apparently Roundup borked the URL. Let's try another one (or look into
Modules/_io/_iomodule.h):
http://code.python.org/hg/trunk/file/b9bc35171668/Modules/_io/_iomodule.h#l88

----------

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue7211>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Oct 29, 2009, 7:59 AM

Post #11 of 14 (102 views)
Permalink

[issue7211] select module - kevent ident field 64 bit issue [In reply to]

Michael Broghton <mbroughton[at]advanis.ca> added the comment:

Antoine, thanks for the tips and the example. I have updated the patch.

I checked and this does apply cleanly to py3k.

----------
Added file: http://bugs.python.org/file15228/kevent.patch

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue7211>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Oct 29, 2009, 8:58 AM

Post #12 of 14 (102 views)
Permalink

[issue7211] select module - kevent ident field 64 bit issue [In reply to]

Eric Smith <eric[at]trueblade.com> added the comment:

The patch (http://bugs.python.org/file15228/kevent.patch) works for me
under OS X 10.5.8 Intel.

All tests pass, except test_telnetlib which fails intermittently with
and without the patch.

Python 3.2a0 (py3k:75951, Oct 29 2009, 11:38:58)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin

----------
nosy: +eric.smith

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue7211>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 4, 2009, 10:15 AM

Post #13 of 14 (45 views)
Permalink

[issue7211] select module - kevent ident field 64 bit issue [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

For me, the patch is worth trying out on the buildbots - to see if there
are any configuration problems we might have overlooked.

----------

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue7211>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 4, 2009, 1:57 PM

Post #14 of 14 (44 views)
Permalink

[issue7211] select module - kevent ident field 64 bit issue [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

Patch was committed in r76108 (trunk) and r76111 (py3k) and didn't
introduce any regression on the FreeBSD and OS X buildbots. Thanks!

----------
resolution: -> fixed
stage: -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue7211>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com