Mailing List Archive: Python: Bugs

[issue15466] Python/importlib.h is different on 32bit and 64bit

Index | Next | Previous | View Threaded

report at bugs

Jul 27, 2012, 5:50 AM

Post #1 of 21 (142 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit

New submission from Amaury Forgeot d'Arc <amauryfa [at] gmail>:

As shown in a patch in issue15431, frozen.c does not output the same data on different platforms.

The first difference looks like this (extracted from the patch):
- 101,73,255,255,255,255,0,0,0,0,40,10,0,0,0,117,
+ 101,108,3,0,0,0,255,127,255,127,3,0,40,10,0,0,

On first row, 'I' followed by 0xFFFFFFFF on 8 bytes.
On second row, 'l' followed by 3 followed by 0xFFFFFFFF (in 3 chunks of 15 bits).
The Python number 0xFFFFFFFF is marshalled with TYPE_INT64 when SIZEOF_LONG>4 (Unix 64bit), and with TYPE_LONG on other platforms (32bit, or win64)

The C "long" type has much less importance in 3.x Python, because PyIntObject does not exist anymore.
I suggest to remove this distinction, and allow TYPE_INT64 on all platforms.

I don't see any compatibility issue, on unmarshalling both methods produce the same object.
I'll try to suggest a patch later today.

----------
messages: 166559
nosy: amaury.forgeotdarc, pitrou
priority: high
severity: normal
status: open
title: Python/importlib.h is different on 32bit and 64bit

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 27, 2012, 5:53 AM

Post #2 of 21 (138 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit [In reply to]

Amaury Forgeot d'Arc <amauryfa [at] gmail> added the comment:

I realize that uppercase I and lowercase L are easily confused:

On first row, 'I' (=TYPE_INT64) followed by 0xFFFFFFFF on 8 bytes.
On second row, 'l' (=TYPE_LONG) followed by 3 followed by 0xFFFFFFFF (in 3 chunks of 15 bits).

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 27, 2012, 6:21 AM

Post #3 of 21 (139 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit [In reply to]

Antoine Pitrou <pitrou [at] free> added the comment:

> I suggest to remove this distinction, and allow TYPE_INT64 on all
> platforms.

You have to find a 64-bit C int type that works on all platforms and for which a PyLongAsXXX() function exists, though. Or perhaps you want to restrict yourself to systems which have "long long" and where it is at least 64 bits wide.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 27, 2012, 6:45 AM

Post #4 of 21 (138 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit [In reply to]

Amaury Forgeot d'Arc <amauryfa [at] gmail> added the comment:

My primary goal is to have a stable importlib.h, and currently all developer work on machines where PY_LONG_LONG is >64bit. So the restriction is OK.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 27, 2012, 7:59 AM

Post #5 of 21 (131 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit [In reply to]

Changes by Brett Cannon <brett [at] python>:

----------
nosy: +brett.cannon

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 28, 2012, 3:16 AM

Post #6 of 21 (128 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit [In reply to]

Martin v. Löwis <martin [at] v> added the comment:

I suggest the opposite: never emit TYPE_INT64 at all. For compatibility, we can still support reading it in 3.3 (and drop that support in 3.4).

----------
nosy: +loewis

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 28, 2012, 3:47 AM

Post #7 of 21 (128 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit [In reply to]

Martin v. Löwis <martin [at] v> added the comment:

Here is a patch to stop generating TYPE_INT64. Ok for 3.30b2?

----------
keywords: +patch
priority: high -> release blocker
Added file: http://bugs.python.org/file26553/marshal.diff

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 28, 2012, 4:09 AM

Post #8 of 21 (128 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit [In reply to]

Amaury Forgeot d'Arc <amauryfa [at] gmail> added the comment:

Here is a patch that write TYPE_INT64 on most platforms (where either long or long long is 64bit).
It is admittedly much larger than Martin's...

----------
Added file: http://bugs.python.org/file26554/marshal_use_int64.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 28, 2012, 6:22 AM

Post #9 of 21 (127 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit [In reply to]

Serhiy Storchaka <storchaka [at] gmail> added the comment:

I can see yet another problem under the hood. Marshalling will output different data on platforms with a different representation of negative numbers. Worse still, these data are non-portable, written on one platform will be read incorrectly on the other. Mark, time for your inquisitor sword.

----------
nosy: +storchaka

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 28, 2012, 6:25 AM

Post #10 of 21 (127 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit [In reply to]

Changes by Serhiy Storchaka <storchaka [at] gmail>:

----------
nosy: +mark.dickinson

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 28, 2012, 6:38 AM

Post #11 of 21 (132 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit [In reply to]

Amaury Forgeot d'Arc <amauryfa [at] gmail> added the comment:

Are there really platforms which don't use two's complement?
(If there are, I'm not sure to want to share binary data with them. Same for EBCDIC)

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 28, 2012, 6:39 AM

Post #12 of 21 (127 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit [In reply to]

Martin v. Löwis <martin [at] v> added the comment:

Serhiy: this is safe to ignore.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 28, 2012, 10:25 AM

Post #13 of 21 (126 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit [In reply to]

Eric Snow added the comment:

> I suggest the opposite: never emit TYPE_INT64 at all. For
> compatibility, we can still support reading it in 3.3 (and drop
> that support in 3.4).

+1 essentially we maintain the status quo

----------
nosy: +eric.snow

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 28, 2012, 10:44 AM

Post #14 of 21 (126 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit [In reply to]

Roundup Robot added the comment:

New changeset 461e84fc8c60 by Martin v. Löwis in branch 'default':
Issue #15466: Stop using TYPE_INT64 in marshal,
http://hg.python.org/cpython/rev/461e84fc8c60

----------
nosy: +python-dev

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 28, 2012, 10:45 AM

Post #15 of 21 (125 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit [In reply to]

Changes by Eric Snow <ericsnowcurrently [at] gmail>:

----------
versions: +Python 3.3

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 28, 2012, 10:46 AM

Post #16 of 21 (124 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit [In reply to]

Amaury Forgeot d'Arc added the comment:

Removing TYPE_INT64 was indeed the most reasonable choice.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 28, 2012, 10:47 AM

Post #17 of 21 (129 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit [In reply to]

Changes by Martin v. Löwis <martin [at] v>:

----------
resolution: -> fixed
status: open -> closed

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 28, 2012, 10:49 AM

Post #18 of 21 (124 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit [In reply to]

Martin v. Löwis added the comment:

I put the complete removal of TYPE_INT64 into issue15480

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 28, 2012, 10:54 AM

Post #19 of 21 (129 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit [In reply to]

Eric Snow added the comment:

Looks like the Misc/NEWS entry got truncated. Also, does this change need a new PYC magic number (now in Lib/importlib/_bootstrap.py)?

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 28, 2012, 11:28 AM

Post #20 of 21 (126 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit [In reply to]

Martin v. Löwis added the comment:

For the truncation, see ec7267fa8b84. I don't see why a new pyc magic number should be necessary. Python continues to support the existing files.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 28, 2012, 12:57 PM

Post #21 of 21 (127 views)
Permalink

[issue15466] Python/importlib.h is different on 32bit and 64bit [In reply to]

Eric Snow added the comment:

> I don't see why a new pyc magic number should be necessary. Python
> continues to support the existing files.

Good point.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15466>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads