Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Bugs

[issue3026] mmap broken with large files on 64bit system

 

 

Python bugs RSS feed   Index | Next | Previous | View Threaded


report at bugs

Jun 1, 2008, 7:24 PM

Post #1 of 6 (378 views)
Permalink
[issue3026] mmap broken with large files on 64bit system

New submission from Matthew Mueller <donut [at] users>:

mmap on large files on 64 bit platforms in python >=2.5 returns some
sort of garbage. In 2.4 it would just throw an exception. Now I get
something like this (script runs md5.md5 on mmap object, and then runs
os.system md5sum for comparison):

This is python2.5 from Ubuntu 8.04 AMD64
/tmp$ python2.5 testbigfile.py
python mmap md5: 1230552d39b7c1751f86bae5205ec0c8
abe59e28c9a3f11b883f62c80a3833a5 *bigfile


This is python svn as of 20080601, compiled the on same system.
/tmp$ python2.6 testbigfile.py
testbigfile.py:5: DeprecationWarning: the md5 module is deprecated; use
hashlib instead
import md5
python mmap md5: 1230552d39b7c1751f86bae5205ec0c8
abe59e28c9a3f11b883f62c80a3833a5 *bigfile


Also note how the python md5 call returns immediately, not something you
would expect when md5ing 4GB of data.

----------
components: Extension Modules
files: testbigfile.py
messages: 67623
nosy: donut
severity: normal
status: open
title: mmap broken with large files on 64bit system
type: behavior
versions: Python 2.6
Added file: http://bugs.python.org/file10500/testbigfile.py

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue3026>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Jun 1, 2008, 7:29 PM

Post #2 of 6 (377 views)
Permalink
[issue3026] mmap broken with large files on 64bit system [In reply to]

Matthew Mueller <donut [at] users> added the comment:

Actually, I just realized that this might be a problem with md5 module
instead. Either way, something is busted.

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue3026>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Jun 4, 2008, 1:43 PM

Post #3 of 6 (354 views)
Permalink
[issue3026] mmap broken with large files on 64bit system [In reply to]

Changes by Ralf Schmitt <schmir [at] gmail>:


----------
nosy: +schmir

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue3026>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Jun 4, 2008, 2:16 PM

Post #4 of 6 (351 views)
Permalink
[issue3026] mmap broken with large files on 64bit system [In reply to]

Ralf Schmitt <schmir [at] gmail> added the comment:

I tested this with python 2.6 and can confirm the issue.
The problem is that unsigned int isn't big enough to hold the size of
the objects, but the size is downcasted to an unsigned int at several
places in _hashopenssl.c. All of these occurences of Py_SAFE_DOWNCAST
seem problematic to me (Py_SAFE_DOWNCAST(len, Py_ssize_t, unsigned int))

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue3026>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Jun 6, 2008, 8:13 AM

Post #5 of 6 (331 views)
Permalink
[issue3026] mmap broken with large files on 64bit system [In reply to]

Ralf Schmitt <schmir [at] gmail> added the comment:

the same bug also occurs when computing the md5 of a string larger than
2**32

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue3026>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Jun 11, 2008, 11:00 PM

Post #6 of 6 (310 views)
Permalink
[issue3026] mmap broken with large files on 64bit system [In reply to]

Changes by Georg Brandl <georg [at] python>:


----------
priority: -> critical
versions: +Python 3.0

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue3026>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Python bugs RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.