Mailing List Archive: Zope: Dev

Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]

Index | Next | Previous | View Threaded

stefan at epy

Jul 18, 2009, 5:08 AM

Post #1 of 20 (1762 views)
Permalink

Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed]

All failures appear to be due to the new box running Linux x86_64.
Tests still pass fine on the Mac.

Any experience with this? MemoryError in Acquisition? WTH?

Stefan

On 18.07.2009, at 14:00, Zope Tests Summarizer wrote:

> Summary of messages to the zope-tests list.
> Period Fri Jul 17 12:00:00 2009 UTC to Sat Jul 18 12:00:00 2009 UTC.
> There were 8 messages: 8 from Zope Tests.
>
>
> Test failures
> -------------
>
> Subject: FAILED (failures=1) : Zope-2.12 Python-2.6.2 : Linux
> From: Zope Tests
> Date: Fri Jul 17 20:50:00 EDT 2009
> URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012082.html
>
> Subject: FAILED (failures=1) : Zope-2.12-alltests Python-2.4.6 : Linux
> From: Zope Tests
> Date: Fri Jul 17 20:52:00 EDT 2009
> URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012083.html
>
> Subject: FAILED (failures=7) : Zope-2.12-alltests Python-2.6.2 : Linux
> From: Zope Tests
> Date: Fri Jul 17 20:54:01 EDT 2009
> URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012084.html
>
> Subject: FAILED (failures=1) : Zope-trunk Python-2.6.2 : Linux
> From: Zope Tests
> Date: Fri Jul 17 20:56:01 EDT 2009
> URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012085.html
>
> Subject: FAILED (failures=7) : Zope-trunk-alltests Python-2.6.2 :
> Linux
> From: Zope Tests
> Date: Fri Jul 17 20:58:01 EDT 2009
> URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012086.html

--
Stefan H. Holek
stefan [at] epy

_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

tseaver at palladion

Jul 21, 2009, 9:40 AM

Post #2 of 20 (1690 views)
Permalink

Re: Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed] [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stefan H. Holek wrote:
> All failures appear to be due to the new box running Linux x86_64.
> Tests still pass fine on the Mac.
>
> Any experience with this? MemoryError in Acquisition? WTH?

I can reproduce the OFS.tests.testRange failures on my x86_64 box:

$ uname -a
Linux mred 2.6.27-14-generic #1 SMP Wed Apr 15 19:29:46 UTC \
2009 x86_64 GNU/Linux
$ cd /tmp
$ svn co svn+ssh://svn.zope.org/repos/main/Zope/trunk Zope-trunk
$ cd Zope-trunk
$ /opt/Python-2.6.2/bin/python bootstrap.py
...
$ bin/buildout
...
$ bin/test --all
...
Failure in test testMultipleRangesBigFileOutOfOrder
(OFS.tests.testRanges.TestRequestRange)
Traceback (most recent call last):
File "/opt/Python-2.6.2/lib/python2.6/unittest.py", line 279, in run
testMethod()
File "/home/tseaver/projects/Zope-trunk/src/OFS/tests/testRanges.py",
line 332, in testMultipleRangesBigFileOutOfOrder
(70000, 80001)])
File "/home/tseaver/projects/Zope-trunk/src/OFS/tests/testRanges.py",
line 209, in expectMultipleRanges
str(len(body)), rsp.getHeader('content-length')))
File "/opt/Python-2.6.2/lib/python2.6/unittest.py", line 321, in
failIf
if expr: raise self.failureException, msg
AssertionError: Incorrect Content-Length is set! Expected 20425,
got 20426.

I don't grok the range support code at all: probably Martijn Pieters is
the only person in the world who does. The tests are quite obscure to
me, so I can't diagnose even whether the failure is a testing artifact
or a real bug.

I see another failure in a test Hanno recently added:

Failure in test test_datetime
(Products.PluginIndexes.DateRangeIndex.tests.test_DateRangeIndex.DRI_Tests)
Traceback (most recent call last):
File "/opt/Python-2.6.2/lib/python2.6/unittest.py", line 279, in run
testMethod()
File
"/home/tseaver/projects/Zope-trunk/src/Products/PluginIndexes/DateRangeIndex/tests/test_DateRangeIndex.py",
line 155, in test_datetime
self.assertEqual(work.getEntryForObject(0), (20790915, 20790990))
File "/opt/Python-2.6.2/lib/python2.6/unittest.py", line 350, in
failUnlessEqual
(msg or '%r != %r' % (first, second))
AssertionError: (20791275, 20791350) != (20790915, 20790990)

This one looks to be the result of using "naive" datetime objects while
expecting them to be converted to hard-wired integers, so that tests
which pass on Hanno's machine fail on mine, because of different timezones.

Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 tseaver [at] palladion
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKZe9j+gerLs4ltQ4RAl4nAKDJQyWHdnlEEQv1zbQJeJiR7gGuTACfQxDi
n5GO/tP+jpRhqUddn2CbaKg=
=RdUR
-----END PGP SIGNATURE-----

_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

lists at zopyx

Jul 23, 2009, 9:14 AM

Post #3 of 20 (1669 views)
Permalink

Re: Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed] [In reply to]

I can reproduce the failures in ExtensionClass and also get some weird
errors while trying to buildout the Acquisition package:

ajung [at] blackmoo:/develop/repository/svn.zope.org/Acquisition/trunk>
bin/buildout
Develop: '/data/develop/repository/svn.zope.org/Acquisition/trunk/.'
Unused options for buildout: 'download-directory'.
Installing interpreter.
While:
Installing interpreter.

An internal error occured due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
File
"/tmp/tmpt2EP2J/zc.buildout-1.3.0-py2.6.egg/zc/buildout/buildout.py",
line 1519, in main
File
"/tmp/tmpt2EP2J/zc.buildout-1.3.0-py2.6.egg/zc/buildout/buildout.py",
line 473, in install
File
"/tmp/tmpt2EP2J/zc.buildout-1.3.0-py2.6.egg/zc/buildout/buildout.py",
line 1101, in _call
File
"/home/ajung/.buildout/eggs/zc.recipe.egg-1.2.2-py2.6.egg/zc/recipe/egg/egg.py",
line 169, in install
relative_paths=self._relative_paths,
File
"/tmp/tmpt2EP2J/zc.buildout-1.3.0-py2.6.egg/zc/buildout/easy_install.py", line
964, in scripts
File
"/tmp/tmpt2EP2J/zc.buildout-1.3.0-py2.6.egg/zc/buildout/easy_install.py", line
1111, in _pyscript
IOError: [Errno 26] Text file busy:
'/data/develop/repository/svn.zope.org/Acquisition/trunk/bin/python'

Both issues are severe and are a blocker for the next release.

Andreas

On 18.07.09 14:08, Stefan H. Holek wrote:
> All failures appear to be due to the new box running Linux x86_64.
> Tests still pass fine on the Mac.
>
> Any experience with this? MemoryError in Acquisition? WTH?
>
> Stefan
>
>
> On 18.07.2009, at 14:00, Zope Tests Summarizer wrote:
>
>
>> Summary of messages to the zope-tests list.
>> Period Fri Jul 17 12:00:00 2009 UTC to Sat Jul 18 12:00:00 2009 UTC.
>> There were 8 messages: 8 from Zope Tests.
>>
>>
>> Test failures
>> -------------
>>
>> Subject: FAILED (failures=1) : Zope-2.12 Python-2.6.2 : Linux
>> From: Zope Tests
>> Date: Fri Jul 17 20:50:00 EDT 2009
>> URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012082.html
>>
>> Subject: FAILED (failures=1) : Zope-2.12-alltests Python-2.4.6 : Linux
>> From: Zope Tests
>> Date: Fri Jul 17 20:52:00 EDT 2009
>> URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012083.html
>>
>> Subject: FAILED (failures=7) : Zope-2.12-alltests Python-2.6.2 : Linux
>> From: Zope Tests
>> Date: Fri Jul 17 20:54:01 EDT 2009
>> URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012084.html
>>
>> Subject: FAILED (failures=1) : Zope-trunk Python-2.6.2 : Linux
>> From: Zope Tests
>> Date: Fri Jul 17 20:56:01 EDT 2009
>> URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012085.html
>>
>> Subject: FAILED (failures=7) : Zope-trunk-alltests Python-2.6.2 :
>> Linux
>> From: Zope Tests
>> Date: Fri Jul 17 20:58:01 EDT 2009
>> URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012086.html
>>
>
> --
> Stefan H. Holek
> stefan [at] epy
>
>
>
>
> _______________________________________________
> Zope-Dev maillist - Zope-Dev [at] zope
> http://mail.zope.org/mailman/listinfo/zope-dev
> ** No cross posts or HTML encoding! **
> (Related lists -
> http://mail.zope.org/mailman/listinfo/zope-announce
> http://mail.zope.org/mailman/listinfo/zope )
>

--
ZOPYX Ltd. & Co KG \ ZOPYX & Friends
Charlottenstr. 37/1 \ The experts for your Python, Zope and
D-72070 T�bingen \ Plone projects
www.zopyx.com, info [at] zopyx \ www.zopyx.de/friends, friends [at] zopyx
------------------------------------------------------------------------
E-Publishing, Python, Zope & Plone development, Consulting

Attachments:

lists.vcf (0.31 KB)

hanno at hannosch

Jul 25, 2009, 4:57 AM

Post #4 of 20 (1648 views)
Permalink

Re: Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed] [In reply to]

On Sat, Jul 18, 2009 at 2:08 PM, Stefan H. Holek<stefan [at] epy> wrote:
> All failures appear to be due to the new box running Linux x86_64.
> Tests still pass fine on the Mac.
>
> Any experience with this? MemoryError in Acquisition? WTH?

I don't have any experience with either 64-bit nor can I write any C.
And it seems obvious that these kind of crashes are caused by C
extension code.

I kind of suspect that we are seeing the results of
http://www.python.org/dev/peps/pep-0353 though.

>From what I understand we need to change code in C extension modules
following the instructions in that PEP, to be able to run code
reliably on 64-bit platforms. To my knowledge nobody changed any C
code in Zope yet to do so.

The other interesting bit to this is, that it seems doable but hard to
do these changes while preserving compatibility with Python 2.4. These
changes along with the definition of "Py_ssize_t" were introduced into
Python 2.5.

At this point I think we need to declare 64-bit platforms as
unsupported. Hopefully people with the right kind of knowledge would
like to step up and start working on these issues.

Hanno
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

shane at hathawaymix

Jul 25, 2009, 11:01 AM

Post #5 of 20 (1635 views)
Permalink

Re: Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed] [In reply to]

Hanno Schlichting wrote:
> I kind of suspect that we are seeing the results of
> http://www.python.org/dev/peps/pep-0353 though.

That is very likely. BTW, that PEP links to a handy tool that reveals
most 64 bit portability issues in Python-oriented C code.

>>From what I understand we need to change code in C extension modules
> following the instructions in that PEP, to be able to run code
> reliably on 64-bit platforms. To my knowledge nobody changed any C
> code in Zope yet to do so.

We fixed the C portability problems in Zope 3 a while back, but it seems
we didn't fix the extensions that are specific to Zope 2.

> The other interesting bit to this is, that it seems doable but hard to
> do these changes while preserving compatibility with Python 2.4. These
> changes along with the definition of "Py_ssize_t" were introduced into
> Python 2.5.

Preprocessor macros solve that fairly cleanly, actually:

#ifndef Py_ssize_t
#define Py_ssize_t int
#endif

> At this point I think we need to declare 64-bit platforms as
> unsupported. Hopefully people with the right kind of knowledge would
> like to step up and start working on these issues.

I just checked in some code that solves the portability problem in
zope.index, although I haven't created a release. It looks like the
Acquisition module needs some 64 bit TLC.

Shane
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

hanno at hannosch

Jul 25, 2009, 5:55 PM

Post #6 of 20 (1622 views)
Permalink

Re: Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed] [In reply to]

On Sat, Jul 25, 2009 at 8:01 PM, Shane Hathaway<shane [at] hathawaymix> wrote:
> Hanno Schlichting wrote:
>>
>> I kind of suspect that we are seeing the results of
>> http://www.python.org/dev/peps/pep-0353 though.
>
> That is very likely. BTW, that PEP links to a handy tool that reveals most
> 64 bit portability issues in Python-oriented C code.

Right, I've given that a shot and it reports plenty of problems for
Acquisition.c for example. In addition to the general compiler
warnings it spits out, which don't tell me much.

>> From what I understand we need to change code in C extension modules
>> following the instructions in that PEP, to be able to run code
>> reliably on 64-bit platforms. To my knowledge nobody changed any C
>> code in Zope yet to do so.
>
> We fixed the C portability problems in Zope 3 a while back, but it seems we
> didn't fix the extensions that are specific to Zope 2.

How thorough has that been? I can see the Py_ssize_t in some of the c
files in zope.* packages, but looking for example at:

http://svn.zope.org/zope.proxy/trunk/src/zope/proxy/_zope_proxy_proxy.c?rev=95468&view=markup

I see a stuff like this:

PyObject *
WrapperType_Lookup(PyTypeObject *type, PyObject *name)
{
int i, n;

[...]

n = PyTuple_GET_SIZE(mro) - 1;

[...]
}

where PyTuple_GET_SIZE is defined to return a Py_ssize_t and not an int.

Either I misunderstand something here or I was just "lucky" to find
something like this as my first test.

>> The other interesting bit to this is, that it seems doable but hard to
>> do these changes while preserving compatibility with Python 2.4. These
>> changes along with the definition of "Py_ssize_t" were introduced into
>> Python 2.5.
>
> Preprocessor macros solve that fairly cleanly, actually:
>
> #ifndef Py_ssize_t
> #define Py_ssize_t int
> #endif

Ah ok. I was more concerned about the function pointer changes.
intargfunc to ssizeargfunc and so on. I wasn't sure if these are as
easy to redefine as simple data types.

>> At this point I think we need to declare 64-bit platforms as
>> unsupported. Hopefully people with the right kind of knowledge would
>> like to step up and start working on these issues.
>
> I just checked in some code that solves the portability problem in
> zope.index, although I haven't created a release. It looks like the
> Acquisition module needs some 64 bit TLC.

Cool! I kinda suspect that specific code is also somewhere in Zope2 in
the PluginIndexes where it has been ported from recently.

Hanno
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

marius at gedmin

Jul 27, 2009, 12:18 PM

Post #7 of 20 (1609 views)
Permalink

Re: Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed] [In reply to]

On Sun, Jul 26, 2009 at 02:55:58AM +0200, Hanno Schlichting wrote:
> On Sat, Jul 25, 2009 at 8:01 PM, Shane Hathaway<shane [at] hathawaymix> wrote:
> > Hanno Schlichting wrote:
> >>
> >> I kind of suspect that we are seeing the results of
> >> http://www.python.org/dev/peps/pep-0353 though.
> >
> > That is very likely. �BTW, that PEP links to a handy tool that reveals most
> > 64 bit portability issues in Python-oriented C code.
>
> Right, I've given that a shot and it reports plenty of problems for
> Acquisition.c for example. In addition to the general compiler
> warnings it spits out, which don't tell me much.
>
> >> From what I understand we need to change code in C extension modules
> >> following the instructions in that PEP, to be able to run code
> >> reliably on 64-bit platforms. To my knowledge nobody changed any C
> >> code in Zope yet to do so.
> >
> > We fixed the C portability problems in Zope 3 a while back, but it seems we
> > didn't fix the extensions that are specific to Zope 2.
>
> How thorough has that been?

I'm not familiar with the source code of the extension modules, and I
haven't reviewed them for possible 64-bit issues. But we've been
running a Zope 3 app in production on 64-bit Linux for almost two years
now. No 64-bit-related problems bit us so far.

There's a buildbot for Zope 3.4 packages that runs their tests on 32 and
64 bit platforms on Python 2.4 and 2.5: http://zope3.pov.lt/buildbot/

Currently ZODB tests are disabled on it because they used to have
irritating spurious failures that seem to be related to the speed of the
machine rather than its platform.

I believe there's a similar buildbot for Zope 3.5 KGS out there
somewhere.

> I can see the Py_ssize_t in some of the c
> files in zope.* packages, but looking for example at:
>
> http://svn.zope.org/zope.proxy/trunk/src/zope/proxy/_zope_proxy_proxy.c?rev=95468&view=markup
>
> I see a stuff like this:
>
> PyObject *
> WrapperType_Lookup(PyTypeObject *type, PyObject *name)
> {
> int i, n;
>
> [...]
>
> n = PyTuple_GET_SIZE(mro) - 1;
>
> [...]
> }
>
> where PyTuple_GET_SIZE is defined to return a Py_ssize_t and not an int.

It shouldn't be a problem until you start seeing tuples with 2 billion
items in them, no?

> Either I misunderstand something here or I was just "lucky" to find
> something like this as my first test.

Marius Gedminas
--
http://pov.lt/ -- Zope 3 consulting and development

Attachments:

signature.asc (0.18 KB)

lists at zopyx

Aug 1, 2009, 9:18 AM

Post #8 of 20 (1519 views)
Permalink

Re: Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed] [In reply to]

On 25.07.09 13:57, Hanno Schlichting wrote:
>
> At this point I think we need to declare 64-bit platforms as
> unsupported. Hopefully people with the right kind of knowledge would
> like to step up and start working on these issues.
This is really not an option. Zope 2.12 must run on 32 and 64 bit
systems without issues. Since this is a major blocker I will defer
the next release (scheduled for this week) until we have fixed the issue.

Andreas

Attachments:

lists.vcf (0.31 KB)

tseaver at palladion

Aug 2, 2009, 11:30 AM

Post #9 of 20 (1498 views)
Permalink

Re: Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed] [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Andreas Jung wrote:
> On 25.07.09 13:57, Hanno Schlichting wrote:
>> At this point I think we need to declare 64-bit platforms as
>> unsupported. Hopefully people with the right kind of knowledge would
>> like to step up and start working on these issues.
> This is really not an option. Zope 2.12 must run on 32 and 64 bit
> systems without issues. Since this is a major blocker I will defer
> the next release (scheduled for this week) until we have fixed the issue.

I think I have fixed the ExtensionClass and Acquisition problems for
64-bit machines: I released new eggs to PyPI for them:

- - http://pypi.python.org/pypi/ExtensionClass/2.11.3

- - http://pypi.python.org/pypi/Acquisition/2.12.2

Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 tseaver [at] palladion
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKddsu+gerLs4ltQ4RArPiAJ0USyr/xHsBthWCdG4GAfpan1tirQCgh711
eu6hqMsep/VtglUXSshIgoY=
=6amd
-----END PGP SIGNATURE-----

_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

hanno at hannosch

Aug 2, 2009, 11:45 AM

Post #10 of 20 (1496 views)
Permalink

Re: Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed] [In reply to]

On Sun, Aug 2, 2009 at 8:30 PM, Tres Seaver<tseaver [at] palladion> wrote:
> I think I have fixed the ExtensionClass and Acquisition problems for
> 64-bit machines: I released new eggs to PyPI for them:

Awesome!

You wouldn't be interested to look at the C code in Persistence and
Zope2 itself? ;-)

Hanno
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

lists at zopyx

Aug 2, 2009, 11:50 AM

Post #11 of 20 (1489 views)
Permalink

Re: Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed] [In reply to]

On 02.08.09 20:30, Tres Seaver wrote:
> Andreas Jung wrote:
> > On 25.07.09 13:57, Hanno Schlichting wrote:
> >> At this point I think we need to declare 64-bit platforms as
> >> unsupported. Hopefully people with the right kind of knowledge would
> >> like to step up and start working on these issues.
> > This is really not an option. Zope 2.12 must run on 32 and 64 bit
> > systems without issues. Since this is a major blocker I will defer
> > the next release (scheduled for this week) until we have fixed the
> issue.
>
> I think I have fixed the ExtensionClass and Acquisition problems for
> 64-bit machines: I released new eggs to PyPI for them:
>
> - http://pypi.python.org/pypi/ExtensionClass/2.11.3
>
> - http://pypi.python.org/pypi/Acquisition/2.12.2
>

Thanks Tres - the issue seems to be resolved.

Right now there is only one failure with the http ranges tests
(Martijn, can you check this?) and a complete failure of the zope.index
tests (see other mail).

Andreas

Attachments:

lists.vcf (0.31 KB)

tseaver at palladion

Aug 2, 2009, 2:31 PM

Post #12 of 20 (1490 views)
Permalink

Re: Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed] [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hanno Schlichting wrote:
> On Sun, Aug 2, 2009 at 8:30 PM, Tres Seaver<tseaver [at] palladion> wrote:
>> I think I have fixed the ExtensionClass and Acquisition problems for
>> 64-bit machines: I released new eggs to PyPI for them:
>
> Awesome!
>
> You wouldn't be interested to look at the C code in Persistence and
> Zope2 itself? ;-)

I have updated the cAccessControl and Record extensions for conformance
with the PEP.

I don't think the Persistence code needs anything: AFAICT, it is
already good to go.

Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 tseaver [at] palladion
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKdgWp+gerLs4ltQ4RApPJAJ4neMSFrdUaLomc5fnImtkuDQVkwgCgsZNL
1iBevGRfBSM5m1OFJILnqp0=
=oXK6
-----END PGP SIGNATURE-----

_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

mj at zopatista

Aug 3, 2009, 12:04 AM

Post #13 of 20 (1485 views)
Permalink

Re: Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed] [In reply to]

On Tue, Jul 21, 2009 at 18:40, Tres Seaver <tseaver [at] palladion> wrote:
> File "/home/tseaver/projects/Zope-trunk/src/OFS/tests/testRanges.py",
> line 332, in testMultipleRangesBigFileOutOfOrder
> (70000, 80001)])
> File "/home/tseaver/projects/Zope-trunk/src/OFS/tests/testRanges.py",
> line 209, in expectMultipleRanges
> str(len(body)), rsp.getHeader('content-length')))
> File "/opt/Python-2.6.2/lib/python2.6/unittest.py", line 321, in
> failIf
> if expr: raise self.failureException, msg
> AssertionError: Incorrect Content-Length is set! Expected 20425,
> got 20426.
>
> I don't grok the range support code at all: probably Martijn Pieters is
> the only person in the world who does. The tests are quite obscure to
> me, so I can't diagnose even whether the failure is a testing artifact
> or a real bug.

Here's my response to Tres; for some reason my mail reader didn't
include zope-dev. I can add now that I'll be at a sprint this weekend
where I'll probably have time to take a look myself:

----8<------------------

Sorry you find them obscure; I followed unit test expectations as best
as I could.

The test that fails does the following:

1) Create a big file (uploadBigFile, enough random data to be more
than one pdata chunk)
2) Request for multiple HTTP ranges (slices) of the file to be sent.
These slices should be returned a) in the same order, b) of the
correct length, and of course c) the correct slice of the big file
created in 1.
In this case line 330 specifies that bytes 10-15 (inclusive), the last
10000 bytes, and bytes 70000-80000 (inclusive) are are to be returned.
The call to expectMultipleRanges includes the expected slices (python
syntax, so end value is exclusive).
3) On line 209 of expectMultipleRanges it then tests if the resulting
response has the correct content length header for the whole response
set. The failure reported is that the Content-Length header reports 1
byte more than what was actually sent, so the body was 20425 bytes
long, while the content-header claims it would be 20426 bytes.

Because this is a multi-part range request, Image.py line 277 and
onwards handle this request. The response ends up looking something
like this:

Content-length: size we pre-calculate
Content-type: multipart/byteranges; boundary="boundarystring"
More headers

--boundarystring
Content-type: image/whatever
Content-range: bytes start-end/total-file-size

.... data ...

--boundarystring
... Another section
... etc.

--boundarystring--

There are 2 ways the failure could be caused.

Something could go wrong with the size calculation on lines 280-287;
the total body length is the length of the range data plus the length
of the headers per multi-part plus the length of the MIME boundaries
between them. There is thus a length per part plus the length of the
last boundary.

Or something goes wrong when writing the range data in lines 309-358.
Because there is only one byte difference between what was calculated
on lines 280-287 and what is produced here this is probably a newline
somewhere.

I do find it puzzling this only fails on 64 bit platforms. It could be
that the pdata chunk handling is borken and has a off-by-one error
when running on 64-bit platforms.

Hope this clarifies things a little. I have little time right now to
chase this myself right now, sorry!

----8<------------------

--
Martijn Pieters
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

lists at zopyx

Aug 3, 2009, 5:21 AM

Post #14 of 20 (1471 views)
Permalink

Re: Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed] [In reply to]

On 03.08.09 09:04, Martijn Pieters wrote:
> On Tue, Jul 21, 2009 at 18:40, Tres Seaver <tseaver [at] palladion> wrote:
>
>> File "/home/tseaver/projects/Zope-trunk/src/OFS/tests/testRanges.py",
>> line 332, in testMultipleRangesBigFileOutOfOrder
>> (70000, 80001)])
>> File "/home/tseaver/projects/Zope-trunk/src/OFS/tests/testRanges.py",
>> line 209, in expectMultipleRanges
>> str(len(body)), rsp.getHeader('content-length')))
>> File "/opt/Python-2.6.2/lib/python2.6/unittest.py", line 321, in
>> failIf
>> if expr: raise self.failureException, msg
>> AssertionError: Incorrect Content-Length is set! Expected 20425,
>> got 20426.
>>
>> I don't grok the range support code at all: probably Martijn Pieters is
>> the only person in the world who does. The tests are quite obscure to
>> me, so I can't diagnose even whether the failure is a testing artifact
>> or a real bug.
>>
> Here's my response to Tres; for some reason my mail reader didn't
> include zope-dev. I can add now that I'll be at a sprint this weekend
> where I'll probably have time to take a look myself:

Oki.

Tnx,
Andreas

Attachments:

lists.vcf (0.32 KB)

mj at zopatista

Aug 7, 2009, 6:55 AM

Post #15 of 20 (1343 views)
Permalink

Re: Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed] [In reply to]

On Mon, Aug 3, 2009 at 08:04, Martijn Pieters <mj [at] zopatista> wrote:

>> AssertionError: Incorrect Content-Length is set! Expected 20425,
>> got 20426.
>>
>> I don't grok the range support code at all: probably Martijn Pieters is
>> the only person in the world who does. The tests are quite obscure to
>> me, so I can't diagnose even whether the failure is a testing artifact
>> or a real bug.
>
> Here's my response to Tres; for some reason my mail reader didn't
> include zope-dev. I can add now that I'll be at a sprint this weekend
> where I'll probably have time to take a look myself:

I have found the problem, but do not understand where exactly the error occurs.

The problem is caused by an acquisition wrapped object with a
__getslice__ method get the wrong indices passed in when the end
parameter is ommitted:

data[start:]

Image.Pdata classes have a __getslice__ method, and when data is
acquisition wrapped, that method gets (start, -1) passed in on 64-bit
platforms, while on my Macbook (start, sys.maxint) is passed in
instead. Obviously this means that the slice returns not everything
from 'start' to the end, but from 'start' until the one-but-last byte.

However, as far as I can tell cAcquisition.c doesn't touch the start
and end values. Anyone want to investigate why this goes wrong?

Here is the simple test case:

from Acquisition import Implicit

class Root(Implicit):
pass

class Slicer(Implicit):
def __getslice__(self, start, end):
return [start, end]

root = Root()
slicer = Slicer().__of__(root)
print slicer[1:]

Run this as bin/zopepy acquisition.py and verify that [1, -1] is
printed on 64-bit platforms as opposed to [1, 2147483647] on 32-bit
platforms.

--
Martijn Pieters
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

mj at zopatista

Aug 7, 2009, 7:08 AM

Post #16 of 20 (1341 views)
Permalink

Re: Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed] [In reply to]

On Fri, Aug 7, 2009 at 14:55, Martijn Pieters <mj [at] zopatista> wrote:
> Here is the simple test case:
>
> from Acquisition import Implicit
>
> class Root(Implicit):
> pass
>
> class Slicer(Implicit):
> def __getslice__(self, start, end):
> return [start, end]
>
> root = Root()
> slicer = Slicer().__of__(root)
> print slicer[1:]
>
> Run this as bin/zopepy acquisition.py and verify that [1, -1] is
> printed on 64-bit platforms as opposed to [1, 2147483647] on 32-bit
> platforms.

Further datapoints, the following extra lines for the above script
give more context to compare:

import sys
print slicer[sys.maxint:]
print slicer[sys.maxint-1:]
print Slicer()[1:]

So we pass in sys.maxint and sys.maxint-1 as start values, and also
run the Slicer without acquisition wrapping to demonstrate that the
wrapper is the culprit, not python itself. Python passes in sys.maxint
for the end value for the slice if you omit it, and these high values
overflow somewhere in the wraper.

--
Martijn Pieters
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

mj at zopatista

Aug 7, 2009, 7:26 AM

Post #17 of 20 (1341 views)
Permalink

Re: Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed] [In reply to]

On Fri, Aug 7, 2009 at 14:55, Martijn Pieters <mj [at] zopatista> wrote:
> The problem is caused by an acquisition wrapped object with a
> __getslice__ method get the wrong indices passed in when the end
> parameter is ommitted:
>
> data[start:]
>
> Image.Pdata classes have a __getslice__ method, and when data is
> acquisition wrapped, that method gets (start, -1) passed in on 64-bit
> platforms, while on my Macbook (start, sys.maxint) is passed in
> instead. Obviously this means that the slice returns not everything
> from 'start' to the end, but from 'start' until the one-but-last byte.
>
> However, as far as I can tell cAcquisition.c doesn't touch the start
> and end values. Anyone want to investigate why this goes wrong?

The following checkin fixed this particular problem:

http://svn.zope.org/Acquisition/trunk/src/Acquisition/_Acquisition.c?rev=102564&view=rev

The acquisition slice wrapper accepted Py_ssize_t arguments, but then
passed these of as C ints (format string 'i'). Changing the format
string to 'n' (Py_ssize_t) fixed *this particular case*. Likely more
such fixes must be made.

In any case, the HTTP range test no longer fails with this fix.

--
Martijn Pieters
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

hanno at hannosch

Aug 8, 2009, 5:19 AM

Post #18 of 20 (1296 views)
Permalink

Re: Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed] [In reply to]

On Fri, Aug 7, 2009 at 3:26 PM, Martijn Pieters<mj [at] zopatista> wrote:
> The following checkin fixed this particular problem:
>
> http://svn.zope.org/Acquisition/trunk/src/Acquisition/_Acquisition.c?rev=102564&view=rev
>
> The acquisition slice wrapper accepted Py_ssize_t arguments, but then
> passed these of as C ints (format string 'i'). Changing the format
> string to 'n' (Py_ssize_t) fixed *this particular case*. Likely more
> such fixes must be made.

One thing to note here is that the above change is indeed only of of
probably many that need to be made to support 64-bit platforms
properly. The other thing to note is that the change now introduces a
hard requirement on Python 2.5 or later.

Neither Martijn nor me could figure out a way to make the function
calls to Build have different format identifier arguments dependent on
the Python version. On Python 2.4 there is only "i" for integer and
2.5 introduces "n" for Py_ssize_t. Since this is an argument to a
function showing up in various combinations of "nn", "nO" and others,
there seems to be no easy way to make this work.

Someone with more knowledge about pre-processor tricks might come up
with a solution to this, otherwise we will have to choose between
dropping 64-bit support or dropping the unofficial Python 2.4 support.

Hanno
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

lists at zopyx

Aug 8, 2009, 5:21 AM

Post #19 of 20 (1305 views)
Permalink

Re: Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed] [In reply to]

On 08.08.09 14:19, Hanno Schlichting wrote:
> On Fri, Aug 7, 2009 at 3:26 PM, Martijn Pieters<mj [at] zopatista> wrote:
>
>> The following checkin fixed this particular problem:
>>
>> http://svn.zope.org/Acquisition/trunk/src/Acquisition/_Acquisition.c?rev=102564&view=rev
>>
>> The acquisition slice wrapper accepted Py_ssize_t arguments, but then
>> passed these of as C ints (format string 'i'). Changing the format
>> string to 'n' (Py_ssize_t) fixed *this particular case*. Likely more
>> such fixes must be made.
>>
> One thing to note here is that the above change is indeed only of of
> probably many that need to be made to support 64-bit platforms
> properly. The other thing to note is that the change now introduces a
> hard requirement on Python 2.5 or later.
>
> Neither Martijn nor me could figure out a way to make the function
> calls to Build have different format identifier arguments dependent on
> the Python version. On Python 2.4 there is only "i" for integer and
> 2.5 introduces "n" for Py_ssize_t. Since this is an argument to a
> function showing up in various combinations of "nn", "nO" and others,
> there seems to be no easy way to make this work.
>
> Someone with more knowledge about pre-processor tricks might come up
> with a solution to this, otherwise we will have to choose between
> dropping 64-bit support or dropping the unofficial Python 2.4 support.
>

Dropping 64-bit support is no option, dropping Python 2.4 support is :)

Adreas

Attachments:

lists.vcf (0.32 KB)

mj at zopatista

Aug 8, 2009, 6:35 AM

Post #20 of 20 (1303 views)
Permalink

Re: Linux x86_64 [Was: Zope Tests: 3 OK, 5 Failed] [In reply to]

On Sat, Aug 8, 2009 at 13:19, Hanno Schlichting <hanno [at] hannosch> wrote:
> One thing to note here is that the above change is indeed only of of
> probably many that need to be made to support 64-bit platforms
> properly. The other thing to note is that the change now introduces a
> hard requirement on Python 2.5 or later.
>
> Neither Martijn nor me could figure out a way to make the function
> calls to Build have different format identifier arguments dependent on
> the Python version. On Python 2.4 there is only "i" for integer and
> 2.5 introduces "n" for Py_ssize_t. Since this is an argument to a
> function showing up in various combinations of "nn", "nO" and others,
> there seems to be no easy way to make this work.
>
> Someone with more knowledge about pre-processor tricks might come up
> with a solution to this, otherwise we will have to choose between
> dropping 64-bit support or dropping the unofficial Python 2.4 support.

It turned out to be really simple as consecutive string literals are
concatenated into one after macros are expanded. The following
changeset should make Acquisition python 2.4 compatible again and also
properly 64-bit capable:

http://svn.zope.org/Acquisition/trunk/src/Acquisition/_Acquisition.c?rev=102577&view=rev

To summarize: when passing Py_BuildValue a Py_ssize_t value to build a
python int, you need to use the 'n' format string instead of 'i'. But
because 'n' and Py_ssize_t are both defined as of python 2.5, for C
code to work in python 2.4 you need to redefine these two back to 'i'
and int.

--
Martijn Pieters
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads