Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Bugs

[issue7244] itertools.zip_longest behaves strangely with an iterable class

 

 

Python bugs RSS feed   Index | Next | Previous | View Threaded


report at bugs

Oct 31, 2009, 2:07 AM

Post #1 of 15 (513 views)
Permalink
[issue7244] itertools.zip_longest behaves strangely with an iterable class

New submission from Daniel Urban <urban.dani+py [at] gmail>:

I'm trying to write an iterable class, and it behaves strangely with
itertools.zip_longest. The following example demonstrates this:

class Repeater: # this class is similar to itertools.repeat
def __init__(self, o, t):
self.o = o
self.t = int(t)
def __iter__(self): # its iterator is itself
return self
def __next__(self):
if self.t > 0:
self.t -= 1
return self.o
else:
raise StopIteration

(Of course this is a trivial class, which could be substituted with
itertools.repeat, but I wanted to keep it simple for this example.)

The following code shows my problem:
>>> r1 = Repeater(1, 3)
>>> r2 = Repeater(2, 4)
>>> for i, j in zip_longest(r1, r2, fillvalue=0):
... print(i, j)
...
1 2
1 2
1 2
0Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "zip_longest_test_case.py", line 30, in __next__
raise StopIteration
StopIteration
>>>

It seems, that zip_longest lets through the StopIteration exception,
which it shouldn't.

The strange thing is, that if I use the python implementation of
zip_longest, as it is in the documentation [1], I get the expected
result:

# zip_longest as it is in the documentation:
def zip_longest(*args, fillvalue=None):
# zip_longest('ABCD', 'xy', fillvalue='-') --> Ax By C- D-
def sentinel(counter = ([fillvalue]*(len(args)-1)).pop):
yield counter() # yields the fillvalue, or raises IndexError
fillers = repeat(fillvalue)
iters = [chain(it, sentinel(), fillers) for it in args]
try:
for tup in zip(*iters):
yield tup
except IndexError:
pass

Test code again:
>>> r1 = Repeater(1, 3)
>>> r2 = Repeater(2, 4)
>>> for i, j in zip_longest(r1, r2, fillvalue=0):
... print(i, j)
...
1 2
1 2
1 2
0 2

I would think, that this is the expected behaviour.

Also, Matthew Dixon Cowles discovered, that if using list(), the C
implementation of itertools.zip_longest also works fine:

>>> r1=Repeater(1,3)
>>> r2=Repeater(2,5)
>>> list(itertools.zip_longest(r1,r2,fillvalue=0))
[(1, 2), (1, 2), (1, 2), (0, 2), (0, 2)]

This is strange, and I think it really shouldn't work this way.
(Thanks for Matthew Dixon Cowles' help on the python-help mailing list.)

I'm attaching a test script, which tries all 4 variations (library
zip_longest with and without list(), and the documentation's zip_longest
impplementation with and without list()).

And another thing: it works fine in 2.6.4 (with izip_longest).

----------
components: Extension Modules
files: zip_longest_test_case.py
messages: 94746
nosy: durban
severity: normal
status: open
title: itertools.zip_longest behaves strangely with an iterable class
type: behavior
versions: Python 3.1
Added file: http://bugs.python.org/file15238/zip_longest_test_case.py

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7244>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Oct 31, 2009, 2:52 AM

Post #2 of 15 (498 views)
Permalink
[issue7244] itertools.zip_longest behaves strangely with an iterable class [In reply to]

Hirokazu Yamamoto <ocean-city [at] m2> added the comment:

I saw strange thing with following code + release26-maint/trunk.

from itertools import *

class Repeater(object): # this class is similar to itertools.repeat
def __init__(self, o, t):
self.o = o
self.t = int(t)
def __iter__(self): # its iterator is itself
return self
def next(self):
if self.t > 0:
self.t -= 1
return self.o
else:
raise StopIteration

r1 = Repeater(1, 3)
r2 = Repeater(2, 4)
for i, j in izip_longest(r1, r2, fillvalue=0):
print(i, j)

Be care that Repeater is using new-style class. (it's default on py3k) I
couldn't see any problem with officially released windows binary, but I
could see following error with VC6 debug build.

(1, 2)
(1, 2)
(1, 2)
(0, 2)
XXX undetected error
Traceback (most recent call last):
File "a.py", line 20, in <module>
print(i, j)
File "a.py", line 15, in next
raise StopIteration
StopIteration

# Still there is possibility this is VC6 bug though.

----------
nosy: +ocean-city

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7244>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Oct 31, 2009, 3:12 AM

Post #3 of 15 (514 views)
Permalink
[issue7244] itertools.zip_longest behaves strangely with an iterable class [In reply to]

Hirokazu Yamamoto <ocean-city [at] m2> added the comment:

I hope an attached patch will fix this issue. (this patch is for trunk)
I think PyErr_Clear() is needed to clear StopIteration there.

----------
keywords: +patch
versions: +Python 2.6, Python 2.7, Python 3.2
Added file: http://bugs.python.org/file15239/fix_izip_longest.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7244>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Oct 31, 2009, 3:31 AM

Post #4 of 15 (502 views)
Permalink
[issue7244] itertools.zip_longest behaves strangely with an iterable class [In reply to]

Georg Brandl <georg [at] python> added the comment:

The patch is incorrect; tp_iternext can raise exceptions other than
StopIteration which must be let through.

----------
assignee: -> rhettinger
nosy: +georg.brandl, rhettinger

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7244>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Oct 31, 2009, 4:35 AM

Post #5 of 15 (496 views)
Permalink
[issue7244] itertools.zip_longest behaves strangely with an iterable class [In reply to]

Daniel Urban <urban.dani+py [at] gmail> added the comment:

> I saw strange thing with following code + release26-maint/trunk.

I tried your code (with the new-style class) with Python 2.6.4 and 2.7a0
(trunk), and both worked fine. I built them with GCC 4.2.4 on Ubuntu 8.04.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7244>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Oct 31, 2009, 9:51 AM

Post #6 of 15 (493 views)
Permalink
[issue7244] itertools.zip_longest behaves strangely with an iterable class [In reply to]

Georg Brandl <georg [at] python> added the comment:

> I tried your code (with the new-style class) with Python 2.6.4 and 2.7a0
> (trunk), and both worked fine. I built them with GCC 4.2.4 on Ubuntu 8.04.

The problem seems to only show up in debug builds on 2.x, but it is there.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7244>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Oct 31, 2009, 10:13 AM

Post #7 of 15 (486 views)
Permalink
[issue7244] itertools.zip_longest behaves strangely with an iterable class [In reply to]

Raymond Hettinger <rhettinger [at] users> added the comment:

I've got it from here.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7244>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Nov 1, 2009, 1:32 AM

Post #8 of 15 (488 views)
Permalink
[issue7244] itertools.zip_longest behaves strangely with an iterable class [In reply to]

Hirokazu Yamamoto <ocean-city [at] m2> added the comment:

I created the patch to improve test which was checked in r76004. This
patch checks if correct elements are returned even when RuntimeError is
raised. Could you take a look?

----------
Added file: http://bugs.python.org/file15245/improve_test_itertools.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7244>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Nov 1, 2009, 1:37 AM

Post #9 of 15 (471 views)
Permalink
[issue7244] itertools.zip_longest behaves strangely with an iterable class [In reply to]

Hirokazu Yamamoto <ocean-city [at] m2> added the comment:

> correct elements are returned even when RuntimeError is
raised.

Or maybe it is not guaranteed. :-)

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7244>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Nov 1, 2009, 9:41 AM

Post #10 of 15 (471 views)
Permalink
[issue7244] itertools.zip_longest behaves strangely with an iterable class [In reply to]

Raymond Hettinger <rhettinger [at] users> added the comment:

Already checked-in a fix for Py2.6 in r76004.
Will forward port shortly.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7244>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Nov 1, 2009, 9:52 AM

Post #11 of 15 (469 views)
Permalink
[issue7244] itertools.zip_longest behaves strangely with an iterable class [In reply to]

Raymond Hettinger <rhettinger [at] users> added the comment:

Will take a look at your patch.
Am also still doing work to complete r76004
for the case where the refcnt > 1.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7244>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Nov 1, 2009, 10:46 AM

Post #12 of 15 (471 views)
Permalink
[issue7244] itertools.zip_longest behaves strangely with an iterable class [In reply to]

Changes by Raymond Hettinger <rhettinger [at] users>:


----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7244>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Nov 1, 2009, 10:46 AM

Post #13 of 15 (467 views)
Permalink
[issue7244] itertools.zip_longest behaves strangely with an iterable class [In reply to]

Changes by Raymond Hettinger <rhettinger [at] users>:


----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7244>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Nov 1, 2009, 10:48 AM

Post #14 of 15 (471 views)
Permalink
[issue7244] itertools.zip_longest behaves strangely with an iterable class [In reply to]

Raymond Hettinger <rhettinger [at] users> added the comment:

Added r76018 to use Hirokazu's test for the RuntimeError case and to
redirect stdout to a file for the StopIteration case. Also, fixed-up
weird indentation in the C code.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7244>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Nov 1, 2009, 1:04 PM

Post #15 of 15 (461 views)
Permalink
[issue7244] itertools.zip_longest behaves strangely with an iterable class [In reply to]

Raymond Hettinger <rhettinger [at] users> added the comment:

Forward-ported in r76025 r76026 and r76027.

----------
resolution: -> fixed
status: open -> closed

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7244>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Python bugs RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.