Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Dev

problem with recursive "yield from" delegation

 

 

Python dev RSS feed   Index | Next | Previous | View Threaded


stefan_ml at behnel

Mar 7, 2012, 12:40 PM

Post #1 of 16 (926 views)
Permalink
problem with recursive "yield from" delegation

Hi,

I found a problem in the current "yield from" implementation that I think
is worth discussing:

http://bugs.python.org/issue14220

Test code:

def g1():
yield "y1"
yield from g2()
yield "y4"

def g2():
yield "y2"
try:
yield from gi
except ValueError:
pass # catch "already running" error
yield "y3"

gi = g1()
for y in gi:
print("Yielded: %s" % (y,))

This is what it currently does:

1) g1() delegates to a new g2(), propagates its "y2" value and asks for the
next value

2) g2 delegates back to the g1 instance and asks for its next value

3) Python sees the active delegation in g1 and asks g2 for its next value

4) g2 sees that it's already running and throws an exception

Ok so far. Now:

5) the exception is propagated into g1 at call level 3) instead of the
original requestor g2 one level above

6) g1 undelegates and terminates by the exception

7) g2 catches the exception, yields "y3" and then terminates normally

8) g1 gets control back but has already terminated and does nothing

Effect: g1 does not yield "y4" anymore.

The problem is in steps 5) and 6), which are handled by g1 at the wrong
call level. They shouldn't lead to undelegation and termination in g1, just
to an exception being raised in g2.

I ran into this while trying to adapt the implementation for Cython, which
has a different generator type implementation but otherwise uses more or
less the same code now. But I'm not sure how to fix this one without major
changes to the implementation, especially not without special casing the
generator type on delegation (which won't work because CPython doesn't know
about Cython generators). Any ideas?

Stefan

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


benjamin at python

Mar 7, 2012, 3:01 PM

Post #2 of 16 (916 views)
Permalink
Re: problem with recursive "yield from" delegation [In reply to]

2012/3/7 Stefan Behnel <stefan_ml [at] behnel>:
> The problem is in steps 5) and 6), which are handled by g1 at the wrong
> call level. They shouldn't lead to undelegation and termination in g1, just
> to an exception being raised in g2.

That looks wrong indeed.

>
> I ran into this while trying to adapt the implementation for Cython, which
> has a different generator type implementation but otherwise uses more or
> less the same code now. But I'm not sure how to fix this one without major
> changes to the implementation

Cython's or CPython's implementation?


--
Regards,
Benjamin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


ncoghlan at gmail

Mar 7, 2012, 3:57 PM

Post #3 of 16 (911 views)
Permalink
Re: problem with recursive "yield from" delegation [In reply to]

On Thu, Mar 8, 2012 at 6:40 AM, Stefan Behnel <stefan_ml [at] behnel> wrote:
> I ran into this while trying to adapt the implementation for Cython, which
> has a different generator type implementation but otherwise uses more or
> less the same code now. But I'm not sure how to fix this one without major
> changes to the implementation, especially not without special casing the
> generator type on delegation (which won't work because CPython doesn't know
> about Cython generators). Any ideas?

After tinkering with it a bit, a couple of my original guesses as to
the underlying problem were clearly wrong. I'm moving house next week,
so it'll be a while before I get to look at it in detail, but I added
Mark Shannon to the issue's nosy list. He's been working on a few
patches lately to clean up generator related state handling in
general, so he may have some insight into this.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan [at] gmail   |   Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


benjamin at python

Mar 7, 2012, 4:00 PM

Post #4 of 16 (907 views)
Permalink
Re: problem with recursive "yield from" delegation [In reply to]

2012/3/7 Benjamin Peterson <benjamin [at] python>:
> 2012/3/7 Stefan Behnel <stefan_ml [at] behnel>:
>> The problem is in steps 5) and 6), which are handled by g1 at the wrong
>> call level. They shouldn't lead to undelegation and termination in g1, just
>> to an exception being raised in g2.
>
> That looks wrong indeed.

Fixed as of 3357eac1ba62


--
Regards,
Benjamin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


ncoghlan at gmail

Mar 7, 2012, 4:11 PM

Post #5 of 16 (918 views)
Permalink
Re: problem with recursive "yield from" delegation [In reply to]

On Thu, Mar 8, 2012 at 10:00 AM, Benjamin Peterson <benjamin [at] python> wrote:
> 2012/3/7 Benjamin Peterson <benjamin [at] python>:
>> 2012/3/7 Stefan Behnel <stefan_ml [at] behnel>:
>>> The problem is in steps 5) and 6), which are handled by g1 at the wrong
>>> call level. They shouldn't lead to undelegation and termination in g1, just
>>> to an exception being raised in g2.
>>
>> That looks wrong indeed.
>
> Fixed as of 3357eac1ba62

Thanks. And, since the fix was entirely internal to the generator
implementation, Stefan should be right for the Cython generators, too.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan [at] gmail   |   Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


jimjjewett at gmail

Mar 7, 2012, 4:32 PM

Post #6 of 16 (917 views)
Permalink
problem with recursive "yield from" delegation [In reply to]

http://mail.python.org/pipermail/python-dev/2012-March/117396.html
Stefan Behnel posted:

> I found a problem in the current "yield from" implementation ...

[paraphrasing]

g1 yields from g2
g2 yields from g1
XXX python follows the existing delegation without checking re-entrancy
g2 (2nd call) checks re-entrancy, and raises an exception
g1 (2nd call) gets to handle the exception, and doesn't
g2 (1st call) gets to handle the exception, and does


How is this a problem?

Re-entering a generator is a bug. Python caught it and raised an
appropriate exception.

It would be nice if python caught the generator cycle as soon as it was
created, just as it would be nice if reference cycles were collected as
soon as they became garbage. But python doesn't promise to catch cycles
immediately, and the checks required to do so would slow down all code,
so in practice the checks are delayed.


-jJ

--

If there are still threading problems with my replies, please
email me with details, so that I can try to resolve them. -jJ

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


ncoghlan at gmail

Mar 7, 2012, 4:47 PM

Post #7 of 16 (912 views)
Permalink
Re: problem with recursive "yield from" delegation [In reply to]

On Thu, Mar 8, 2012 at 10:32 AM, Jim J. Jewett <jimjjewett [at] gmail> wrote:
> How is this a problem?
>
> Re-entering a generator is a bug.  Python caught it and raised an
> appropriate exception.

No, the problem was that the interpreter screwed up the state of the
generators while attempting to deal with the erroneous reentry. The
ValueError *should* just be caught and completely suppressed by the
try/except block, but that wasn't quite happening properly - the
failed attempt at reentry left the generators in a dodgy state (which
is why the subsequent "3" was being produced, but then the expected
final "4" vanished into the electronic ether).

Benjamin figured out where the generator's reentrancy check was going
wrong, so Stefan's example should do the right thing in the next alpha
(i.e. the ValueError will still get raised and suppressed by the
try/except block, the inner generator will complete, but the outer
generator will also continue on to produce the final value).

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan [at] gmail   |   Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


mark at hotpy

Mar 8, 2012, 3:52 AM

Post #8 of 16 (897 views)
Permalink
Re: problem with recursive "yield from" delegation [In reply to]

Stefan Behnel wrote:
> Hi,
>
> I found a problem in the current "yield from" implementation that I think
> is worth discussing:
>
> http://bugs.python.org/issue14220
>
[snip]

I've been experimenting with the implementation of PEP 380, and I found
a couple of interesting things.

First of all, the semantics described in the PEP do not match the tests.
If you substitute the supposedly semantically equivalent code
based on normal yields for each yield from in the test code
(Lib/test/test_pep380.py) and run it, then it fails.

My second experiment involved stripping away all the code relating to
yield from outside of the interpreter and changing the YIELD_FROM
bytecode to repeat itself, by setting the last instruction to the
instruction immediately before itself.
To do this I added 4 lines of code and removed over 120 lines :)

This fails many of the tests*, but works for the most straightforward
use cases. Many of these failures seem to me to be more 'natural' than
the current behaviour

It might be possible to fix most, or all?, of the other failures
by compiling "yield from" into a two opcode sequence:
YIELD_FROM_START and YIELD_FROM_REPEAT.
Both opcodes should be reasonably simple and __next__() and send() would
not have to worry about the subiterator, although close() and throw()
might (the subiterator would be top-of-stack in the generator's frame)

Overall the semantics of PEP 380 seem far too complex,
trying to do several things at once.
An example:
Plain yield makes no distinction between a receiving a None and any
other value, so send(None) and __next__() are the same. Yield from
makes this distinction so has to test for None, meaning the semantics of
send now changes with its argument.

I would recommend changing one of two things in the PEP:
Either, close and throw should not close/throw in subiterators
(this would simplify the semantics and implementation immensely)
Or, only allow subgenerators, not subiterators
(this would fix the next/send problem).

I would also suggest a change in implementation to perform all yields
within the interpreter. A simpler implementation is likely to be a
more reliable one.

Finally, the PEP itself makes no mention of coroutines, stackless or
greenlet in the alternatives or criticisms section. Perhaps it should.

Cheers,
Mark.

*Tests in PEP 380. It passes all other tests.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


ncoghlan at gmail

Mar 8, 2012, 4:04 AM

Post #9 of 16 (899 views)
Permalink
Re: problem with recursive "yield from" delegation [In reply to]

On Thu, Mar 8, 2012 at 9:52 PM, Mark Shannon <mark [at] hotpy> wrote:
> I would recommend changing one of two things in the PEP:
> Either, close and throw should not close/throw in subiterators
> (this would simplify the semantics and implementation immensely)
> Or, only allow subgenerators, not subiterators
> (this would fix the next/send problem).

Either of those changes would completely defeat the point of the PEP.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan [at] gmail   |   Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


ncoghlan at gmail

Mar 8, 2012, 4:07 AM

Post #10 of 16 (902 views)
Permalink
Re: problem with recursive "yield from" delegation [In reply to]

On Thu, Mar 8, 2012 at 9:52 PM, Mark Shannon <mark [at] hotpy> wrote:
> First of all, the semantics described in the PEP do not match the tests.
> If you substitute the supposedly semantically equivalent code
> based on normal yields for each yield from in the test code
> (Lib/test/test_pep380.py) and run it, then it fails.

What's more important is whether or not it matches the semantics of
inlining the subgenerator bodies. The expansion in the PEP was an
attempt to define a way to achieve that in current Python without
interpreter support.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan [at] gmail   |   Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


mark at hotpy

Mar 8, 2012, 5:45 AM

Post #11 of 16 (898 views)
Permalink
Re: problem with recursive "yield from" delegation [In reply to]

Nick Coghlan wrote:
> On Thu, Mar 8, 2012 at 9:52 PM, Mark Shannon <mark [at] hotpy> wrote:
>> First of all, the semantics described in the PEP do not match the tests.
>> If you substitute the supposedly semantically equivalent code
>> based on normal yields for each yield from in the test code
>> (Lib/test/test_pep380.py) and run it, then it fails.
>
> What's more important is whether or not it matches the semantics of
> inlining the subgenerator bodies. The expansion in the PEP was an
> attempt to define a way to achieve that in current Python without
> interpreter support.

So "yield from X" means "inline X here", if X is a generator.
That is much, much easier to understand than the big block of
code in the PEP.
It really ought say that "yield from" is equivalent to inlining
in the PEP.

Cheers,
Mark

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


ncoghlan at gmail

Mar 8, 2012, 6:11 AM

Post #12 of 16 (895 views)
Permalink
Re: problem with recursive "yield from" delegation [In reply to]

On Thu, Mar 8, 2012 at 11:45 PM, Mark Shannon <mark [at] hotpy> wrote:
> It really ought say that "yield from" is equivalent to inlining
> in the PEP.

That's what the motivation section is about. There's also an entire
subsection called "The Refactoring Principle" that describes this
intent. However, we needed something more concrete to flesh out the
original implementation details, which is what the code equivalent was
designed to provide (it was also designed to point out that the days
of "you can just use a simple for loop" actually went away when PEP
342 was implemented).

Now, it may be that we fixed things during implementation that should
be reflected back into the formal semantic spec in the PEP - so if you
can point out specific cases where what we implemented doesn't match
the nominal behaviour, I'm open to updating the PEP accordingly. (Of
course, if there are any tests that fail solely due to the two known
differences in semantics that are already noted in the PEP, they don't
count).

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan [at] gmail   |   Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


stefan_ml at behnel

Mar 8, 2012, 11:00 AM

Post #13 of 16 (899 views)
Permalink
Re: problem with recursive "yield from" delegation [In reply to]

Stefan Behnel, 07.03.2012 21:40:
> I found a problem in the current "yield from" implementation

... and here's another one, also in genobject.c:

"""
int
PyGen_FetchStopIterationValue(PyObject **pvalue) {
PyObject *et, *ev, *tb;
PyObject *value = NULL;

if (PyErr_ExceptionMatches(PyExc_StopIteration)) {
PyErr_Fetch(&et, &ev, &tb);
Py_XDECREF(et);
Py_XDECREF(tb);
if (ev) {
value = ((PyStopIterationObject *)ev)->value;
Py_INCREF(value);
Py_DECREF(ev);
}
} else if (PyErr_Occurred()) {
return -1;
}
if (value == NULL) {
value = Py_None;
Py_INCREF(value);
}
*pvalue = value;
return 0;
}
"""

When the StopIteration was set using PyErr_SetObject(), "ev" points to the
value, not the exception instance, so this code lacks exception
normalisation. I use slightly different code in Cython (which needs to be
compatible with Py2.x), but CPython 3.3 could do something along these lines:

"""
if (ev) {
if (PyObject_IsInstance(ev, PyExc_StopIteration)) {
value = ((PyStopIterationObject *)ev)->value;
Py_INCREF(value); // or maybe XINCREF()?
Py_DECREF(ev);
} else {
/* PyErr_SetObject() puts the value directly into ev */
value = ev;
}
} else ...
"""

Would that be acceptable for CPython as well or would you prefer full
fledged normalisation?

Stefan

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


benjamin at python

Mar 8, 2012, 12:36 PM

Post #14 of 16 (898 views)
Permalink
Re: problem with recursive "yield from" delegation [In reply to]

2012/3/8 Stefan Behnel <stefan_ml [at] behnel>:
> Would that be acceptable for CPython as well or would you prefer full
> fledged normalisation?

I think we have to normalize for correctness. Consider that it may be
some StopIteration subclass which set "value" on construction.


--
Regards,
Benjamin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


solipsis at pitrou

Mar 8, 2012, 12:36 PM

Post #15 of 16 (898 views)
Permalink
Re: problem with recursive "yield from" delegation [In reply to]

On Thu, 8 Mar 2012 14:36:06 -0600
Benjamin Peterson <benjamin [at] python> wrote:
> 2012/3/8 Stefan Behnel <stefan_ml [at] behnel>:
> > Would that be acceptable for CPython as well or would you prefer full
> > fledged normalisation?
>
> I think we have to normalize for correctness. Consider that it may be
> some StopIteration subclass which set "value" on construction.

Perhaps it would be time to drop the whole delayed normalization thing,
provided the benchmarks don't exhibit a slowdown? It complicates a lot
of code paths.

cheers

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


benjamin at python

Mar 8, 2012, 12:45 PM

Post #16 of 16 (893 views)
Permalink
Re: problem with recursive "yield from" delegation [In reply to]

2012/3/8 Antoine Pitrou <solipsis [at] pitrou>:
> On Thu, 8 Mar 2012 14:36:06 -0600
> Benjamin Peterson <benjamin [at] python> wrote:
>> 2012/3/8 Stefan Behnel <stefan_ml [at] behnel>:
>> > Would that be acceptable for CPython as well or would you prefer full
>> > fledged normalisation?
>>
>> I think we have to normalize for correctness. Consider that it may be
>> some StopIteration subclass which set "value" on construction.
>
> Perhaps it would be time to drop the whole delayed normalization thing,
> provided the benchmarks don't exhibit a slowdown? It complicates a lot
> of code paths.

+1 Also, it delays errors from exception initialization to arbitrary points.

--
Regards,
Benjamin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Python dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.