Mailing List Archive: Python: Bugs

[issue7268] 2to3 execfile conversion changes behavior

Index | Next | Previous | View Threaded

report at bugs

Nov 5, 2009, 4:08 PM

Post #1 of 18 (578 views)
Permalink

[issue7268] 2to3 execfile conversion changes behavior

New submission from flashk <flashk [at] gmail>:

I recently ran 2to3 on some of my scripts and noticed a change in behavior.

I had a script that used the built-in execfile function. After the conversion, it was
changed to manually open the file and read the contents into the exec function. Now I am
getting a SyntaxError with the modified code.

The SyntaxError is caused when the new exec code is called on a script that ends with an
empty indented line. I'm not sure if this is an error with the 2to3 script or the built-
in exec function.

I've attached a very minimal test case. The test.py file is a simple hello world script
that ends with an empty indented line. The execfile_example.py file simply calls
execfile on the test.py script. The execfile_example_converted.py file is the 2to3
converted version of execfile_example.py.

I'm running Python 2.6.2 on Windows XP.

----------
components: 2to3 (2.x to 3.0 conversion tool), Interpreter Core
files: execfile_problem.zip
messages: 94949
nosy: flashk
severity: normal
status: open
title: 2to3 execfile conversion changes behavior
type: behavior
versions: Python 2.6
Added file: http://bugs.python.org/file15267/execfile_problem.zip

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7268>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 5, 2009, 4:18 PM

Post #2 of 18 (562 views)
Permalink

[issue7268] 2to3 execfile conversion changes behavior [In reply to]

Benjamin Peterson <benjamin [at] python> added the comment:

Could you attach the files separately or paste them into the bug? zip
files are hard to work with.

----------
nosy: +benjamin.peterson

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7268>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 5, 2009, 4:22 PM

Post #3 of 18 (555 views)
Permalink

[issue7268] 2to3 execfile conversion changes behavior [In reply to]

Changes by flashk <flashk [at] gmail>:

Added file: http://bugs.python.org/file15268/test.py

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7268>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 5, 2009, 4:22 PM

Post #4 of 18 (556 views)
Permalink

[issue7268] 2to3 execfile conversion changes behavior [In reply to]

Changes by flashk <flashk [at] gmail>:

Added file: http://bugs.python.org/file15269/execfile_example.py

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7268>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 5, 2009, 4:22 PM

Post #5 of 18 (562 views)
Permalink

[issue7268] 2to3 execfile conversion changes behavior [In reply to]

Changes by flashk <flashk [at] gmail>:

Added file: http://bugs.python.org/file15270/execfile_example_converted.py

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7268>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 5, 2009, 4:24 PM

Post #6 of 18 (560 views)
Permalink

[issue7268] 2to3 execfile conversion changes behavior [In reply to]

flashk <flashk [at] gmail> added the comment:

I just attached the files individually.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7268>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 5, 2009, 4:27 PM

Post #7 of 18 (554 views)
Permalink

[issue7268] 2to3 execfile conversion changes behavior [In reply to]

Benjamin Peterson <benjamin [at] python> added the comment:

test.py is invalid Python 3 syntax.

----------
resolution: -> invalid
status: open -> closed

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7268>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 5, 2009, 4:44 PM

Post #8 of 18 (563 views)
Permalink

[issue7268] 2to3 execfile conversion changes behavior [In reply to]

flashk <flashk [at] gmail> added the comment:

I'm running this code under 2.6, so the print statement should not be the
issue. I've attached a new version of test.py that simply performs a
variable assignment and I still get the syntax error on both 2.6 and 3.1
with the exec function. Also, the syntax error is reported on line 3,
which is the empty indented line. Sorry for the confusion.

----------
Added file: http://bugs.python.org/file15271/test.py

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7268>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 5, 2009, 5:00 PM

Post #9 of 18 (559 views)
Permalink

[issue7268] 2to3 execfile conversion changes behavior [In reply to]

Benjamin Peterson <benjamin [at] python> added the comment:

This is because you have DOS newlines which the Python compiler cannot
handle. In 2.x, open("test.py", "r") does not translate newlines. In
3.x, it does.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7268>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 5, 2009, 5:15 PM

Post #10 of 18 (562 views)
Permalink

[issue7268] 2to3 execfile conversion changes behavior [In reply to]

flashk <flashk [at] gmail> added the comment:

Ok, I converted test.py to use Unix style newlines and still get the
syntax error on both 2.6 and 3.1. I'm confused as to why execfile works on
the file but reading the contents and passing it to exec behaves
differently under 2.6. Sorry if I'm just being dense.

----------
Added file: http://bugs.python.org/file15272/test.py

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7268>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 5, 2009, 5:25 PM

Post #11 of 18 (549 views)
Permalink

[issue7268] 2to3 execfile conversion changes behavior [In reply to]

Benjamin Peterson <benjamin [at] python> added the comment:

2009/11/5 flashk <report [at] bugs>:
>
> flashk <flashk [at] gmail> added the comment:
>
> Ok, I converted test.py to use Unix style newlines and still get the
> syntax error on both 2.6 and 3.1. I'm confused as to why execfile works on
> the file but reading the contents and passing it to exec behaves
> differently under 2.6. Sorry if I'm just being dense.

open in 2.x does not translate newlines by default. It does so in 3.x.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7268>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 5, 2009, 5:38 PM

Post #12 of 18 (560 views)
Permalink

[issue7268] 2to3 execfile conversion changes behavior [In reply to]

flashk <flashk [at] gmail> added the comment:

Ok, but why am I still getting a syntax error in both 2.6 and 3.1 on the
file, even after converting the newlines?

If I remove the trailing indentation then everything works properly on 2.6
and 3.1, even with DOS newlines.

It just seems that exec does not properly handle code that ends with an
empty indented line, but execfile handles it correctly.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7268>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 5, 2009, 5:48 PM

Post #13 of 18 (561 views)
Permalink

[issue7268] 2to3 execfile conversion changes behavior [In reply to]

Benjamin Peterson <benjamin [at] python> added the comment:

2009/11/5 flashk <report [at] bugs>:
>
> flashk <flashk [at] gmail> added the comment:
>
> Ok, but why am I still getting a syntax error in both 2.6 and 3.1 on the
> file, even after converting the newlines?
>
> If I remove the trailing indentation then everything works properly on 2.6
> and 3.1, even with DOS newlines.
>
> It just seems that exec does not properly handle code that ends with an
> empty indented line, but execfile handles it correctly.

Well, it works for me with the empty newline. Can you isolate the exact problem?

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7268>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 5, 2009, 6:58 PM

Post #14 of 18 (548 views)
Permalink

[issue7268] 2to3 execfile conversion changes behavior [In reply to]

flashk <flashk [at] gmail> added the comment:

On Thu, Nov 5, 2009 at 5:48 PM, Benjamin Peterson <report [at] bugs>wrote:
>
>
> Well, it works for me with the empty newline. Can you isolate the exact
> problem?

For me, the exact problem seems to be that exec raises a SyntaxError if the
code contains a trailing indentation. Here's a summary of everything I've
tried:

* test.py runs successfully on 2.6 and 3.1, regardless of line ending style
(LF, CR+LF)

* "execfile('test.py')" runs successfully on 2.6, regardless of line ending
style

* "exec(compile(open('test.py').read(), 'test.py', 'exec'))" raises a
SyntaxError on line 3 of test.py under 2.6 and 3.1, regardless of line
ending style.

* Removing the trailing indentation from test.py causes the above code to
succeed under 2.6 and 3.1, regardless of line ending style.

You mentioned that it worked for you with an empty newline. To be clear, the
last line of test.py contains a single tab, with no newline character.

----------
Added file: http://bugs.python.org/file15273/unnamed

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7268>
_______________________________________

Attachments:

unnamed (1.44 KB)

report at bugs

Nov 5, 2009, 7:13 PM

Post #15 of 18 (550 views)
Permalink

[issue7268] 2to3 execfile conversion changes behavior [In reply to]

flashk <flashk [at] gmail> added the comment:

I noticed that calling "exec('\t')" raises a SyntaxError, so maybe this is
the root of the problem. I manually added a newline character to the end
of the file contents and it fixes the issue for me:

exec(compile(open('test.py').read()+'\n', 'test.py', 'exec'))

So maybe the 2to3 script should automatically do this in order to be
compatible with previous execfile behavior? Or is this a problem with
exec?

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7268>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 6, 2009, 10:14 PM

Post #16 of 18 (543 views)
Permalink

[issue7268] 2to3 execfile conversion changes behavior [In reply to]

Gabriel Genellina <gagsl-py2 [at] yahoo> added the comment:

The docs for the built-in function compile() say:

"Note: When compiling a string with multi-line statements, line endings
must be represented by a single newline character ('\n'), and the input
must be terminated by at least one newline character. If line endings
are represented by '\r\n', use str.replace() to change them into '\n'."
http://www.python.org/doc/3.1/library/functions.html#compile

And the standard module py_compile ensures the source text ends with
'\n' before calling the built-in compile function:

if codestring and codestring[-1] != '\n':
codestring = codestring + '\n'

I'd say the corresponding 2to3 fixer should do the same.

----------
nosy: +gagenellina

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7268>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 7, 2009, 2:21 AM

Post #17 of 18 (535 views)
Permalink

[issue7268] 2to3 execfile conversion changes behavior [In reply to]

Gabriel Genellina <gagsl-py2 [at] yahoo> added the comment:

This is a patch for the execfile fixer, so it converts
execfile("fn")
into this:
exec(compile(open("fn").read()+'\n', "fn", 'exec'))

(Yes, it looks ugly. A better way would be to fix the compile() builtin
so it does not require the last line of source to end with '\n')

This is my very first incursion into 2to3 so it may be terribly wrong...

----------
keywords: +patch
Added file: http://bugs.python.org/file15281/lib2to3.diff

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7268>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 16, 2009, 3:08 PM

Post #18 of 18 (478 views)
Permalink

[issue7268] 2to3 execfile conversion changes behavior [In reply to]

Benjamin Peterson <benjamin [at] python> added the comment:

I've fixed underlying compile() newline problem in the trunk and py3k.
However, I think that change is big enough that I don't want to to
backport it. I'm leaving this open to think about what to do with 2to3
in this situation.

----------
priority: -> normal
resolution: invalid ->
status: closed -> open

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue7268>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads