Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Bugs

[issue15592] subprocess.communicate() breaks on no input with universal newlines true

 

 

Python bugs RSS feed   Index | Next | Previous | View Threaded


report at bugs

Aug 8, 2012, 9:38 AM

Post #1 of 12 (417 views)
Permalink
[issue15592] subprocess.communicate() breaks on no input with universal newlines true

New submission from Chris Jerdonek:

Popen.communicate() raises an exception if passed no input when stdin=PIPE and universal_newlines=True. With universal_newlines=False, no exception is raised. For example, the following yields--

args = [sys.executable, '-c', 'pass']

popen = Popen(args, universal_newlines=False, stdin=PIPE, stdout=PIPE, stderr=PIPE)
print(popen.communicate())

popen = Popen(args, universal_newlines=True, stdin=PIPE, stdout=PIPE, stderr=PIPE)
print(popen.communicate())

(b'', b'[41449 refs]\n')
Traceback (most recent call last):
...
File ".../Lib/subprocess.py", line 1581, in _communicate_with_poll
self._input = self._input.encode(self.stdin.encoding)
AttributeError: 'NoneType' object has no attribute 'encode'

It seems like communicate() should not be trying to encode input if there is none.

I can provide a patch with tests if it is agreed this is an issue (along with a fix assuming it is reasonable).

----------
components: Library (Lib)
messages: 167694
nosy: cjerdonek
priority: normal
severity: normal
stage: test needed
status: open
title: subprocess.communicate() breaks on no input with universal newlines true
type: behavior
versions: Python 3.3

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15592>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 8, 2012, 9:40 AM

Post #2 of 12 (409 views)
Permalink
[issue15592] subprocess.communicate() breaks on no input with universal newlines true [In reply to]

Chris Jerdonek added the comment:

See issue 12623 for another issue related to communicate() and universal newline support.

----------
nosy: +pitrou

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15592>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 13, 2012, 5:50 AM

Post #3 of 12 (400 views)
Permalink
[issue15592] subprocess.communicate() breaks on no input with universal newlines true [In reply to]

Andrew Svetlov added the comment:

_communicate_with_select has the same problem as _communicate_with_poll.

I don't understand why input has encoded if universal_newlines and passed unchanged otherwise.

>From my perspective input should be encoded (converted to bytes) if it is str regardless of universal_newlines value.

----------
nosy: +asvetlov

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15592>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 13, 2012, 7:31 AM

Post #4 of 12 (402 views)
Permalink
[issue15592] subprocess.communicate() breaks on no input with universal newlines true [In reply to]

Chris Jerdonek added the comment:

> From my perspective input should be encoded (converted to bytes) if it is str regardless of universal_newlines value.

I don't know the reason, but the limitation is documented:

"The optional input argument should be data to be sent to the child process, or None, if no data should be sent to the child. The type of input must be bytes or, if universal_newlines was True, a string."

http://docs.python.org/dev/library/subprocess.html#subprocess.Popen.communicate

Maybe "explicit better than implicit"?

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15592>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 13, 2012, 10:03 AM

Post #5 of 12 (399 views)
Permalink
[issue15592] subprocess.communicate() breaks on no input with universal newlines true [In reply to]

Chris Jerdonek added the comment:

This issue may be the bug referenced here:

# BUG: can't give a non-empty stdin because it breaks both the
# select- and poll-based communicate() implementations.
(stdout, stderr) = p.communicate()

http://hg.python.org/cpython/file/843e0da7e91f/Lib/test/test_subprocess.py#l618

The reason is that the above test would fail if stdin were not None since the input to communicate is None.

I am preparing a test and patch for this issue.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15592>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 13, 2012, 12:55 PM

Post #6 of 12 (400 views)
Permalink
[issue15592] subprocess.communicate() breaks on no input with universal newlines true [In reply to]

Andrew Svetlov added the comment:

I like to accept both str and bytes if universal_newlines is False and raise explicit exception for bytes if universal_newlines is True.

Looks like universal_newlines for bytes doesn't make sense, also we prefer to use str everywhere. Bytes should be used if it is really byte-stream or if unicode cannot be passed by some limitations of underlying OS etc.

There are couple dark corners:
1. What to do if stdin doesn't have `encoding` attribute? Convert to bytes using filesystemencoding? Raise explicit exception? Adding `encode` parameter for .communicate doesn't looks elegant.
2. How .communicate works on Windows? I see completely different implementation of this method for win32.

I need some time to investigate before publish proposal to python-dev list.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15592>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 13, 2012, 1:09 PM

Post #7 of 12 (399 views)
Permalink
[issue15592] subprocess.communicate() breaks on no input with universal newlines true [In reply to]

Chris Jerdonek added the comment:

Attaching a failing test case.

Also, to confirm, this issue does not seem to affect 3.2.

----------
keywords: +patch
Added file: http://bugs.python.org/file26787/failing-test-case-1.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15592>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 13, 2012, 1:09 PM

Post #8 of 12 (398 views)
Permalink
[issue15592] subprocess.communicate() breaks on no input with universal newlines true [In reply to]

Changes by Chris Jerdonek <chris.jerdonek [at] gmail>:


----------
stage: test needed -> needs patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15592>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 13, 2012, 2:22 PM

Post #9 of 12 (398 views)
Permalink
[issue15592] subprocess.communicate() breaks on no input with universal newlines true [In reply to]

Chris Jerdonek added the comment:

Attaching patch.

----------
stage: needs patch -> patch review
Added file: http://bugs.python.org/file26788/issue-15592-1.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15592>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 14, 2012, 8:43 AM

Post #10 of 12 (388 views)
Permalink
[issue15592] subprocess.communicate() breaks on no input with universal newlines true [In reply to]

Roundup Robot added the comment:

New changeset 839bd8f98539 by Andrew Svetlov in branch '3.2':
Add test to explicit check the absence regression in subprocess (issue #15592).
http://hg.python.org/cpython/rev/839bd8f98539

New changeset 9d86480cc177 by Andrew Svetlov in branch 'default':
Issue #15592. Fix regression: subprocess.communicate() breaks on no input with universal newlines true.
http://hg.python.org/cpython/rev/9d86480cc177

----------
nosy: +python-dev

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15592>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 14, 2012, 8:49 AM

Post #11 of 12 (391 views)
Permalink
[issue15592] subprocess.communicate() breaks on no input with universal newlines true [In reply to]

Andrew Svetlov added the comment:

Close the issue. Thanks, Chris.
It was the real regression, 3.2 works fine.

See also #15649 for accepting str if not universal_newlines.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15592>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 14, 2012, 9:29 AM

Post #12 of 12 (391 views)
Permalink
[issue15592] subprocess.communicate() breaks on no input with universal newlines true [In reply to]

Chris Jerdonek added the comment:

Thanks, Andrew!

----------
resolution: -> fixed
stage: patch review -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15592>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Python bugs RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.