Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Dev

Issue 13524: subprocess on Windows

 

 

Python dev RSS feed   Index | Next | Previous | View Threaded


mail at timgolden

Dec 4, 2011, 2:59 AM

Post #1 of 17 (1478 views)
Permalink
Issue 13524: subprocess on Windows

http://bugs.python.org/issue13524

Someone raised issue13524 yesterday to illustrate that a
subprocess will crash immediately if an environment block is
passed which does not contain a valid SystemRoot environment
variable.

Note that the calling (Python) process is unaffected; this
isn't - strictly - a Python crash. The issue is essentially
a Windows one where a fairly unusual cornercase -- passing
an empty environment -- has unforseen effects.

The smallest reproducible example is this:

import os, sys
import subprocess
subprocess.Popen(
[sys.executable],
env={}
)

and it can be prevented like this:

import os, sys
import subprocess
subprocess.Popen(
[sys.executable],
env={"SystemRoot" : os.environ['SystemRoot']}
)

There's a blog post here which gives a worked example:


http://jpassing.com/2009/12/28/the-hidden-danger-of-forgetting-to-specify-systemroot-in-a-custom-environment-block/

but as the author points out, nowhere on MSDN is there a warning
that SystemRoot is mandatory. (And, in effect, it's not as it
would just be possible to write code which had no need of it).

So... what's our take on this? As I see it we could:

1) Do nothing: it's the caller's responsibility to understand the
complications of the chosen Operating System.

2) Add a doc warning (ironically, considering the recent to-and-fro
on doc warnings in this very module).

3) Add a check into the subprocess.Popen code which would raise some
exception if the environment block is empty (or doesn't contain
SystemRoot) on the grounds that this probably wasn't what the user
thought they were doing.

4) Automatically add an entry for SystemRoot to the env block if it's
not present already.


It's tempting to opt for (1) and if we were exposing an API called
CreateProcess which mimicked the underlying Windows API I would be
inclined to go that way. But we're abstracting a little bit away
from that and I think that that layer of abstraction carries its
own responsibilities.

Option (3) seems to give the best balance. It *is* a cornercase, but at
the same time it's easy to misunderstand that the env block you're
passing in *replaces* rather than *augments* that of the current
process.

Thoughts?

TJG
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


ncoghlan at gmail

Dec 4, 2011, 3:42 AM

Post #2 of 17 (1459 views)
Permalink
Re: Issue 13524: subprocess on Windows [In reply to]

On Sun, Dec 4, 2011 at 8:59 PM, Tim Golden <mail [at] timgolden> wrote:
> So... what's our take on this? As I see it we could:
>
> 1) Do nothing: it's the caller's responsibility to understand the
>   complications of the chosen Operating System.
>
> 2) Add a doc warning (ironically, considering the recent to-and-fro
>   on doc warnings in this very module).
>
> 3) Add a check into the subprocess.Popen code which would raise some
>   exception if the environment block is empty (or doesn't contain
>   SystemRoot) on the grounds that this probably wasn't what the user
>   thought they were doing.
>
> 4) Automatically add an entry for SystemRoot to the env block if it's
>   not present already.
>
>
> It's tempting to opt for (1) and if we were exposing an API called
> CreateProcess which mimicked the underlying Windows API I would be
> inclined to go that way. But we're abstracting a little bit away
> from that and I think that that layer of abstraction carries its
> own responsibilities.
>
> Option (3) seems to give the best balance. It *is* a cornercase, but at
> the same time it's easy to misunderstand that the env block you're
> passing in *replaces* rather than *augments* that of the current
> process.

There's actually two questions to be answered:
1. What should we do in 3.2 and 2.7?
2. Should we do anything more in 3.3?

Raising an exception is not really an appropriate response for any of
them - running without SystemRoot actually works fine in most cases,
so raising an exception could break currently working code. As the
blog post noted, it's only some specific modules that don't work if
SystemRoot is not set. Should we really be inserting workarounds in
subprocess for buggy platform code that doesn't fall back to a
sensible default if a particular environment variable isn't set?

So, I don't think this is really a subprocess problem at all. It's a
platform bug on Windows that means the 'random' module may fail if
SystemRoot is not set in the environment. So, I think the right
approach is to:

1. Unset 'SystemRoot' in a windows shell
2. Run the test suite and observe the scale of the breakage
3. Then either:
- figure out a workaround that allows us to set an appropriate default
value for SystemRoot if needed (depending on the scope of the problem,
either do this at interpreter startup, or only in affected modules)
- if no feasible workaround is found, detect the failures related to
this problem and report a more meaningful error message

Either way, add explicit tests to the test suite to ensure that
affected modules behave as expected when SystemRoot is not set.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan [at] gmail   |   Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


mail at timgolden

Dec 4, 2011, 4:20 AM

Post #3 of 17 (1457 views)
Permalink
Re: Issue 13524: subprocess on Windows [In reply to]

On 04/12/2011 11:42, Nick Coghlan wrote:
> There's actually two questions to be answered:
> 1. What should we do in 3.2 and 2.7?
> 2. Should we do anything more in 3.3?

Agreed.

> 1. Unset 'SystemRoot' in a windows shell
> 2. Run the test suite and observe the scale of the breakage

Sorry; something I should have highlighted in the earlier post.
Behaviour varies between Windows versions. On WinXP, if you
unset SystemRoot in a cmd shell, you won't be able to run the
test suite: Python won't even start up. On Win7 Python will
start but, eg, the random module will fail.

This is actually a separate issue: how much of Python will work
without a valid SystemRoot. The OP's issue was that if you use
subprocess to start an arbitrary process (you get the same problem
if you try "notepad.exe") and pass it an env block without a valid
SystemRoot then that process will likely fail to start up. And it
won't be obvious why.

The case where someone tries to run Python (in general) without
a valid SystemRoot is a tiny cornercase and you'd be quite right
to push that back and say "Don't do that". I don't believe we have
to test for it or add code to work around it.

While I put the idea forward, I agree that an exception is more likely
than not to break existing code. I just can't see any clear alternative,
apart from option 1: we do nothing.

TJG
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


p.f.moore at gmail

Dec 4, 2011, 4:41 AM

Post #4 of 17 (1450 views)
Permalink
Re: Issue 13524: subprocess on Windows [In reply to]

On 4 December 2011 12:20, Tim Golden <mail [at] timgolden> wrote:
> On 04/12/2011 11:42, Nick Coghlan wrote:
>>
>> There's actually two questions to be answered:
>> 1. What should we do in 3.2 and 2.7?
>> 2. Should we do anything more in 3.3?

See below...

> This is actually a separate issue: how much of Python will work
> without a valid SystemRoot. The OP's issue was that if you use
> subprocess to start an arbitrary process (you get the same problem
> if you try "notepad.exe") and pass it an env block without a valid
> SystemRoot then that process will likely fail to start up. And it
> won't be obvious why.

I'm not 100% clear on the problem here. From how I'm reading things,
the problem is that not supplying SystemRoot will cause (some or all)
invocations of subprocess.Popen to fail - it's not specific to
starting Python. In that case, it seems to me that it's an OS issue,
but one that we should work around.

My feeling is that option 4 is best - set SystemRoot to its current
value if it's not been set by the user. This leaves the user unable to
set an environment with SystemRoot missing, but if the OS fails to
handle that properly, then I'm OK with that limitation.

As regards the version question above, I'd take the view that as an OS
issue, it's OK to leave it unchanged in 2.7 and 3.2, but add the above
to 3.3.

Paul.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


mail at timgolden

Dec 4, 2011, 6:08 AM

Post #5 of 17 (1456 views)
Permalink
Re: Issue 13524: subprocess on Windows [In reply to]

On 04/12/2011 12:41, Paul Moore wrote:
> I'm not 100% clear on the problem here. From how I'm reading things,
> the problem is that not supplying SystemRoot will cause (some or all)
> invocations of subprocess.Popen to fail - it's not specific to
> starting Python.

That's basically the situation.

>
> My feeling is that option 4 is best - set SystemRoot to its current
> value if it's not been set by the user. This leaves the user unable to
> set an environment with SystemRoot missing, but if the OS fails to
> handle that properly, then I'm OK with that limitation.

FWIW if we went this route we could set it if it's missing but
that still allows the user to set it to blank. I'm just a little
bit wary of altering the environment which the user believes has
been set.

TJG
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


martin.packman at canonical

Dec 4, 2011, 8:48 AM

Post #6 of 17 (1438 views)
Permalink
Re: Issue 13524: subprocess on Windows [In reply to]

On 04/12/2011, Tim Golden <mail [at] timgolden> wrote:
>
> Someone raised issue13524 yesterday to illustrate that a
> subprocess will crash immediately if an environment block is
> passed which does not contain a valid SystemRoot environment
> variable.
...
> 2) Add a doc warning (ironically, considering the recent to-and-fro
> on doc warnings in this very module).

There appears to already be such a warning, added because of a similar
earlier bug:

<http://bugs.python.org/issue3440>

Really this is a problem with the subprocess api making a common case
harder to do than necessary. If you read the documentation, you'll get
it right, but that's not ideal:

<http://sourcefrog.net/weblog/software/aesthetics/interface-levels.html>

>From the bug, the problem with the reporter's code is he passes a dict
with the one value he cares about as `env` to subprocess.Popen without
realising that it will prevent the inheriting of the current
environment. Your suggested fix for him also has an issue, it changes
the environment of the parent process without resetting it. Instead
you need something like:

e = dict(os.environ)
e['PATH_TO_MY_APPS'] = "path/to/my/apps"

The bzrlib TestCase has a method using subprocess that provides an
`env_changes` argument. With that, it's much easier to override or
remove just one variable without accidentally clearing the current
environment.

Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


ncoghlan at gmail

Dec 4, 2011, 12:52 PM

Post #7 of 17 (1451 views)
Permalink
Re: Issue 13524: subprocess on Windows [In reply to]

That's why I'm suggesting we look specifically at the cases where *Python*
misbehaves in an empty environment on Windows. Those are legitimately our
issue.

The problem in *general* is a platform one, so I don't think it makes sense
for us to modify the environment that has explicitly been passed in (e.g.
how would you test running without SystemRoot if subprocess added it
automatically?).

An extra parameter in the already confusing Popen signature wouldn't be
clearer than explicitly copying os.environ and modifying it.

--
Nick Coghlan (via Gmail on Android, so likely to be more terse than usual)
On Dec 4, 2011 10:22 PM, "Tim Golden" <mail [at] timgolden> wrote:

> On 04/12/2011 11:42, Nick Coghlan wrote:
>
>> There's actually two questions to be answered:
>> 1. What should we do in 3.2 and 2.7?
>> 2. Should we do anything more in 3.3?
>>
>
> Agreed.
>
> 1. Unset 'SystemRoot' in a windows shell
>> 2. Run the test suite and observe the scale of the breakage
>>
>
> Sorry; something I should have highlighted in the earlier post.
> Behaviour varies between Windows versions. On WinXP, if you
> unset SystemRoot in a cmd shell, you won't be able to run the
> test suite: Python won't even start up. On Win7 Python will
> start but, eg, the random module will fail.
>
> This is actually a separate issue: how much of Python will work
> without a valid SystemRoot. The OP's issue was that if you use
> subprocess to start an arbitrary process (you get the same problem
> if you try "notepad.exe") and pass it an env block without a valid
> SystemRoot then that process will likely fail to start up. And it
> won't be obvious why.
>
> The case where someone tries to run Python (in general) without
> a valid SystemRoot is a tiny cornercase and you'd be quite right
> to push that back and say "Don't do that". I don't believe we have
> to test for it or add code to work around it.
>
> While I put the idea forward, I agree that an exception is more likely
> than not to break existing code. I just can't see any clear alternative,
> apart from option 1: we do nothing.
>
> TJG
> ______________________________**_________________
> Python-Dev mailing list
> Python-Dev [at] python
> http://mail.python.org/**mailman/listinfo/python-dev<http://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
> ncoghlan%40gmail.com<http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com>
>


tjreedy at udel

Dec 4, 2011, 1:08 PM

Post #8 of 17 (1438 views)
Permalink
Re: Issue 13524: subprocess on Windows [In reply to]

On 12/4/2011 5:59 AM, Tim Golden wrote:
> http://bugs.python.org/issue13524
>
> Someone raised issue13524 yesterday to illustrate that a
> subprocess will crash immediately if an environment block is
> passed which does not contain a valid SystemRoot environment
> variable.
>
> Note that the calling (Python) process is unaffected; this
> isn't - strictly - a Python crash. The issue is essentially
> a Windows one where a fairly unusual cornercase -- passing
> an empty environment -- has unforseen effects.
>
> The smallest reproducible example is this:
>
> import os, sys
> import subprocess
> subprocess.Popen(
> [sys.executable],
> env={}
> )
>
> and it can be prevented like this:
>
> import os, sys
> import subprocess
> subprocess.Popen(
> [sys.executable],
> env={"SystemRoot" : os.environ['SystemRoot']}
> )
>
> There's a blog post here which gives a worked example:
>
>
> http://jpassing.com/2009/12/28/the-hidden-danger-of-forgetting-to-specify-systemroot-in-a-custom-environment-block/
>
>
> but as the author points out, nowhere on MSDN is there a warning
> that SystemRoot is mandatory. (And, in effect, it's not as it
> would just be possible to write code which had no need of it).
>
> So... what's our take on this? As I see it we could:
>
> 1) Do nothing: it's the caller's responsibility to understand the
> complications of the chosen Operating System.
>
> 2) Add a doc warning (ironically, considering the recent to-and-fro
> on doc warnings in this very module).
>
> 3) Add a check into the subprocess.Popen code which would raise some
> exception if the environment block is empty (or doesn't contain
> SystemRoot) on the grounds that this probably wasn't what the user
> thought they were doing.
>
> 4) Automatically add an entry for SystemRoot to the env block if it's
> not present already.
>
>
> It's tempting to opt for (1) and if we were exposing an API called
> CreateProcess which mimicked the underlying Windows API I would be
> inclined to go that way. But we're abstracting a little bit away
> from that and I think that that layer of abstraction carries its
> own responsibilities.
>
> Option (3) seems to give the best balance. It *is* a cornercase, but at
> the same time it's easy to misunderstand that the env block you're
> passing in *replaces* rather than *augments* that of the current
> process.
>
> Thoughts?

My inclination would be #4 on Windows, certainly for 3.3, unless there
is a clear reason not to.

For 2.7/3.2, at least say (not warn, just say) in the doc that that a
subprocess on Windows may require that SystemRoot be set.

The blog post says the problem is worse on Win 7. So it is not going away.

The blog post has a comment from Martin Loewis a year ago linking to
http://mail.python.org/pipermail/python-dev/2010-November/105866.html
That thread refers to a bug that was not posted on the tracker. This
makes at least three (including #3440).

--
Terry Jan Reedy

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


ncoghlan at gmail

Dec 4, 2011, 4:16 PM

Post #9 of 17 (1428 views)
Permalink
Re: Issue 13524: subprocess on Windows [In reply to]

On Mon, Dec 5, 2011 at 7:08 AM, Terry Reedy <tjreedy [at] udel> wrote:
> My inclination would be #4 on Windows, certainly for 3.3, unless there is a
> clear reason not to.

Yes, there is: that environment is the *exact* environment that should
be passed to the child processes. It's not our place to go implicitly
adding things to it. If MS aren't willing to add SystemRoot
automatically in CreateProcess (despite releasing libraries that
require it to be set), there's no way we should be adding it for them.

Fixing our stuff (like importing the random module) to work to at
least some degree even if SystemRoot isn't set should definitely be
done, but beyond that a comment in the docs pointing out the problem
(i.e. MS releasing things that require SystemRoot be set without
updating CreateProcess to ensure that it *is* set) is as far as we
should go.

Regards,
Nick.

--
Nick Coghlan   |   ncoghlan [at] gmail   |   Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


martin at v

Dec 5, 2011, 12:10 AM

Post #10 of 17 (1429 views)
Permalink
Re: Issue 13524: subprocess on Windows [In reply to]

> Thoughts?

Apparently, there are at least two "users" of SystemRoot:
- side-by-side (fusion?) apparently uses it to locate the WinSxS
folder, at least on some Windows releases,
- certain registry keys contain SystemRoot, in particular the
path names of crypto providers (this apparently is XP only,
and fixed on Windows 7)

I agree with Nick that we shouldn't do anything except perhaps
for documentation changes. There are many other environment variables
whose absence could also cause failures to run the executable,
such as PATH, LD_LIBRARY_PATH, etc. Even not passing DISPLAY may
cause the subprocess to fail starting.

IOW, users should "normally" pass all environment variables, and
only augment it with any specific additions and deletions that
they know are needed for the subprocess. If a user deliberately
passes a small set of environment variables (e.g. none), we must
assume that it was deliberate, and that any resulting failures
are desired. People do such stuff for security reasons, and
side-stepping their enforcement is not appropriate for Python
to do.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


mail at timgolden

Dec 5, 2011, 1:01 AM

Post #11 of 17 (1430 views)
Permalink
Re: Issue 13524: subprocess on Windows [In reply to]

On 05/12/2011 08:10, "Martin v. Löwis" wrote:
> I agree with Nick that we shouldn't do anything except perhaps
> for documentation changes. There are many other environment variables
> whose absence could also cause failures to run the executable,
> such as PATH, LD_LIBRARY_PATH, etc. Even not passing DISPLAY may
> cause the subprocess to fail starting.
>
> IOW, users should "normally" pass all environment variables, and
> only augment it with any specific additions and deletions that
> they know are needed for the subprocess. If a user deliberately
> passes a small set of environment variables (e.g. none), we must
> assume that it was deliberate, and that any resulting failures
> are desired. People do such stuff for security reasons, and
> side-stepping their enforcement is not appropriate for Python
> to do.

Having slept on this I must confess that this is pretty much the
conclusion I'd come to: we can't do anything in code which is
guaranteed to be correct in every case. The best we can do is
document. And, as Martin Packman pointed out (and I had missed),
this particular condition is already documented, at least enough
to point a user to.

We could probably do with a HOWTO (or blog post or whatever) on using
subprocess on Windows, not least because a fair amount of the docs
are Unix-centric and actually very slightly confusing for naive
Windows-based developers.

I think my proposal now is: do nothing. I'm aware that Nick Coghlan
has been making fairly extensive changes to the subprocess docs
recently and I don't I can propose anything on this matter which
amounts to more than shuffling the pieces around.

TJG
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


ncoghlan at gmail

Dec 5, 2011, 1:41 AM

Post #12 of 17 (1427 views)
Permalink
Re: Issue 13524: subprocess on Windows [In reply to]

On Mon, Dec 5, 2011 at 7:01 PM, Tim Golden <mail [at] timgolden> wrote:
> We could probably do with a HOWTO (or blog post or whatever) on using
> subprocess on Windows, not least because a fair amount of the docs
> are Unix-centric and actually very slightly confusing for naive
> Windows-based developers.
>
> I think my proposal now is: do nothing. I'm aware that Nick Coghlan
> has been making fairly extensive changes to the subprocess docs
> recently and I don't I can propose anything on this matter which
> amounts to more than shuffling the pieces around.

The subprocess module could probably do with a HOWTO, full stop.
Subprocess invocation is something where platform details are always
going to matter a lot, and there are subtle details even on Unix that
are confusing (e.g. I have a command in my current project that I've
only managed to get working by running it via the shell - I still
don't know why direct invocation of the binary with the appropriate
arguments doesn't work).

At the moment, we're still trying to cram an entire essay on
subprocess invocation into the subprocess.Popen constructor
definition, which is far from optimal.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan [at] gmail   |   Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


bradallen137 at gmail

Mar 21, 2012, 1:38 PM

Post #13 of 17 (1298 views)
Permalink
Re: Issue 13524: subprocess on Windows [In reply to]

I tripped over this one trying to make one of our Python at work
Windows compatible. We had no idea that a magic 'SystemRoot'
environment variable would be required, and it was causing issues for
pyzmq.

It might be nice to reflect the findings of this email thread on the
subprocess documentation page:

http://docs.python.org/library/subprocess.html

Currently the docs mention this:

"Note If specified, env must provide any variables required for the
program to execute. On Windows, in order to run a side-by-side
assembly the specified env must include a valid SystemRoot."

How about rewording that to:

"Note If specified, env must provide any variables required for the
program to execute. On Windows, a valid SystemRoot environment
variable is required for some Python libraries such as the 'random'
module. Also, in order to run a side-by-side assembly the specified
env must include a valid SystemRoot."
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


glyph at twistedmatrix

Mar 22, 2012, 12:35 PM

Post #14 of 17 (1292 views)
Permalink
Re: Issue 13524: subprocess on Windows [In reply to]

On Mar 21, 2012, at 4:38 PM, Brad Allen wrote:

> I tripped over this one trying to make one of our Python at work
> Windows compatible. We had no idea that a magic 'SystemRoot'
> environment variable would be required, and it was causing issues for
> pyzmq.
>
> It might be nice to reflect the findings of this email thread on the
> subprocess documentation page:
>
> http://docs.python.org/library/subprocess.html
>
> Currently the docs mention this:
>
> "Note If specified, env must provide any variables required for the
> program to execute. On Windows, in order to run a side-by-side
> assembly the specified env must include a valid SystemRoot."
>
> How about rewording that to:
>
> "Note If specified, env must provide any variables required for the
> program to execute. On Windows, a valid SystemRoot environment
> variable is required for some Python libraries such as the 'random'
> module. Also, in order to run a side-by-side assembly the specified
> env must include a valid SystemRoot."

Also, in order to execute in any installation environment where libraries are found in non-default locations, you will need to set LD_LIBRARY_PATH. Oh, and you will also need to set $PATH on UNIX so that libraries can find their helper programs and %PATH% on Windows so that any compiled dynamically-loadable modules and/or DLLs can be loaded. And by the way you will also need to relay DYLD_LIBRARY_PATH if you did a UNIX-style build on OS X, not LD_LIBRARY_PATH. Don't forget that you probably also need PYTHONPATH to make sure any subprocess environments can import the same modules as their parent. Not to mention SSH_AUTH_SOCK if your application requires access to _remote_ process spawning, rather than just local. Oh and DISPLAY in case your subprocesses need GUI support from an X11 program (which sometimes you need just to initialize certain libraries which don't actually do anything with a GUI). Oh and __CF_USER_TEXT_ENCODING is important sometimes too, don't forget that. And
if your subprocess is in Perl or Ruby or Java you may need a couple dozen other variables which your deployment environment has set for you too. Did I mention CFLAGS or LC_ALL yet? Let me tell you a story about this one HP/UX machine...

Ahem.

Bottom line: it seems like screwing with the process spawning environment to make it minimal is a good idea for simplicity, for security, and for modularity. But take it from me, it isn't. I guarantee you that you don't actually know what is in your operating system's environment, and initializing it is a complicated many-step dance which some vendor or sysadmin or product integrator figured out how to do much better than your hapless Python program can.

%SystemRoot% is just the tip of a very big, very nasty iceberg. Better not to keep refining why exactly it's required, or someone will eventually be adding a new variable (starting with %APPDATA% and %HOMEPATH%) that can magically cause your subprocess not to spawn properly to this page every six months for eternity. If you're spawning processes as a regular user, you should just take the environment you're given, perhaps with a few specific light additions whose meaning you understand. If you're spawning a process as an administrator or root, you should probably initialize the environment for the user you want to spawn that process as using an OS-specific mechanism like login(1). (Sorry that I don't know the Windows equivalent.)

-glyph

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


bradallen137 at gmail

Mar 23, 2012, 10:26 AM

Post #15 of 17 (1285 views)
Permalink
Re: Issue 13524: subprocess on Windows [In reply to]

On Thu, Mar 22, 2012 at 2:35 PM, Glyph Lefkowitz
<glyph [at] twistedmatrix> wrote:

> Also, in order to execute in any installation environment where libraries are found in non-default locations, you will need to set LD_LIBRARY_PATH.  Oh, and you will also need to set $PATH on UNIX so that libraries can find their helper programs and %PATH% on Windows so that any compiled dynamically-loadable modules and/or DLLs can be loaded.  And by the way you will also need to relay DYLD_LIBRARY_PATH if you did a UNIX-style build on OS X, not LD_LIBRARY_PATH.  Don't forget that you probably also need PYTHONPATH to make sure any subprocess environments can import the same modules as their parent.  Not to mention SSH_AUTH_SOCK if your application requires access to _remote_ process spawning, rather than just local.  Oh and DISPLAY in case your subprocesses need GUI support from an X11 program (which sometimes you need just to initialize certain libraries which don't actually do anything with a GUI).  Oh and __CF_USER_TEXT_ENCODING is important sometimes too, don't forget that.  And if your subprocess is in Perl or Ruby or Java you may need a couple dozen other variables which your deployment environment has set for you too.  Did I mention CFLAGS or LC_ALL yet?  Let me tell you a story about this one HP/UX machine...
>
> Ahem.
>
> Bottom line: it seems like screwing with the process spawning environment to make it minimal is a good idea for simplicity, for security, and for modularity.  But take it from me, it isn't.  I guarantee you that you don't actually know what is in your operating system's environment, and initializing it is a complicated many-step dance which some vendor or sysadmin or product integrator figured out how to do much better than your hapless Python program can.
>
> %SystemRoot% is just the tip of a very big, very nasty iceberg.  Better not to keep refining why exactly it's required, or someone will eventually be adding a new variable (starting with %APPDATA% and %HOMEPATH%) that can magically cause your subprocess not to spawn properly to this page every six months for eternity.  If you're spawning processes as a regular user, you should just take the environment you're given, perhaps with a few specific light additions whose meaning you understand.  If you're spawning a process as an administrator or root, you should probably initialize the environment for the user you want to spawn that process as using an OS-specific mechanism like login(1).  (Sorry that I don't know the Windows equivalent.)


Thanks, Glyph. In that case maybe the Python subprocess docs need not
single out SystemRoot, but instead plaster a big warning around the
use of the 'env' parameter.:

Here is what the docs currently state for the Popen constructor 'env' parameter:

>If env is not None, it must be a mapping that defines the environment variables for the new process; these are used instead of inheriting the current process’ environment, which is the default behavior.

> Note: If specified, env must provide any variables required for the program to execute. On Windows, in order to run a side-by-side assembly the specified env must include a valid SystemRoot.

The "Note" section could instead state something like: "In most cases,
the child process will need many of the same environment variables as
the current process. Usually the safest course of action is to build
the env dict to contain all the same keys and values from os.environ.
For example... <insert Glyph's examples here>"
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


glyph at twistedmatrix

Mar 23, 2012, 1:46 PM

Post #16 of 17 (1286 views)
Permalink
Re: Issue 13524: subprocess on Windows [In reply to]

On Mar 23, 2012, at 1:26 PM, Brad Allen wrote:

> Thanks, Glyph. In that case maybe the Python subprocess docs need not
> single out SystemRoot, but instead plaster a big warning around the
> use of the 'env' parameter.

I agree. I'm glad that my bitter experience here might be useful to someone in the future - all those late nights trying desperately to get my unit tests to run on some newly configured, slightly weird buildbot didn't go to waste :).

> The "Note" section could instead state something like: "In most cases,
> the child process will need many of the same environment variables as
> the current process. Usually the safest course of action is to build
> the env dict to contain all the same keys and values from os.environ.
> For example... <insert Glyph's examples here>"

I think including all the examples might be overstating the case. It is probably best to say that other operating systems, vendors, and integration tools may set necessary environment variables that there is no way for you to be aware of in advance, unless you are an expert sysadmin on every platform where you expect your code to run, and that many of these variables are required for libraries to function properly, both libraries bundled with python and those from third parties.

-glyph


bradallen137 at gmail

Mar 23, 2012, 2:31 PM

Post #17 of 17 (1282 views)
Permalink
Re: Issue 13524: subprocess on Windows [In reply to]

On Fri, Mar 23, 2012 at 3:46 PM, Glyph <glyph [at] twistedmatrix> wrote:
> On Mar 23, 2012, at 1:26 PM, Brad Allen wrote:
>
> Thanks, Glyph. In that case maybe the Python subprocess docs need not
> single out SystemRoot, but instead plaster a big warning around the
> use of the 'env' parameter.
>
>
> I agree.  I'm glad that my bitter experience here might be useful to someone
> in the future - all those late nights trying desperately to get my unit
> tests to run on some newly configured, slightly weird buildbot didn't go to
> waste :).

Ok, I'll open a ticket on the bugtracker for this over the weekend.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Python dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.