Mailing List Archive: Python: Dev

Inherance of file descriptor and handles on Windows (PEP 446)

1 2

View All

Index | Next | Previous | View Threaded

victor.stinner at gmail

Jul 23, 2013, 3:45 PM

Post #1 of 34 (200 views)
Permalink

Inherance of file descriptor and handles on Windows (PEP 446)

Hi,

Guido van Rossum and others asked me details on how file descriptors
and handles are inherited on Windows, for the PEP 446.
http://www.python.org/dev/peps/pep-0446/

I hacked Python 3.4 to add a os.get_cloexec() function (extracted from
my implementation of the PEP 433), here are some results.

Python functions open(), os.open() and os.dup() create file
descriptors with the HANDLE_FLAG_INHERIT flag set (cloexec=False),
whereas os.pipe() creates 2 file descriptors with the
HANDLE_FLAG_INHERIT flag unset (cloexec=False, see also issue #4708).

Even if the HANDLE_FLAG_INHERIT flag is set, all handles are closed if
subprocess is used with close_fds=True (which is the default value of
the parameter), and all file descriptors are closed except 0, 1 and 2.

If close_fds=False, handles with the HANDLE_FLAG_INHERIT flag set are
inherited, but all file descriptors are still closed except 0, 1 and
2.

(I didn't check if file descriptors 0, 1 and 2 are inherited,
duplicated or new file descriptors.)

The PEP 466 allows you to control which handles are inherited to child
process when you use subprocess with close_fds=False. (The subprocess
parameter should be called "close_handles" on Windows to avoid
confusion.)

Said differently: the HANDLE_FLAG_INHERIT flag only has an effect on
*handles*, as indicated in its name. On Windows, file *descriptors*
are never inherited (are always closed) in child processes. I don't
think that it is possible to inherit file descriptors on Windows.

By the way, using pass_fds on Windows raises an assertion error
("pass_fds not supported on Windows").

Another example in Python:
---
import subprocess, sys

code = """
import os, sys
fd = int(sys.argv[1])
f = os.fdopen(fd, "rb")
print(f.read())
"""

f = open(__file__, "rb")
fd = f.fileno()
subprocess.call([sys.executable, "-c", code, str(fd)], close_fds=False)
---

On Unix, the child process will write the script into stdout. On
Windows, you just get an OSError(9, "Bad file descriptor") exception.

To fix this example on Windows, you have to:

* Retrieve the handle of the file using msvcrt.get_osfhandle() ;
* Pass the handle, instead of the file descriptor, to the child ;
* Create a file descriptor from the handle using
msvcrt.open_osfhandle() in the child.

The fix would be simpler if Python would provide the handle of a file
object (ex: in a method) and if open() supported opening a handle as
it does with file descriptors on UNIX.

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

shibturn at gmail

Jul 23, 2013, 4:21 PM

Post #2 of 34 (189 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On 23/07/2013 11:45pm, Victor Stinner wrote:
> Said differently: the HANDLE_FLAG_INHERIT flag only has an effect on
> *handles*, as indicated in its name. On Windows, file*descriptors*
> are never inherited (are always closed) in child processes. I don't
> think that it is possible to inherit file descriptors on Windows.

Actually, you can inherit fds if you use os.spawnv() instead of
subprocess.Popen().

--
Richard

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

victor.stinner at gmail

Jul 23, 2013, 4:40 PM

Post #3 of 34 (189 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

The multiprocessing module is an example of use case relying on
inherance of handles. It calls CreateProcess() with
bInheritHandles=TRUE to share a pipe between the manager (parent) and
the worker (child process).

Note: subprocess and multiprocess have their own function to set the
HANDLE_FLAG_INHERIT flag: they use DuplicateHandle(), whereas
SetHandleInformation() could be used (to reuse the existing handle
instead of creating a new handle).

2013/7/24 Victor Stinner <victor.stinner [at] gmail>:
> Python functions open(), os.open() and os.dup() create file
> descriptors with the HANDLE_FLAG_INHERIT flag set (cloexec=False),
> whereas os.pipe() creates 2 file descriptors with the
> HANDLE_FLAG_INHERIT flag unset (cloexec=False, see also issue #4708).
> (...)
> If close_fds=False, handles with the HANDLE_FLAG_INHERIT flag set are
> inherited, but all file descriptors are still closed except 0, 1 and
> 2.

Leaking handles in child processes is also an issue on Windows. Random examples.

http://bugs.python.org/issue17634
"Win32: shutil.copy leaks file handles to child processes"
"Win32's native CopyFile API call doesn't leak file handles to child processes."

http://ghc.haskell.org/trac/ghc/ticket/2650
"The case in which I originally ran into this was
System.Directory.copyFile intermittently reporting a "permission
denied" error for a temp file it was using. I think it was trying to
delete it, but failing because a child process somewhere was hanging
on to the Handle."
According to the issue, GHC calls CreateProcess with
bInheritHandles=TRUE (as Python did until Python 3.2).

http://support.microsoft.com/kb/315939
"This behavior can occur if two threads simultaneously create child
processes and redirect the STD handles through pipes. In this
scenario, there is a race condition during the creation of the pipes
and processes, in which it is possible for one child to inherit file
handles intended for the other child."
"STARTF_USESTDHANDLES" flag

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6428742
Java is still calling CreateProcess() with bInheritHandles=TRUE which
causes issues like "6347873: (so) Ports opened with
ServerSocketChannel blocks when using Runtime.exec". Interesting
comment: "Submitter has a point. Very risky to fix." :-/

For the record, the default value of close_fds parameter was also set
to True on Windows by the following changeset:

changeset: 66889:59b43dc34158
user: Gregory P. Smith <greg [at] mad-scientist>
date: Tue Dec 14 13:43:30 2010 +0000
files: Doc/library/subprocess.rst Lib/subprocess.py
Lib/test/test_subprocess.py Misc/NEWS Modules/_posixsubprocess.c
description:
Issue #6559: fix the subprocess.Popen pass_fds implementation. Add a unittest.
Issue #7213: Change the close_fds default on Windows to better match the new
default on POSIX. True when possible (False if stdin/stdout/stderr are
supplied).

Update the documentation to reflect all of the above.

The changeset was written to fix http://bugs.python.org/issue7213 ;
another example of leak of handles.

> The PEP 466 allows you to control which handles are inherited to child
> process when you use subprocess with close_fds=False. (The subprocess
> parameter should be called "close_handles" on Windows to avoid
> confusion.)

Another advantage of the PEP 446 is that most of the time, the
HANDLE_FLAG_INHERIT flag value can be set atomatically at the creation
of the file descriptor (creation of the handle). It is a nice
enhancement to fight against race conditions with threads ;-) "Most of
the time": for example, socket are inherited by default,
WSA_FLAG_NO_HANDLE_INHERIT flag was only added to Windows Vista.

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

p.f.moore at gmail

Jul 23, 2013, 11:29 PM

Post #4 of 34 (185 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On 23 July 2013 23:45, Victor Stinner <victor.stinner [at] gmail> wrote:

> Said differently: the HANDLE_FLAG_INHERIT flag only has an effect on
> *handles*, as indicated in its name. On Windows, file *descriptors*
> are never inherited (are always closed) in child processes. I don't
> think that it is possible to inherit file descriptors on Windows.
>

That is correct - handles are the OS-level concept, fds are implemented in
the CRT. So code that uses raw Windows APIs to create a new process won't
have any means to inherit fds.

> The fix would be simpler if Python would provide the handle of a file
> object (ex: in a method) and if open() supported opening a handle as
> it does with file descriptors on UNIX.
>

That would give a similar level of functionality to Unix. Whether it is
used sufficiently often to be worth it, is a separate question, of course...

Paul

guido at python

Jul 24, 2013, 10:56 AM

Post #5 of 34 (182 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On Tue, Jul 23, 2013 at 4:21 PM, Richard Oudkerk <shibturn [at] gmail> wrote:
> On 23/07/2013 11:45pm, Victor Stinner wrote:
>>
>> Said differently: the HANDLE_FLAG_INHERIT flag only has an effect on
>> *handles*, as indicated in its name. On Windows, file*descriptors*
>> are never inherited (are always closed) in child processes. I don't
>> think that it is possible to inherit file descriptors on Windows.
>
>
> Actually, you can inherit fds if you use os.spawnv() instead of
> subprocess.Popen().

Wow. Indeed you can -- I just tested this myself. How is this
accomplished? I guess the CRT has a backchannel to talk to itself when
it creates a process using spawn*?

This is the only reason I can think of for the odd default in the CRT
of opening file descriptors inheritable by default, which Victor
discovered. (But it doesn't explain why os.pipe() creates
uninheritable fds.)

If it weren't for this I would definitely vote to change the default
on Windows throughout the stdlib to create file descriptors whose
handles aren't inheritable. (Perhaps with a different policy for
stdin/stdout/stderr, which seem to be treated specially at the handle
level.)

I'm about ready to give up hope that we'll ever have a decent way to
deal with this.

But I'm also ready to propose that all this is such a mess that we
*should* change the default fd/handle inheritance to False, *across
platforms*, and damn the torpedoes -- i.e. accept breaking all
existing 3rd party UNIX code for subprocess creation that bypasses the
subprocess module, as well as breaking uses of os.spawn*() on both
platforms that depend on FD inheritance beyond stdin/stdout/stderr).

With the new, sane default, all we need instead of PEP 446 is a way to
make an FD inheritable after it's been created, which can be a single
os.make_inheritable(fd) call that you must apply to the fileno() of
the stream or socket object you want inherited (or directly to a FD
you created otherwise, e.g. with os.pipe()). On Windows, this should
probably only work with os.spawn*(), since otherwise you need *handle*
inheritance, not *FD* inheritance, and that's a non-portable concept
anyway.

We can fix multiprocessing any anything else in the stdlib that this
breaks, I presume.

To reduce the need for 3rd party subprocess creation code, we should
have better daemon creation code in the stdlib -- I wrote some damn
robust code for this purpose in my previous job, but it never saw the
light of day.

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

shibturn at gmail

Jul 24, 2013, 11:13 AM

Post #6 of 34 (183 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

> Wow. Indeed you can -- I just tested this myself. How is this
> accomplished? I guess the CRT has a backchannel to talk to itself when
> it creates a process using spawn*?

CreateProcess() takes a STARTUPINFO argument with undocumented fields
cbReserved2, lpReserved2. They are used to pass an array of fds.

> But I'm also ready to propose that all this is such a mess that we
> *should* change the default fd/handle inheritance to False, *across
> platforms*, and damn the torpedoes -- i.e. accept breaking all
> existing 3rd party UNIX code for subprocess creation that bypasses the
> subprocess module, as well as breaking uses of os.spawn*() on both
> platforms that depend on FD inheritance beyond stdin/stdout/stderr).

+1

> We can fix multiprocessing any anything else in the stdlib that this
> breaks, I presume.

In the experimental branch of multiprocessing, child processes no longer
inherit unnecessary handles.

--
Richard
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

guido at python

Jul 24, 2013, 11:17 AM

Post #7 of 34 (183 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On Wed, Jul 24, 2013 at 11:13 AM, Richard Oudkerk <shibturn [at] gmail> wrote:
>> Wow. Indeed you can -- I just tested this myself. How is this
>> accomplished? I guess the CRT has a backchannel to talk to itself when
>> it creates a process using spawn*?
>
> CreateProcess() takes a STARTUPINFO argument with undocumented fields
> cbReserved2, lpReserved2. They are used to pass an array of fds.

Does it also inherit sockets (which take up a different namespace than
regular FDs in CRT, unlike UNIX)?

>> But I'm also ready to propose that all this is such a mess that we
>>
>> *should* change the default fd/handle inheritance to False, *across
>> platforms*, and damn the torpedoes -- i.e. accept breaking all
>> existing 3rd party UNIX code for subprocess creation that bypasses the
>> subprocess module, as well as breaking uses of os.spawn*() on both
>> platforms that depend on FD inheritance beyond stdin/stdout/stderr).
>
> +1

Thanks! This was a difficult conclusion to come to. "Damn the
torpedoes" is occasionally a nice meme. :-(

>> We can fix multiprocessing any anything else in the stdlib that this
>> breaks, I presume.
>
> In the experimental branch of multiprocessing, child processes no longer
> inherit unnecessary handles.

And if we default individual handles to uninheritable, we can
presumably fix the ones that multiprocessing creates with the express
purpose of being inherited easily. (If it even uses that -- I haven't
read the source code, maybe it uses named pipes? :-)

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

shibturn at gmail

Jul 24, 2013, 11:36 AM

Post #8 of 34 (182 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On 24/07/2013 7:17pm, Guido van Rossum wrote:
> Does it also inherit sockets (which take up a different namespace than
> regular FDs in CRT, unlike UNIX)?

Not reliably. Processes created with CreateProcess() seems to inherit
socket handles just like normal handles on my computer, but on some
other computers -- with the same Windows version! -- it appears not to
work. See http://bugs.python.org/issue17399.

I think WSADuplicateSocket() should be used instead.

> And if we default individual handles to uninheritable, we can
> presumably fix the ones that multiprocessing creates with the express
> purpose of being inherited easily. (If it even uses that -- I haven't
> read the source code, maybe it uses named pipes?

multiprocessing never really needs to create any inheritable handles: it
can use DuplicateHandle() to transfer each handle directly to the child
process.

--
Richard

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

victor.stinner at gmail

Jul 24, 2013, 2:50 PM

Post #9 of 34 (182 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

2013/7/24 Richard Oudkerk <shibturn [at] gmail>:
>> Wow. Indeed you can -- I just tested this myself. How is this
>> accomplished? I guess the CRT has a backchannel to talk to itself when
>> it creates a process using spawn*?
>
> CreateProcess() takes a STARTUPINFO argument with undocumented fields
> cbReserved2, lpReserved2. They are used to pass an array of fds.

So would it be possible to implement the pass_fds parameter of
subprocess using spawnl() or the undocumented fields?

And is it possible to close all handles except one (implement "pass_handles")?

The idea would be to use something subprocess.Popen(cmd,
pass_fds=[pipe_rfd], close_fds=True) or subprocess.Popen(cmd,
pass_handles=[pipe_rhandle], close_handles=True) instead of
subprocess.Popen(cmd, close_fds=False).

> In the experimental branch of multiprocessing, child processes
> no longer inherit unnecessary handles.

Where is this branch? How did you create the channel between the
manager and the worker?

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

solipsis at pitrou

Jul 24, 2013, 2:57 PM

Post #10 of 34 (182 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On Wed, 24 Jul 2013 10:56:05 -0700
Guido van Rossum <guido [at] python> wrote:
>
> But I'm also ready to propose that all this is such a mess that we
> *should* change the default fd/handle inheritance to False, *across
> platforms*, and damn the torpedoes -- i.e. accept breaking all
> existing 3rd party UNIX code for subprocess creation that bypasses the
> subprocess module, as well as breaking uses of os.spawn*() on both
> platforms that depend on FD inheritance beyond stdin/stdout/stderr).

So I suppose you mean "change it to False except for
stdin/stdout/stderr"?

> With the new, sane default, all we need instead of PEP 446 is a way to
> make an FD inheritable after it's been created, which can be a single
> os.make_inheritable(fd) call that you must apply to the fileno() of
> the stream or socket object you want inherited (or directly to a FD
> you created otherwise, e.g. with os.pipe()). On Windows, this should
> probably only work with os.spawn*(), since otherwise you need *handle*
> inheritance, not *FD* inheritance, and that's a non-portable concept
> anyway.

I'm not sure how *fd* inheritance could work without *handle*
inheritance.
(since a fd seems to just be a proxy for a handle)

> To reduce the need for 3rd party subprocess creation code, we should
> have better daemon creation code in the stdlib -- I wrote some damn
> robust code for this purpose in my previous job, but it never saw the
> light of day.

What do you call "daemon"? An actual Unix-like daemon?

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

guido at python

Jul 24, 2013, 3:25 PM

Post #11 of 34 (182 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On Wed, Jul 24, 2013 at 2:57 PM, Antoine Pitrou <solipsis [at] pitrou> wrote:
> On Wed, 24 Jul 2013 10:56:05 -0700
> Guido van Rossum <guido [at] python> wrote:
>>
>> But I'm also ready to propose that all this is such a mess that we
>> *should* change the default fd/handle inheritance to False, *across
>> platforms*, and damn the torpedoes -- i.e. accept breaking all
>> existing 3rd party UNIX code for subprocess creation that bypasses the
>> subprocess module, as well as breaking uses of os.spawn*() on both
>> platforms that depend on FD inheritance beyond stdin/stdout/stderr).
>
> So I suppose you mean "change it to False except for
> stdin/stdout/stderr"?

Hm, that's a tricky detail. I expect that on UNIX those are pre-opened
and inherited from the shell, and we should not change their cloexec
status; IIRC on Windows inheritance of the stdin/out/err handles is
guided by a separate flag to CreateProcess().

>> With the new, sane default, all we need instead of PEP 446 is a way to
>> make an FD inheritable after it's been created, which can be a single
>> os.make_inheritable(fd) call that you must apply to the fileno() of
>> the stream or socket object you want inherited (or directly to a FD
>> you created otherwise, e.g. with os.pipe()). On Windows, this should
>> probably only work with os.spawn*(), since otherwise you need *handle*
>> inheritance, not *FD* inheritance, and that's a non-portable concept
>> anyway.
>
> I'm not sure how *fd* inheritance could work without *handle*
> inheritance.
> (since a fd seems to just be a proxy for a handle)

The MS spawn() implementation takes care of this -- I presume they
make the handles inheritable (I think some of the bugs probably refer
to this fact) and then use the "internal" interface to pass the
mapping from FDs to handles to the subprocess.

>> To reduce the need for 3rd party subprocess creation code, we should
>> have better daemon creation code in the stdlib -- I wrote some damn
>> robust code for this purpose in my previous job, but it never saw the
>> light of day.
>
> What do you call "daemon"? An actual Unix-like daemon?

Yeah, a background process with parent PID 1 and not associated with
any terminal group.

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

solipsis at pitrou

Jul 24, 2013, 3:35 PM

Post #12 of 34 (182 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On Wed, 24 Jul 2013 15:25:50 -0700
Guido van Rossum <guido [at] python> wrote:
> >> To reduce the need for 3rd party subprocess creation code, we should
> >> have better daemon creation code in the stdlib -- I wrote some damn
> >> robust code for this purpose in my previous job, but it never saw the
> >> light of day.
> >
> > What do you call "daemon"? An actual Unix-like daemon?
>
> Yeah, a background process with parent PID 1 and not associated with
> any terminal group.

But is that relevant to the PEP? A daemon only uses fork(), not exec().
Or have I misunderstood your concern?

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

shibturn at gmail

Jul 24, 2013, 3:52 PM

Post #13 of 34 (182 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On 24/07/2013 10:50pm, Victor Stinner wrote:
> So would it be possible to implement the pass_fds parameter of
> subprocess using spawnl() or the undocumented fields?

Not in a non-racy way.

spawnv() calls CreateProcess() with bInheritHandles=TRUE, so *all*
inheritable handles are inherited by the child.

The passing of the arrays of fds just makes the fds in the child process
match the fds in the parent.

If you have a Visual Studio installed then the relevant code is in

.../Microsoft Visual Studio 10.0/VC/crt/src/dospawn.c

> And is it possible to close all handles except one (implement "pass_handles")?

I don't know how to do that.

>> In the experimental branch of multiprocessing, child processes
>> no longer inherit unnecessary handles.
>
> Where is this branch? How did you create the channel between the
> manager and the worker?

http://hg.python.org/sandbox/sbt/

The parent creates a pipe and starts the child process. The pid of the
parent, and the handle for the read end of the pipe are passed on the
command line. Then the child "steals" the handle from the parent using
OpenProcess() and DuplicateHandle() using the DUPLICATE_CLOSE_SOURCE flag.

--
Richard

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

eric at trueblade

Jul 24, 2013, 4:12 PM

Post #14 of 34 (182 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On 7/24/2013 6:25 PM, Guido van Rossum wrote:
>>> To reduce the need for 3rd party subprocess creation code, we should
>>> have better daemon creation code in the stdlib -- I wrote some damn
>>> robust code for this purpose in my previous job, but it never saw the
>>> light of day.
>>
>> What do you call "daemon"? An actual Unix-like daemon?
>
> Yeah, a background process with parent PID 1 and not associated with
> any terminal group.

There's PEP 3143 and https://pypi.python.org/pypi/python-daemon. I've
used it often, with great success.

--
Eric.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

cs at zip

Jul 24, 2013, 4:54 PM

Post #15 of 34 (182 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On 25Jul2013 00:35, Antoine Pitrou <solipsis [at] pitrou> wrote:
| On Wed, 24 Jul 2013 15:25:50 -0700
| Guido van Rossum <guido [at] python> wrote:
| > >> To reduce the need for 3rd party subprocess creation code, we should
| > >> have better daemon creation code in the stdlib -- I wrote some damn
| > >> robust code for this purpose in my previous job, but it never saw the
| > >> light of day.
| > >
| > > What do you call "daemon"? An actual Unix-like daemon?
| >
| > Yeah, a background process with parent PID 1 and not associated with
| > any terminal group.
|
| But is that relevant to the PEP? A daemon only uses fork(), not exec().

Not necessarily.

I routinely run other commands in a daemonlike mode, without the
command itself having "make myself a daemon" functionality. If I
were writing the launcher in Python (and my "setsid" script is
Python already), I would be doing all the daemon bits in Python
including the fork, then execing the other tool.

Cheers,
--

If it can't be turned off, it's not a feature. - Karl Heuer
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

guido at python

Jul 24, 2013, 5:41 PM

Post #16 of 34 (183 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On Wed, Jul 24, 2013 at 3:35 PM, Antoine Pitrou <solipsis [at] pitrou> wrote:
> On Wed, 24 Jul 2013 15:25:50 -0700
> Guido van Rossum <guido [at] python> wrote:
>> >> To reduce the need for 3rd party subprocess creation code, we should
>> >> have better daemon creation code in the stdlib -- I wrote some damn
>> >> robust code for this purpose in my previous job, but it never saw the
>> >> light of day.
>> >
>> > What do you call "daemon"? An actual Unix-like daemon?
>>
>> Yeah, a background process with parent PID 1 and not associated with
>> any terminal group.
>
> But is that relevant to the PEP? A daemon only uses fork(), not exec().
> Or have I misunderstood your concern?

Actually it's common for "daemonization" libraries to run an arbitrary
executable as a daemon, and then it would be relevant. E.g. zdaemon
(https://pypi.python.org/pypi/zdaemon) does this. (Disclosure: I wrote
the first version of zdaemon when I worked for Zope Corp over a decade
ago. :-)

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

victor.stinner at gmail

Jul 26, 2013, 5:08 AM

Post #17 of 34 (165 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

2013/7/24 Guido van Rossum <guido [at] python>:
> But I'm also ready to propose that all this is such a mess that we
> *should* change the default fd/handle inheritance to False, *across
> platforms*, and damn the torpedoes -- i.e. accept breaking all
> existing 3rd party UNIX code for subprocess creation that bypasses the
> subprocess module, as well as breaking uses of os.spawn*() on both
> platforms that depend on FD inheritance beyond stdin/stdout/stderr).
>
> With the new, sane default, all we need instead of PEP 446 is a way to
> make an FD inheritable after it's been created, which can be a single
> os.make_inheritable(fd) call that you must apply to the fileno() of
> the stream or socket object you want inherited (or directly to a FD
> you created otherwise, e.g. with os.pipe()). On Windows, this should
> probably only work with os.spawn*(), since otherwise you need *handle*
> inheritance, not *FD* inheritance, and that's a non-portable concept
> anyway.

After having written 2 PEP on the topic, I slowly agree that make all
file descriptors non-inheritable is the best *compromise*. It solves
most, or all, issues.

The main drawback is the additionnal syscalls: on some platforms, 2
additional syscalls are need to make a file descriptor non-inheritable
for each creation of file descriptor. According to my benchmark on the
implementation of the PEP 433: the overhead of making a file
descriptor non-inheritable is between 1% and 3% (7.8 µs => 7.9 or 8.0
µs) on Linux 3.6.
http://www.python.org/dev/peps/pep-0433/#performances

Having to make a file descriptor inheritable after creating it
non-inheritable is also not optimal. Making it first non-inheritable
requires 0, 1 or 2 extra syscalls, and making it inheritable again
require also 1 or 2 syscalls. So f=open(...);
os.make_inheritable(f.fileno()) can take up to 5 syscalls (1 open + 4
fnctl), whereas it can be done in only 1 syscall (1 open). One of the
motivation of the PEP 433 an 446 is to reduce the number of syscalls,
even if the use case was to make sockets *non-inheritable*.

If we consider that the most common case is to use non-inheritable
file descriptors, having to call os.make_inheritable() may be
acceptable. Windows and recent operating syscalls support creating
file descriptor directly non-inheritable in a single syscalls. ioctl()
can be also be used instead of fcntl() to use 1 syscall instead of 2.

> We can fix multiprocessing any anything else in the stdlib that this
> breaks, I presume.

The CGI code rely on inheritance of file descriptors 0, 1 and 2 which
are pipes. The file descriptors 0, 1 and 2 are replaced with the pipes
using os.dup2().

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

solipsis at pitrou

Jul 26, 2013, 5:25 AM

Post #18 of 34 (166 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

Le Fri, 26 Jul 2013 14:08:35 +0200,
Victor Stinner <victor.stinner [at] gmail> a écrit :
>
> After having written 2 PEP on the topic, I slowly agree that make all
> file descriptors non-inheritable is the best *compromise*. It solves
> most, or all, issues.

Even stdin/stdout/stderr? I think inheriting them is the sane default.

> The main drawback is the additionnal syscalls: on some platforms, 2
> additional syscalls are need to make a file descriptor non-inheritable
> for each creation of file descriptor. According to my benchmark on the
> implementation of the PEP 433: the overhead of making a file
> descriptor non-inheritable is between 1% and 3% (7.8 µs => 7.9 or 8.0
> µs) on Linux 3.6.

1% and 3% of what?
You're telling us there's a 0.1µs overhead. It's positively tiny.

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

victor.stinner at gmail

Jul 26, 2013, 1:17 PM

Post #19 of 34 (162 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

2013/7/26 Antoine Pitrou <solipsis [at] pitrou>:
>> The main drawback is the additionnal syscalls: on some platforms, 2
>> additional syscalls are need to make a file descriptor non-inheritable
>> for each creation of file descriptor. According to my benchmark on the
>> implementation of the PEP 433: the overhead of making a file
>> descriptor non-inheritable is between 1% and 3% (7.8 µs => 7.9 or 8.0
>> µs) on Linux 3.6.
>
> 1% and 3% of what?
> You're telling us there's a 0.1µs overhead. It's positively tiny.

Copy-paste of the link:

"""
On Linux, setting the close-on-flag has a low overhead on
performances. Results of bench_cloexec.py on Linux 3.6:

- close-on-flag not set: 7.8 us
- O_CLOEXEC: 1% slower (7.9 us)
- ioctl(): 3% slower (8.0 us)
- fcntl(): 3% slower (8.0 us)
"""

The overhead is between 0.1 and 0.2 µs (100 and 200 ns) according to
my micro-benchmark.

"python -c pass" takes 19,000 µs (0.019 sec) on my PC. It uses 207
syscalls creating file descriptors (open() and openat()): 67 are
successful, 140 are failing with ENOENT. The estimated overhead on
"python -c pass" is 0.2*67=13.4 µs (0.07%).

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

solipsis at pitrou

Jul 26, 2013, 2:24 PM

Post #20 of 34 (162 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On Fri, 26 Jul 2013 22:17:47 +0200
Victor Stinner <victor.stinner [at] gmail> wrote:

> 2013/7/26 Antoine Pitrou <solipsis [at] pitrou>:
> >> The main drawback is the additionnal syscalls: on some platforms, 2
> >> additional syscalls are need to make a file descriptor non-inheritable
> >> for each creation of file descriptor. According to my benchmark on the
> >> implementation of the PEP 433: the overhead of making a file
> >> descriptor non-inheritable is between 1% and 3% (7.8 µs => 7.9 or 8.0
> >> µs) on Linux 3.6.
> >
> > 1% and 3% of what?
> > You're telling us there's a 0.1µs overhead. It's positively tiny.
>
> Copy-paste of the link:
>
> """
> On Linux, setting the close-on-flag has a low overhead on
> performances. Results of bench_cloexec.py on Linux 3.6:
>
> - close-on-flag not set: 7.8 us
> - O_CLOEXEC: 1% slower (7.9 us)
> - ioctl(): 3% slower (8.0 us)
> - fcntl(): 3% slower (8.0 us)
> """

You aren't answering my question: slower than what?
Benchmarking is useless if you aren't telling us what exactly you are
benchmarking.

> The overhead is between 0.1 and 0.2 µs (100 and 200 ns) according to
> my micro-benchmark.

That's what I figured out (see above). It's tiny.

Antoine.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

guido at python

Jul 26, 2013, 3:12 PM

Post #21 of 34 (162 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On Fri, Jul 26, 2013 at 5:08 AM, Victor Stinner
<victor.stinner [at] gmail> wrote:
> After having written 2 PEP on the topic, I slowly agree that make all
> file descriptors non-inheritable is the best *compromise*. It solves
> most, or all, issues.

Right.

> The main drawback is the additionnal syscalls: on some platforms, 2
> additional syscalls are need to make a file descriptor non-inheritable
> for each creation of file descriptor. According to my benchmark on the
> implementation of the PEP 433: the overhead of making a file
> descriptor non-inheritable is between 1% and 3% (7.8 �s => 7.9 or 8.0
> �s) on Linux 3.6.
> http://www.python.org/dev/peps/pep-0433/#performances

Remember that this is going to be Python 3.4 and newer. AFAICT setting
O_CLOEXEC on open works on OSX (at least the man page on OSX 10.8 has
it), on newer Linuxes, and the equivalent on Windows. So even if it
does cost an extra syscall on older systems, those systems will be
obsolete before Python 3.4 becomes mainstream there. And it does look
like the syscalls are pretty cheap. I also don't think I'm not
particularly worried about the cost of syscalls for making a socket
(non)blocking -- although we should probably avoid the second fcntl()
call if the first call shows the flag is already set the way we want
it.

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

victor.stinner at gmail

Jul 26, 2013, 3:18 PM

Post #22 of 34 (163 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

2013/7/26 Antoine Pitrou <solipsis [at] pitrou>:
> On Fri, 26 Jul 2013 22:17:47 +0200
>> """
>> On Linux, setting the close-on-flag has a low overhead on
>> performances. Results of bench_cloexec.py on Linux 3.6:
>>
>> - close-on-flag not set: 7.8 us
>> - O_CLOEXEC: 1% slower (7.9 us)
>> - ioctl(): 3% slower (8.0 us)
>> - fcntl(): 3% slower (8.0 us)
>> """
>
> You aren't answering my question: slower than what?

Ah, you didn't understand the labels. bench_cloexec.py runs a
benchmark on os.open(path, os.O_RDONLY, cloexec=False) and
os.open(path, os.O_RDONLY, cloexec=True) with different implementation
of making the file descriptor non-inheritable.

close-on-flag not set: 7.8 us
=> C code: open(path, O_RDONLY)

O_CLOEXEC: 1% slower (7.9 us)
=> C code: open(path, O_RDONLY|CLOEXEC)
=> 1% slower than open(path, O_RDONLY)

ioctl(): 3% slower (8.0 us)
=> C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0)
=> 3% slower than open(path, O_RDONLY)

fcntl(): 3% slower (8.0 us)
=> C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD);
fcntl(fd, F_SETFD, flags | FD_CLOEXEC)
=> 3% slower than open(path, O_RDONLY)

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

solipsis at pitrou

Jul 26, 2013, 3:23 PM

Post #23 of 34 (162 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On Sat, 27 Jul 2013 00:18:40 +0200
Victor Stinner <victor.stinner [at] gmail> wrote:
> 2013/7/26 Antoine Pitrou <solipsis [at] pitrou>:
> > On Fri, 26 Jul 2013 22:17:47 +0200
> >> """
> >> On Linux, setting the close-on-flag has a low overhead on
> >> performances. Results of bench_cloexec.py on Linux 3.6:
> >>
> >> - close-on-flag not set: 7.8 us
> >> - O_CLOEXEC: 1% slower (7.9 us)
> >> - ioctl(): 3% slower (8.0 us)
> >> - fcntl(): 3% slower (8.0 us)
> >> """
> >
> > You aren't answering my question: slower than what?
>
> Ah, you didn't understand the labels. bench_cloexec.py runs a
> benchmark on os.open(path, os.O_RDONLY, cloexec=False) and
> os.open(path, os.O_RDONLY, cloexec=True) with different implementation
> of making the file descriptor non-inheritable.
>
> close-on-flag not set: 7.8 us
> => C code: open(path, O_RDONLY)
>
> O_CLOEXEC: 1% slower (7.9 us)
> => C code: open(path, O_RDONLY|CLOEXEC)
> => 1% slower than open(path, O_RDONLY)
>
> ioctl(): 3% slower (8.0 us)
> => C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0)
> => 3% slower than open(path, O_RDONLY)
>
> fcntl(): 3% slower (8.0 us)
> => C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD);
> fcntl(fd, F_SETFD, flags | FD_CLOEXEC)
> => 3% slower than open(path, O_RDONLY)

Ok, so I think this it is a totally reasonable compromise.

People who bother about a 3% slowdown when calling os.open() can
optimize the hell out of their code using Cython for all I care :-)

Regards

Antoine.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

greg at krypto

Jul 26, 2013, 9:26 PM

Post #24 of 34 (155 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On Fri, Jul 26, 2013 at 3:23 PM, Antoine Pitrou <solipsis [at] pitrou> wrote:

> On Sat, 27 Jul 2013 00:18:40 +0200
> Victor Stinner <victor.stinner [at] gmail> wrote:
> > 2013/7/26 Antoine Pitrou <solipsis [at] pitrou>:
> > > On Fri, 26 Jul 2013 22:17:47 +0200
> > >> """
> > >> On Linux, setting the close-on-flag has a low overhead on
> > >> performances. Results of bench_cloexec.py on Linux 3.6:
> > >>
> > >> - close-on-flag not set: 7.8 us
> > >> - O_CLOEXEC: 1% slower (7.9 us)
> > >> - ioctl(): 3% slower (8.0 us)
> > >> - fcntl(): 3% slower (8.0 us)
> > >> """
> > >
> > > You aren't answering my question: slower than what?
> >
> > Ah, you didn't understand the labels. bench_cloexec.py runs a
> > benchmark on os.open(path, os.O_RDONLY, cloexec=False) and
> > os.open(path, os.O_RDONLY, cloexec=True) with different implementation
> > of making the file descriptor non-inheritable.
> >
> > close-on-flag not set: 7.8 us
> > => C code: open(path, O_RDONLY)
> >
> > O_CLOEXEC: 1% slower (7.9 us)
> > => C code: open(path, O_RDONLY|CLOEXEC)
> > => 1% slower than open(path, O_RDONLY)
> >
> > ioctl(): 3% slower (8.0 us)
> > => C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0)
> > => 3% slower than open(path, O_RDONLY)
> >
> > fcntl(): 3% slower (8.0 us)
> > => C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD);
> > fcntl(fd, F_SETFD, flags | FD_CLOEXEC)
> > => 3% slower than open(path, O_RDONLY)
>
> Ok, so I think this it is a totally reasonable compromise.
>
> People who bother about a 3% slowdown when calling os.open() can
> optimize the hell out of their code using Cython for all I care :-)
>
>
+1 ;)

and +1 for making the sane default of noinherit / cloexec /
whatever-others-call-it by default for all fds/handles ever opened by
Python. It stops ignoring the issue (ie: the status quo of matching the
default behavior of C as defined in the 1970s)... That is a GOOD thing. :)

-gps

guido at python

Jul 26, 2013, 9:36 PM

Post #25 of 34 (155 views)
Permalink

Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On Fri, Jul 26, 2013 at 9:26 PM, Gregory P. Smith <greg [at] krypto> wrote:
>
> On Fri, Jul 26, 2013 at 3:23 PM, Antoine Pitrou <solipsis [at] pitrou> wrote:
>>
>> On Sat, 27 Jul 2013 00:18:40 +0200
>> Victor Stinner <victor.stinner [at] gmail> wrote:
>> > 2013/7/26 Antoine Pitrou <solipsis [at] pitrou>:
>> > > On Fri, 26 Jul 2013 22:17:47 +0200
>> > >> """
>> > >> On Linux, setting the close-on-flag has a low overhead on
>> > >> performances. Results of bench_cloexec.py on Linux 3.6:
>> > >>
>> > >> - close-on-flag not set: 7.8 us
>> > >> - O_CLOEXEC: 1% slower (7.9 us)
>> > >> - ioctl(): 3% slower (8.0 us)
>> > >> - fcntl(): 3% slower (8.0 us)
>> > >> """
>> > >
>> > > You aren't answering my question: slower than what?
>> >
>> > Ah, you didn't understand the labels. bench_cloexec.py runs a
>> > benchmark on os.open(path, os.O_RDONLY, cloexec=False) and
>> > os.open(path, os.O_RDONLY, cloexec=True) with different implementation
>> > of making the file descriptor non-inheritable.
>> >
>> > close-on-flag not set: 7.8 us
>> > => C code: open(path, O_RDONLY)
>> >
>> > O_CLOEXEC: 1% slower (7.9 us)
>> > => C code: open(path, O_RDONLY|CLOEXEC)
>> > => 1% slower than open(path, O_RDONLY)
>> >
>> > ioctl(): 3% slower (8.0 us)
>> > => C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0)
>> > => 3% slower than open(path, O_RDONLY)
>> >
>> > fcntl(): 3% slower (8.0 us)
>> > => C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD);
>> > fcntl(fd, F_SETFD, flags | FD_CLOEXEC)
>> > => 3% slower than open(path, O_RDONLY)
>>
>> Ok, so I think this it is a totally reasonable compromise.
>>
>> People who bother about a 3% slowdown when calling os.open() can
>> optimize the hell out of their code using Cython for all I care :-)
>>
>
> +1 ;)
>
> and +1 for making the sane default of noinherit / cloexec /
> whatever-others-call-it by default for all fds/handles ever opened by
> Python. It stops ignoring the issue (ie: the status quo of matching the
> default behavior of C as defined in the 1970s)... That is a GOOD thing. :)

Do we even need a new PEP, or should we just do it? Or can we adapt
Victor's PEP 446?

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads