Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Dev

Inherance of file descriptor and handles on Windows (PEP 446)

 

 

First page Previous page 1 2 Next page Last page  View All Python dev RSS feed   Index | Next | Previous | View Threaded


ncoghlan at gmail

Jul 26, 2013, 10:04 PM

Post #26 of 34 (70 views)
Permalink
Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On 27 July 2013 14:36, Guido van Rossum <guido [at] python> wrote:
> On Fri, Jul 26, 2013 at 9:26 PM, Gregory P. Smith <greg [at] krypto> wrote:
>>
>> On Fri, Jul 26, 2013 at 3:23 PM, Antoine Pitrou <solipsis [at] pitrou> wrote:
>>>
>>> On Sat, 27 Jul 2013 00:18:40 +0200
>>> Victor Stinner <victor.stinner [at] gmail> wrote:
>>> > 2013/7/26 Antoine Pitrou <solipsis [at] pitrou>:
>>> > > On Fri, 26 Jul 2013 22:17:47 +0200
>>> > >> """
>>> > >> On Linux, setting the close-on-flag has a low overhead on
>>> > >> performances. Results of bench_cloexec.py on Linux 3.6:
>>> > >>
>>> > >> - close-on-flag not set: 7.8 us
>>> > >> - O_CLOEXEC: 1% slower (7.9 us)
>>> > >> - ioctl(): 3% slower (8.0 us)
>>> > >> - fcntl(): 3% slower (8.0 us)
>>> > >> """
>>> > >
>>> > > You aren't answering my question: slower than what?
>>> >
>>> > Ah, you didn't understand the labels. bench_cloexec.py runs a
>>> > benchmark on os.open(path, os.O_RDONLY, cloexec=False) and
>>> > os.open(path, os.O_RDONLY, cloexec=True) with different implementation
>>> > of making the file descriptor non-inheritable.
>>> >
>>> > close-on-flag not set: 7.8 us
>>> > => C code: open(path, O_RDONLY)
>>> >
>>> > O_CLOEXEC: 1% slower (7.9 us)
>>> > => C code: open(path, O_RDONLY|CLOEXEC)
>>> > => 1% slower than open(path, O_RDONLY)
>>> >
>>> > ioctl(): 3% slower (8.0 us)
>>> > => C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0)
>>> > => 3% slower than open(path, O_RDONLY)
>>> >
>>> > fcntl(): 3% slower (8.0 us)
>>> > => C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD);
>>> > fcntl(fd, F_SETFD, flags | FD_CLOEXEC)
>>> > => 3% slower than open(path, O_RDONLY)
>>>
>>> Ok, so I think this it is a totally reasonable compromise.
>>>
>>> People who bother about a 3% slowdown when calling os.open() can
>>> optimize the hell out of their code using Cython for all I care :-)
>>>
>>
>> +1 ;)
>>
>> and +1 for making the sane default of noinherit / cloexec /
>> whatever-others-call-it by default for all fds/handles ever opened by
>> Python. It stops ignoring the issue (ie: the status quo of matching the
>> default behavior of C as defined in the 1970s)... That is a GOOD thing. :)
>
> Do we even need a new PEP, or should we just do it? Or can we adapt
> Victor's PEP 446?

Adapting the PEP sounds good - while I agree with switching to a sane
default, I think the daemonisation thread suggests there may need to
be a supporting API to help force FDs created by nominated logging
handlers to be inherited.

Cheers,
Nick.

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


victor.stinner at gmail

Jul 27, 2013, 6:44 AM

Post #27 of 34 (64 views)
Permalink
Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

2013/7/27 Guido van Rossum <guido [at] python>:
> Do we even need a new PEP, or should we just do it? Or can we adapt
> Victor's PEP 446?

I can rewrite the PEP 446 to:

* make all file descriptors and handles non-inheritable
* remove the cloexec parameter
* remove everything about non-blocking sockets (O_NONBLOCK), it should
be discussed in a new PEP (it's no more related to O_CLOEXEC /
HANDLE_INHERIT_FLAG)

Should I rename os.set_cloexec(fd, cloexec) to os.set_inheritable(fd,
inheritable), and os.get_cloexec(fd) to os.get_inheritable(fd)?

Or do you prefer a simple os.make_inheritable(fd) with no inheritable
parameter? I prefer an explicit parameter, so it's also possible to
force again non-inheritable, which also makes sense if the file
descriptor was not created by Python.

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


guido at python

Jul 27, 2013, 7:36 AM

Post #28 of 34 (64 views)
Permalink
Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On Saturday, July 27, 2013, Victor Stinner wrote:

> 2013/7/27 Guido van Rossum <guido [at] python <javascript:;>>:
> > Do we even need a new PEP, or should we just do it? Or can we adapt
> > Victor's PEP 446?
>
> I can rewrite the PEP 446 to:
>
> * make all file descriptors and handles non-inheritable
> * remove the cloexec parameter
> * remove everything about non-blocking sockets (O_NONBLOCK), it should
> be discussed in a new PEP (it's no more related to O_CLOEXEC /
> HANDLE_INHERIT_FLAG)


Sounds good.

>
> Should I rename os.set_cloexec(fd, cloexec) to os.set_inheritable(fd,
> inheritable), and os.get_cloexec(fd) to os.get_inheritable(fd)?


Yes.

>
> Or do you prefer a simple os.make_inheritable(fd) with no inheritable
> parameter? I prefer an explicit parameter, so it's also possible to
> force again non-inheritable, which also makes sense if the file
> descriptor was not created by Python.


Agreed.

>
> Victor
>


--
--Guido van Rossum (on iPad)


guido at python

Jul 27, 2013, 7:41 AM

Post #29 of 34 (65 views)
Permalink
Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

P.S. perhaps more important than a PEP rewrite is a working patch to see
how realistic this is. Could you make the alpha 1 release?

On Saturday, July 27, 2013, Guido van Rossum wrote:

>
>
> On Saturday, July 27, 2013, Victor Stinner wrote:
>
>> 2013/7/27 Guido van Rossum <guido [at] python>:
>> > Do we even need a new PEP, or should we just do it? Or can we adapt
>> > Victor's PEP 446?
>>
>> I can rewrite the PEP 446 to:
>>
>> * make all file descriptors and handles non-inheritable
>> * remove the cloexec parameter
>> * remove everything about non-blocking sockets (O_NONBLOCK), it should
>> be discussed in a new PEP (it's no more related to O_CLOEXEC /
>> HANDLE_INHERIT_FLAG)
>
>
> Sounds good.
>
>>
>> Should I rename os.set_cloexec(fd, cloexec) to os.set_inheritable(fd,
>> inheritable), and os.get_cloexec(fd) to os.get_inheritable(fd)?
>
>
> Yes.
>
>>
>> Or do you prefer a simple os.make_inheritable(fd) with no inheritable
>> parameter? I prefer an explicit parameter, so it's also possible to
>> force again non-inheritable, which also makes sense if the file
>> descriptor was not created by Python.
>
>
> Agreed.
>
>>
>> Victor
>>
>
>
> --
> --Guido van Rossum (on iPad)
>


--
--Guido van Rossum (on iPad)


victor.stinner at gmail

Jul 27, 2013, 10:56 AM

Post #30 of 34 (63 views)
Permalink
Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

2013/7/27 Guido van Rossum <guido [at] python>:
> P.S. perhaps more important than a PEP rewrite is a working patch to see how
> realistic this is. Could you make the alpha 1 release?

I already ran the whole Python test suite with non-inheritable file
descriptors when I developed the PEP 433: it just works. So I'm
confident :-)

I "just" had to fix the cgi module, and some tests. For example,
test_socket checks the exact type of sockets, whereas SOCK_CLOEXEC
flag is present in sockobj.type for non-inheritable sockets created
with this flag.

I implemented the *new* PEP 446 (not written yet :-)) in a new repository:
http://hg.python.org/features/pep-446

I had to invert the value of cloexec (inheritable value is just the opposite).

The implementation works but it is not completed:

* The doc should be reviewed
* test_swap_fds() of test_subprocess fails
* The implementation should be tested on Windows, FreeBSD and Solaris
* I have to check if _Py_try_set_inheritable() can/must be replaced
with _Py_set_inheritable()

The implementation can be seen as a patch and reviewed using the
following new issue:
http://bugs.python.org/issue18571

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


victor.stinner at gmail

Aug 5, 2013, 5:52 AM

Post #31 of 34 (25 views)
Permalink
Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

I checked the python-daemon module: it closes all open file
descriptors except 0, 1, 2. It has a files_preserve attribute to keep
some FD opens. It redirects stdin, stdout and stderr to /dev/null and
keep these file descriptors open. If python-daemon is used to execute
a new program, the files_preserve list can be used to mark these file
descriptors as inherited.

The zdaemon.zdrun module closes all open file descriptors except 0, 1,
2. It uses also dup2() to redirect stdout and stderr to the write end
of a pipe.

Victor

2013/7/25 Eric V. Smith <eric [at] trueblade>:
> On 7/24/2013 6:25 PM, Guido van Rossum wrote:
>>>> To reduce the need for 3rd party subprocess creation code, we should
>>>> have better daemon creation code in the stdlib -- I wrote some damn
>>>> robust code for this purpose in my previous job, but it never saw the
>>>> light of day.
>>>
>>> What do you call "daemon"? An actual Unix-like daemon?
>>
>> Yeah, a background process with parent PID 1 and not associated with
>> any terminal group.
>
> There's PEP 3143 and https://pypi.python.org/pypi/python-daemon. I've
> used it often, with great success.
>
> --
> Eric.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev [at] python
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


victor.stinner at gmail

Aug 5, 2013, 5:56 AM

Post #32 of 34 (25 views)
Permalink
Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

2013/7/27 Nick Coghlan <ncoghlan [at] gmail>:
>> Do we even need a new PEP, or should we just do it? Or can we adapt
>> Victor's PEP 446?
>
> Adapting the PEP sounds good - while I agree with switching to a sane
> default, I think the daemonisation thread suggests there may need to
> be a supporting API to help force FDs created by nominated logging
> handlers to be inherited.

Why would a Python logging handler be used in a child process? If the
child process is a fresh Python process, it starts with the default
logging handlers (no handler).

Files opened by the logging module must be closed on exec().

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


ncoghlan at gmail

Aug 5, 2013, 8:12 AM

Post #33 of 34 (25 views)
Permalink
Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On 5 August 2013 22:52, Victor Stinner <victor.stinner [at] gmail> wrote:
> I checked the python-daemon module: it closes all open file
> descriptors except 0, 1, 2. It has a files_preserve attribute to keep
> some FD opens. It redirects stdin, stdout and stderr to /dev/null and
> keep these file descriptors open. If python-daemon is used to execute
> a new program, the files_preserve list can be used to mark these file
> descriptors as inherited.
>
> The zdaemon.zdrun module closes all open file descriptors except 0, 1,
> 2. It uses also dup2() to redirect stdout and stderr to the write end
> of a pipe.

So closed by default, and directing people towards subprocess and
python-daemon if they need to keep descriptors open sounds really
promising.

Cheers,
Nick.

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


solipsis at pitrou

Aug 5, 2013, 11:44 AM

Post #34 of 34 (25 views)
Permalink
Re: Inherance of file descriptor and handles on Windows (PEP 446) [In reply to]

On Mon, 5 Aug 2013 14:56:06 +0200
Victor Stinner <victor.stinner [at] gmail> wrote:
> 2013/7/27 Nick Coghlan <ncoghlan [at] gmail>:
> >> Do we even need a new PEP, or should we just do it? Or can we adapt
> >> Victor's PEP 446?
> >
> > Adapting the PEP sounds good - while I agree with switching to a sane
> > default, I think the daemonisation thread suggests there may need to
> > be a supporting API to help force FDs created by nominated logging
> > handlers to be inherited.
>
> Why would a Python logging handler be used in a child process? If the
> child process is a fresh Python process, it starts with the default
> logging handlers (no handler).
>
> Files opened by the logging module must be closed on exec().

I agree with this. It is only on fork()-without-exec() that the
behaviour of python-daemon is actively anti-social.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

First page Previous page 1 2 Next page Last page  View All Python dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.