Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Python

tee-like behavior in Python

 

 

Python python RSS feed   Index | Next | Previous | View Threaded


mailinglists at xgm

May 9, 2012, 8:35 AM

Post #1 of 6 (639 views)
Permalink
tee-like behavior in Python

Hello,

how can I achieve a behavior like tee in Python?

* execute an application
* leave the output to stdout and stderr untouched
* but capture both and save it to a file (resp. file-like object)

I have this code

proc = subprocess.Popen(shlex.split(cmd), stdout = subprocess.PIPE,
stderr=subprocess.STDOUT)
while True:
out = proc.stdout.readline()
if out == '' and proc.poll() != None:
break
sys.stdout.write(out)
logfile.write(out)

This works so far but always buffers a couple of lines and outputs
them en bloc. The final output is like it is desired but with a
significant delay. Is there a way around that?

Thanks,

Florian
--
http://mail.python.org/mailman/listinfo/python-list


d at davea

May 9, 2012, 9:14 AM

Post #2 of 6 (608 views)
Permalink
Re: tee-like behavior in Python [In reply to]

On 05/09/2012 11:35 AM, Florian Lindner wrote:
> Hello,
>
> how can I achieve a behavior like tee in Python?
>
> * execute an application
> * leave the output to stdout and stderr untouched
> * but capture both and save it to a file (resp. file-like object)
>
> I have this code
>
> proc = subprocess.Popen(shlex.split(cmd), stdout = subprocess.PIPE,
> stderr=subprocess.STDOUT)
> while True:
> out = proc.stdout.readline()
> if out == '' and proc.poll() != None:
> break
> sys.stdout.write(out)
> logfile.write(out)
>
> This works so far but always buffers a couple of lines and outputs
> them en bloc. The final output is like it is desired but with a
> significant delay. Is there a way around that?
>
> Thanks,
>
> Florian
Chances are that other program is buffering its output. Many programs
check if they're running on a tty, and turn off buffering for
interactive use. But redirect them into a pipe, and they'll run as
efficiently as possible, which includes buffering the output.



--

DaveA

--
http://mail.python.org/mailman/listinfo/python-list


dreadpiratejeff at gmail

May 9, 2012, 9:22 AM

Post #3 of 6 (607 views)
Permalink
Re: tee-like behavior in Python [In reply to]

On Wed, May 9, 2012 at 11:35 AM, Florian Lindner <mailinglists [at] xgm> wrote:
> Hello,
>
> how can I achieve a behavior like tee in Python?
>
> * execute an application
> * leave the output to stdout and stderr untouched
> * but capture both and save it to a file (resp. file-like object)
>
> I have this code
>
> proc = subprocess.Popen(shlex.split(cmd), stdout = subprocess.PIPE,
> stderr=subprocess.STDOUT)
> while True:
>    out = proc.stdout.readline()
>    if out == '' and proc.poll() != None:
>       break
>    sys.stdout.write(out)
>    logfile.write(out)
>
> This works so far but always buffers a couple of lines and outputs
> them en bloc. The final output is like it is desired but with a
> significant delay. Is there a way around that?

Perhaps this would help:

http://docs.python.org/library/subprocess.html#popen-constructor

specifically, the bits about setting bufsize to 0 indicating
unbuffered behaviour.
--
http://mail.python.org/mailman/listinfo/python-list


wrw at mac

May 9, 2012, 11:19 AM

Post #4 of 6 (603 views)
Permalink
Re: tee-like behavior in Python [In reply to]

On May 9, 2012, at 11:35 AM, Florian Lindner wrote:

> Hello,
>
> how can I achieve a behavior like tee in Python?
>
> * execute an application
> * leave the output to stdout and stderr untouched
> * but capture both and save it to a file (resp. file-like object)
>
> I have this code
>
> proc = subprocess.Popen(shlex.split(cmd), stdout = subprocess.PIPE,
> stderr=subprocess.STDOUT)
> while True:
> out = proc.stdout.readline()
> if out == '' and proc.poll() != None:
> break
> sys.stdout.write(out)
> logfile.write(out)
>
> This works so far but always buffers a couple of lines and outputs
> them en bloc. The final output is like it is desired but with a
> significant delay. Is there a way around that?
>
> Thanks,
>
> Florian
> --
> http://mail.python.org/mailman/listinfo/python-list

Have you tried explicitly calling file.flush() on the log file? (The docs note that you may have to follow this with os.fsync() on some systems.)

-Bill
--
http://mail.python.org/mailman/listinfo/python-list


drsalists at gmail

May 9, 2012, 1:58 PM

Post #5 of 6 (606 views)
Permalink
Re: tee-like behavior in Python [In reply to]

You've had some good responses already, but here're two more:

1) Easiest would be to use setvbuf in the child process, if you have access
to its source. This allows you to force line-oriented buffering.

2) stdio likes to buffer to tty/pty's in a line-oriented manner, and other
things in a block-oriented manner - by default, so users get pleasing
output increments, and programs get efficiency. To trick stdio into
buffering line-oriented by default when it's sending output to another
program, you may have to talk to the child process using a pty, AKA a
pseudo terminal. There are a few ways of doing this:
A) Use CPython's pty module.
B) Use something like pexpect.
C) Check out the pty program in comp.sources.unix volume 25. This might
be pretty easy too,
assuming that pty still builds on a modern *ix.

On Wed, May 9, 2012 at 8:35 AM, Florian Lindner <mailinglists [at] xgm> wrote:

> Hello,
>
> how can I achieve a behavior like tee in Python?
>
> * execute an application
> * leave the output to stdout and stderr untouched
> * but capture both and save it to a file (resp. file-like object)
>
> I have this code
>
> proc = subprocess.Popen(shlex.split(cmd), stdout = subprocess.PIPE,
> stderr=subprocess.STDOUT)
> while True:
> out = proc.stdout.readline()
> if out == '' and proc.poll() != None:
> break
> sys.stdout.write(out)
> logfile.write(out)
>
> This works so far but always buffers a couple of lines and outputs
> them en bloc. The final output is like it is desired but with a
> significant delay. Is there a way around that?
>
> Thanks,
>
> Florian
> --
> http://mail.python.org/mailman/listinfo/python-list
>


clp2 at rebertia

May 9, 2012, 2:51 PM

Post #6 of 6 (600 views)
Permalink
Re: tee-like behavior in Python [In reply to]

On Wed, May 9, 2012 at 8:35 AM, Florian Lindner <mailinglists [at] xgm> wrote:
> Hello,
>
> how can I achieve a behavior like tee in Python?
>
> * execute an application
> * leave the output to stdout and stderr untouched
> * but capture both and save it to a file (resp. file-like object)
>
> I have this code
>
> proc = subprocess.Popen(shlex.split(cmd), stdout = subprocess.PIPE,

shlex.split() should just be used once, at "development time", to
determine the form of the argument list. It shouldn't generally be
used at runtime. Otherwise, you need to do the proper escaping/quoting
yourself, which defeats the entire purpose of bypassing the shell.

Cheers,
Chris
--
http://mail.python.org/mailman/listinfo/python-list

Python python RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.