Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Python

Interprocess comunication

 

 

Python python RSS feed   Index | Next | Previous | View Threaded


juliosergio at gmail

Jun 7, 2012, 9:04 AM

Post #1 of 8 (435 views)
Permalink
Interprocess comunication

I'm trying to call an external process to filter some of my data, i.e., I'm
trying to pass some information to the called process, and have this information
back transformed. I started testing with the linux 'cat' command, in this way:


->>> import subprocess as sp
->>> p = sp.Popen(["cat"],stdin=sp.PIPE,stdout=sp.PIPE,close_fds=True)
->>> (fi,fo) = (p.stdin, p.stdout)
->>> fi.write("SOMETHING\n")
->>> fi.flush()
->>> fo.readline()
'SOMETHING\n'
->>> fi.write("OTHER\n")
->>> fi.flush()
->>> fo.readline()
'OTHER\n'
->>> fi.write("NEXT\n")
->>> fi.flush()
->>> fo.readline()
'NEXT\n'
->>> fi.write("NEXT1\n")
->>> fi.flush()
->>> s = fo.readline()
->>> s
'NEXT1\n'

Up to this point it worked as expected. However, when I tryied with the methods
that write and read several lines, apparently the process got stalled:

->>> fi.writelines(["uno\n","dos\n","tres\n"])
->>> fi.flush()
->>> s = fo.readlines()
.
.
.

Do you have any comments on this?

Thanks,

--Sergio


--
http://mail.python.org/mailman/listinfo/python-list


oscar.benjamin at bristol

Jun 7, 2012, 9:25 AM

Post #2 of 8 (429 views)
Permalink
Re: Interprocess comunication [In reply to]

On 7 June 2012 17:04, Julio Sergio <juliosergio [at] gmail> wrote:

> I'm trying to call an external process to filter some of my data, i.e., I'm
> trying to pass some information to the called process, and have this
> information
> back transformed. I started testing with the linux 'cat' command, in this
> way:
>
>
> ->>> import subprocess as sp
> ->>> p = sp.Popen(["cat"],stdin=sp.PIPE,stdout=sp.PIPE,close_fds=True)
> ->>> (fi,fo) = (p.stdin, p.stdout)
> ->>> fi.write("SOMETHING\n")
> ->>> fi.flush()
> ->>> fo.readline()
> 'SOMETHING\n'
> ->>> fi.write("OTHER\n")
> ->>> fi.flush()
> ->>> fo.readline()
> 'OTHER\n'
> ->>> fi.write("NEXT\n")
> ->>> fi.flush()
> ->>> fo.readline()
> 'NEXT\n'
> ->>> fi.write("NEXT1\n")
> ->>> fi.flush()
> ->>> s = fo.readline()
> ->>> s
> 'NEXT1\n'
>
> Up to this point it worked as expected. However, when I tryied with the
> methods
> that write and read several lines, apparently the process got stalled:
>
> ->>> fi.writelines(["uno\n","dos\n","tres\n"])
>
->>> fi.flush()
>
->>> s = fo.readlines()
>

The readlines() method is intended to read an entire file and split it into
lines, so it blocks until EOF is found. If you want to use readlines(), you
should first close the subprocesses stdin:

>>> fi.writelines(["uno\n","dos\n","tres\n"])
>>> fi.flush()
>>> fi.close()
>>> s = fo.readlines()

Although now you can no longer use the subprocess for reading or writing.
If that is not what you want in your actual problem then you'll need to
avoid readlines().


> .
> .
> .
>
> Do you have any comments on this?
>
> Thanks,
>
> --Sergio
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>


python at mrabarnett

Jun 7, 2012, 9:29 AM

Post #3 of 8 (430 views)
Permalink
Re: Interprocess comunication [In reply to]

On 07/06/2012 17:04, Julio Sergio wrote:
> I'm trying to call an external process to filter some of my data, i.e., I'm
> trying to pass some information to the called process, and have this information
> back transformed. I started testing with the linux 'cat' command, in this way:
>
>
> ->>> import subprocess as sp
> ->>> p = sp.Popen(["cat"],stdin=sp.PIPE,stdout=sp.PIPE,close_fds=True)
> ->>> (fi,fo) = (p.stdin, p.stdout)
> ->>> fi.write("SOMETHING\n")
> ->>> fi.flush()
> ->>> fo.readline()
> 'SOMETHING\n'
> ->>> fi.write("OTHER\n")
> ->>> fi.flush()
> ->>> fo.readline()
> 'OTHER\n'
> ->>> fi.write("NEXT\n")
> ->>> fi.flush()
> ->>> fo.readline()
> 'NEXT\n'
> ->>> fi.write("NEXT1\n")
> ->>> fi.flush()
> ->>> s = fo.readline()
> ->>> s
> 'NEXT1\n'
>
> Up to this point it worked as expected. However, when I tryied with the methods
> that write and read several lines, apparently the process got stalled:
>
> ->>> fi.writelines(["uno\n","dos\n","tres\n"])
> ->>> fi.flush()
> ->>> s = fo.readlines()
> .
> .
> .
>
> Do you have any comments on this?
>
I believe it's waiting for the end of the input, i.e. for the pipe to
close.

Have you tried calling fo.readline() 3 times instead?
--
http://mail.python.org/mailman/listinfo/python-list


jcd at sdf

Jun 7, 2012, 9:30 AM

Post #4 of 8 (431 views)
Permalink
Re: Interprocess comunication [In reply to]

On Thu, 2012-06-07 at 16:04 +0000, Julio Sergio wrote:

> Up to this point it worked as expected. However, when I tryied with the methods
> that write and read several lines, apparently the process got stalled:
>
> ->>> fi.writelines(["uno\n","dos\n","tres\n"])
> ->>> fi.flush()
> ->>> s = fo.readlines()
> .
> .
> .

readlines() reads all the lines from the filehandle, but the filehandle
hasn't signalled that it is done writing lines, so fo is waiting until
fi is complete. You either need to keep reading one line at a time, and
manually release control when there's nothing more to read, or you need
to do an fi.close() before you try to use fo.readlines().



--
http://mail.python.org/mailman/listinfo/python-list


juliosergio at gmail

Jun 7, 2012, 9:39 AM

Post #5 of 8 (431 views)
Permalink
Re: Interprocess comunication [In reply to]

MRAB <python <at> mrabarnett.plus.com> writes:

>
> I believe it's waiting for the end of the input, i.e. for the pipe to
> close.
>
> Have you tried calling fo.readline() 3 times instead?
>

yeah! It worked!...
A question remains: what is then the purpose of fo.readlines(...)?

Thanks,
--Sergio




--
http://mail.python.org/mailman/listinfo/python-list


juliosergio at gmail

Jun 7, 2012, 9:45 AM

Post #6 of 8 (430 views)
Permalink
Re: Interprocess comunication [In reply to]

J. Cliff Dyer <jcd <at> sdf.lonestar.org> writes:

>
> readlines() reads all the lines from the filehandle, but the filehandle
> hasn't signalled that it is done writing lines, so fo is waiting until
> fi is complete. You either need to keep reading one line at a time, and
> manually release control when there's nothing more to read, or you need
> to do an fi.close() before you try to use fo.readlines().
>
>

Thanks... It worked as you said. Here is how I coded it:

->>> fi.writelines(["uno\n","dos\n","tres\n"])
->>> fi.close()
->>> l = fo.readlines()
->>> l
['uno\n', 'dos\n', 'tres\n']


--Sergio

--
http://mail.python.org/mailman/listinfo/python-list


jcd at sdf

Jun 7, 2012, 9:56 AM

Post #7 of 8 (434 views)
Permalink
Re: Interprocess comunication [In reply to]

It is for reading all the lines from a complete file. If the file is
still being written to, it doesn't have an end yet. File objects do
many things besides RPC. Also, there are instances where all you want
to do is block until the file is done, and then get all the content.
readlines will do that, too.



On Thu, 2012-06-07 at 16:39 +0000, Julio Sergio wrote:
> MRAB <python <at> mrabarnett.plus.com> writes:
>
> >
> > I believe it's waiting for the end of the input, i.e. for the pipe to
> > close.
> >
> > Have you tried calling fo.readline() 3 times instead?
> >
>
> yeah! It worked!...
> A question remains: what is then the purpose of fo.readlines(...)?
>
> Thanks,
> --Sergio
>
>
>
>


--
http://mail.python.org/mailman/listinfo/python-list


__peter__ at web

Jun 7, 2012, 10:42 AM

Post #8 of 8 (427 views)
Permalink
Re: Interprocess comunication [In reply to]

Julio Sergio wrote:

> J. Cliff Dyer <jcd <at> sdf.lonestar.org> writes:
>
>>
>> readlines() reads all the lines from the filehandle, but the filehandle
>> hasn't signalled that it is done writing lines, so fo is waiting until
>> fi is complete. You either need to keep reading one line at a time, and
>> manually release control when there's nothing more to read, or you need
>> to do an fi.close() before you try to use fo.readlines().
>>
>>
>
> Thanks... It worked as you said. Here is how I coded it:
>
> ->>> fi.writelines(["uno\n","dos\n","tres\n"])
> ->>> fi.close()
> ->>> l = fo.readlines()
> ->>> l
> ['uno\n', 'dos\n', 'tres\n']

I believe this may hang on the fi.writelines(...) line if you write "too
much" data.

>From the subprocess documentation:

"""
Warning

Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to
avoid deadlocks due to any of the other OS pipe buffers filling up and
blocking the child process.
"""

--
http://mail.python.org/mailman/listinfo/python-list

Python python RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.