How to read large amounts of output via popen

Discussion in 'Python' started by loial, Aug 6, 2010.

  1. loial

    loial Guest

    I need to read a large amount of data that is being returned in
    standard output by a shell script I am calling.

    (I think the script should really be writing to a file but I have no
    control over that)

    Currently I have the following code. It seeems to work, however I
    suspect this may not work with large amounts of standard output.

    What is the best way to read a large amount of data from standard
    output and write to a file?

    Here is my code.

    process=subprocess.Popen(['myscript', 'param1'],
    shell=False,stdout=subprocess.PIPE,stderr=subprocess.PIPE)

    cmdoutput=process.communicate()

    myfile = open('/home/john/myoutputfile','w')

    myfile.write(cmdoutput[0])

    myfile.close()
    loial, Aug 6, 2010
    #1
    1. Advertising

  2. En Fri, 06 Aug 2010 06:06:29 -0300, loial <> escribió:

    > I need to read a large amount of data that is being returned in
    > standard output by a shell script I am calling.
    >
    > (I think the script should really be writing to a file but I have no
    > control over that)
    >
    > Currently I have the following code. It seeems to work, however I
    > suspect this may not work with large amounts of standard output.
    >
    > What is the best way to read a large amount of data from standard
    > output and write to a file?
    >
    > Here is my code.
    >
    > process=subprocess.Popen(['myscript', 'param1'],
    > shell=False,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    >
    > cmdoutput=process.communicate()
    >
    > myfile = open('/home/john/myoutputfile','w')
    >
    > myfile.write(cmdoutput[0])
    >
    > myfile.close()



    If all you do with the process' output is to write it to the output file,
    you can avoid the intermediate step:


    myfile = open('/home/john/myoutputfile','w')
    myerror = open('/home/john/myerrorfile','w')
    process=subprocess.Popen(['myscript', 'param1'],
    shell=False,stdout=myfile,stderr=myerror)
    process.wait()

    (untested)

    --
    Gabriel Genellina
    Gabriel Genellina, Aug 6, 2010
    #2
    1. Advertising

  3. loial

    Nobody Guest

    On Fri, 06 Aug 2010 02:06:29 -0700, loial wrote:

    > I need to read a large amount of data that is being returned in
    > standard output by a shell script I am calling.
    >
    > (I think the script should really be writing to a file but I have no
    > control over that)


    If the script is writing to stdout, you get to decide whether its stdout
    is a pipe, file, tty, etc.

    > Currently I have the following code. It seeems to work, however I
    > suspect this may not work with large amounts of standard output.


    > process=subprocess.Popen(['myscript', 'param1'],
    > shell=False,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    >
    > cmdoutput=process.communicate()


    It's certainly not the best way to read large amounts of output.
    Unfortunately, better solutions get complicated when you need to read more
    than one of stdout and stderr, or if you also need to write to stdin.

    If you only need stdout, you can just read from process.stdout in a loop.
    You can leave stderr going to wherever the script's stderr goes (e.g. the
    terminal), or redirect it to a file.

    If you really do need both stdout and stderr, then you either need to
    enable non-blocking I/O, or use a separate thread for each stream, or
    redirect at least one of them to a file.

    FWIW, Popen.communicate() uses non-blocking I/O on Unix and separate
    threads on Windows (the standard library doesn't include a mechanism to
    enable non-blocking I/O on Windows).

    > What is the best way to read a large amount of data from standard
    > output and write to a file?


    For this case, the best way is to just redirect stdout to a file, rather
    than passing it through the script, i.e.:

    outfile = open('outputfile', 'w')
    process = subprocess.call(..., stdout = outfile)
    outfile.close()
    Nobody, Aug 6, 2010
    #3
  4. loial

    loial Guest

    Re: How to read large amounts of output via popen

    Ok, thats great. Thanks for the very elegant solution(s)




    On 6 Aug, 13:44, Nobody <> wrote:
    > On Fri, 06 Aug 2010 02:06:29 -0700, loial wrote:
    > > I need to read a large amount of data that is being returned in
    > > standard output by a shell script I am calling.

    >
    > > (I think the script should really be writing to a file but I have no
    > > control over that)

    >
    > If the script is writing to stdout, you get to decide whether its stdout
    > is a pipe, file, tty, etc.
    >
    > > Currently I have the following code. It seeems to work, however I
    > > suspect this may not work with large amounts of standard output.
    > > process=subprocess.Popen(['myscript', 'param1'],
    > > shell=False,stdout=subprocess.PIPE,stderr=subprocess.PIPE)

    >
    > > cmdoutput=process.communicate()

    >
    > It's certainly not the best way to read large amounts of output.
    > Unfortunately, better solutions get complicated when you need to read more
    > than one of stdout and stderr, or if you also need to write to stdin.
    >
    > If you only need stdout, you can just read from process.stdout in a loop.
    > You can leave stderr going to wherever the script's stderr goes (e.g. the
    > terminal), or redirect it to a file.
    >
    > If you really do need both stdout and stderr, then you either need to
    > enable non-blocking I/O, or use a separate thread for each stream, or
    > redirect at least one of them to a file.
    >
    > FWIW, Popen.communicate() uses non-blocking I/O on Unix and separate
    > threads on Windows (the standard library doesn't include a mechanism to
    > enable non-blocking I/O on Windows).
    >
    > > What is the best way to read a large amount of data from standard
    > > output and write to a file?

    >
    > For this case, the best way is to just redirect stdout to a file, rather
    > than passing it through the script, i.e.:
    >
    >         outfile = open('outputfile', 'w')
    >         process = subprocess.call(..., stdout = outfile)
    >         outfile.close()
    loial, Aug 6, 2010
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Brent
    Replies:
    3
    Views:
    496
    Raymond Lewallen
    Apr 7, 2004
  2. Andersen
    Replies:
    2
    Views:
    388
  3. Bint
    Replies:
    1
    Views:
    1,859
    Benjamin Niemann
    Mar 19, 2006
  4. David
    Replies:
    0
    Views:
    326
    David
    Jun 21, 2006
  5. flamesrock
    Replies:
    2
    Views:
    375
    flamesrock
    Mar 12, 2005
Loading...

Share This Page