output to console and to multiple files

Discussion in 'Python' started by nathan.shair@gmail.com, Feb 14, 2007.

  1. Guest

    Hello,

    I searched on Google and in this Google Group, but did not find any
    solution to my problem.

    I'm looking for a way to output stdout/stderr (from a subprocess or
    spawn) to screen and to at least two different files.

    eg.
    stdout/stderr -> screen
    stdout -> log.out
    stderr -> log.err

    and if possible
    stdout/stderr -> screen and log.txt

    3 files from stdout/stderr
    , Feb 14, 2007
    #1
    1. Advertising

  2. Matimus Guest

    I took a look around and I couldn't find anything either. I will be
    keeping an eye on this thread to see if someone posts a more standard
    solution. In the mean time though, I will offer up a potential
    solution. Duck typing is your friend. If you are only using the write
    method of your files, it can be pretty simple to implement a fake file
    object to do what you want.

    Code:
    import sys
    
    class TeeFile(object):
        def __init__(self,*files):
            self.files = files
        def write(self,txt):
            for fp in self.files:
                fp.write(txt)
    
    if __name__ == "__main__":
        outf = file("log.out","w")
        errf = file("log.err","w")
        allf = file("log.txt","w")
        sys.stdout = TeeFile(sys.__stdout__,outf,allf)
        sys.stderr = TeeFile(sys.__stderr__,errf,allf)
    
        print "hello world this is stdout"
        print >> sys.stderr , "hello world this is stderr"
    
    Matimus, Feb 15, 2007
    #2
    1. Advertising

  3. goodwolf Guest

    like this?

    class Writers (object):

    def __init__(self, *writers):
    self.writers = writers

    def write(self, string):
    for w in self.writers:
    w.write(string)

    def flush(self):
    for w in self.writers:
    w.flush():

    import sys

    logfile = open('log.txt', 'w')
    sys.stdout = Writers(aya.stdout, file('log.out', 'w'), logfile)
    sys.stderr = Writers(aya.stdout, file('log.err', 'w'), logfile)
    goodwolf, Feb 15, 2007
    #3
  4. En Wed, 14 Feb 2007 19:28:34 -0300,
    <> escribió:

    > I'm looking for a way to output stdout/stderr (from a subprocess or
    > spawn) to screen and to at least two different files.


    Look at the tee command. If you control the subprocess, and it's written
    in Python, using the Python recipes would be easier and perhaps you have
    more control.
    But if you can't modify the subprocess, you'll have to use tee.

    --
    Gabriel Genellina
    Gabriel Genellina, Feb 15, 2007
    #4
  5. Guest

    On Feb 14, 5:10 pm, "goodwolf" <> wrote:
    > like this?
    >
    > class Writers (object):
    >
    > def __init__(self, *writers):
    > self.writers = writers
    >
    > def write(self, string):
    > for w in self.writers:
    > w.write(string)
    >
    > def flush(self):
    > for w in self.writers:
    > w.flush():
    >
    > import sys
    >
    > logfile = open('log.txt', 'w')
    > sys.stdout = Writers(aya.stdout, file('log.out', 'w'), logfile)
    > sys.stderr = Writers(aya.stdout, file('log.err', 'w'), logfile)



    i've tried simliar methods to this and to what Matimus wrote. I know
    it works great when using print statements.
    However, I'm looking to find something that will work with the output
    from a subprocess, such as from spawn, os.system, os.popen, etc.
    , Feb 15, 2007
    #5
  6. Guest

    On Feb 14, 6:52 pm, "Gabriel Genellina" <> wrote:
    > En Wed, 14 Feb 2007 19:28:34 -0300,
    > <> escribió:
    >
    > > I'm looking for a way to output stdout/stderr (from a subprocess or
    > > spawn) to screen and to at least two different files.

    >
    > Look at the tee command. If you control the subprocess, and it's written
    > in Python, using the Python recipes would be easier and perhaps you have
    > more control.
    > But if you can't modify the subprocess, you'll have to use tee.
    >
    > --
    > Gabriel Genellina


    Tee, the unix function? Or is there a tee that is python?
    , Feb 15, 2007
    #6
  7. Matimus Guest

    On Feb 15, 7:53 am, "" <>
    wrote:
    > On Feb 14, 5:10 pm, "goodwolf" <> wrote:
    >
    >
    >
    > > like this?

    >
    > > class Writers (object):

    >
    > > def __init__(self, *writers):
    > > self.writers = writers

    >
    > > def write(self, string):
    > > for w in self.writers:
    > > w.write(string)

    >
    > > def flush(self):
    > > for w in self.writers:
    > > w.flush():

    >
    > > import sys

    >
    > > logfile = open('log.txt', 'w')
    > > sys.stdout = Writers(aya.stdout, file('log.out', 'w'), logfile)
    > > sys.stderr = Writers(aya.stdout, file('log.err', 'w'), logfile)

    >
    > i've tried simliar methods to this and to what Matimus wrote. I know
    > it works great when using print statements.
    > However, I'm looking to find something that will work with the output
    > from a subprocess, such as from spawn, os.system, os.popen, etc.


    I think you should be able to use my or goodwolf's solution with the
    subprocess module. Something like this (untested):

    Code:
    class TeeFile(object):
        def __init__(self,*files):
            self.files = files
        def write(self,txt):
            for fp in self.files:
                fp.write(txt)
    
    if __name__ == "__main__":
        import sys
        from subprocess import Popen
    
        command = "whatever you want to run"
        outf = file("log.out","w")
        errf = file("log.err","w")
        allf = file("log.txt","w")
        Popen(
            command,
            stdout = TeeFile(sys.__stdout__,outf,allf),
            stderr = TeeFile(sys.__stderr__,errf,allf)
        )
    
    Matimus, Feb 15, 2007
    #7
  8. Matimus Guest

    On Feb 15, 8:51 am, "Matimus" <> wrote:
    > On Feb 15, 7:53 am, "" <>
    > wrote:
    >
    >
    >
    > > On Feb 14, 5:10 pm, "goodwolf" <> wrote:

    >
    > > > like this?

    >
    > > > class Writers (object):

    >
    > > > def __init__(self, *writers):
    > > > self.writers = writers

    >
    > > > def write(self, string):
    > > > for w in self.writers:
    > > > w.write(string)

    >
    > > > def flush(self):
    > > > for w in self.writers:
    > > > w.flush():

    >
    > > > import sys

    >
    > > > logfile = open('log.txt', 'w')
    > > > sys.stdout = Writers(aya.stdout, file('log.out', 'w'), logfile)
    > > > sys.stderr = Writers(aya.stdout, file('log.err', 'w'), logfile)

    >
    > > i've tried simliar methods to this and to what Matimus wrote. I know
    > > it works great when using print statements.
    > > However, I'm looking to find something that will work with the output
    > > from a subprocess, such as from spawn, os.system, os.popen, etc.

    >
    > I think you should be able to use my or goodwolf's solution with the
    > subprocess module. Something like this (untested):
    >
    >
    Code:
    > class TeeFile(object):
    >     def __init__(self,*files):
    >         self.files = files
    >     def write(self,txt):
    >         for fp in self.files:
    >             fp.write(txt)
    >
    > if __name__ == "__main__":
    >     import sys
    >     from subprocess import Popen
    >
    >     command = "whatever you want to run"
    >     outf = file("log.out","w")
    >     errf = file("log.err","w")
    >     allf = file("log.txt","w")
    >     Popen(
    >         command,
    >         stdout = TeeFile(sys.__stdout__,outf,allf),
    >         stderr = TeeFile(sys.__stderr__,errf,allf)
    >     )
    > 


    I tried this at lunch and it doesn't work. Some version of this method
    may work, but Popen tries to call the 'fileno' method of the TeeFile
    object (at least it did on my setup) and it isn't there. This is just
    a preemptive warning before someone comes back to let me know my code
    doesn't work.
    Matimus, Feb 15, 2007
    #8
  9. En Thu, 15 Feb 2007 19:35:10 -0300, Matimus <> escribió:

    >> I think you should be able to use my or goodwolf's solution with the
    >> subprocess module. Something like this (untested):
    >>
    >>
    Code:
    >> class TeeFile(object):
    >>     def __init__(self,*files):
    >>         self.files = files
    >>     def write(self,txt):
    >>         for fp in self.files:
    >>             fp.write(txt)
    >>[/color]
    >
    > I tried this at lunch and it doesn't work. Some version of this method
    > may work, but Popen tries to call the 'fileno' method of the TeeFile
    > object (at least it did on my setup) and it isn't there. This is just
    > a preemptive warning before someone comes back to let me know my code
    > doesn't work.[/color]
    
    I don't think any Python only solution could work. The pipe options  
    available for subprocess are those of the underlying OS, and the OS knows  
    nothing about Python file objects.
    
    -- 
    Gabriel Genellina
    Gabriel Genellina, Feb 16, 2007
    #9
  10. Guest

    On Feb 15, 5:48 pm, "Gabriel Genellina" <> wrote:
    > En Thu, 15 Feb 2007 19:35:10 -0300, Matimus <> escribió:
    >
    >
    >
    > >> I think you should be able to use my or goodwolf's solution with the
    > >> subprocess module. Something like this (untested):

    >
    > >>
    Code:
    > >> class TeeFile(object):
    > >>     def __init__(self,*files):
    > >>         self.files = files
    > >>     def write(self,txt):
    > >>         for fp in self.files:
    > >>             fp.write(txt)[/color][/color]
    >[color=green]
    > > I tried this at lunch and it doesn't work. Some version of this method
    > > may work, but Popen tries to call the 'fileno' method of the TeeFile
    > > object (at least it did on my setup) and it isn't there. This is just
    > > a preemptive warning before someone comes back to let me know my code
    > > doesn't work.[/color]
    >
    > I don't think any Python only solution could work. The pipe options
    > available for subprocess are those of the underlying OS, and the OS knows
    > nothing about Python file objects.
    >
    > --
    > Gabriel Genellina[/color]
    
    I've tried the subprocess method before without any luck.
    
    
    Thanks for all your suggestions. I guess it's time to rethink what I
    want to do.
    , Feb 16, 2007
    #10
  11. On Feb 14, 11:28 pm, "" <>
    wrote:
    > Hello,
    >
    > I searched on Google and in this Google Group, but did not find any
    > solution to my problem.
    >
    > I'm looking for a way to output stdout/stderr (from a subprocess or
    > spawn) to screen and to at least two different files.
    >
    > eg.
    > stdout/stderr -> screen
    > stdout -> log.out
    > stderr -> log.err
    >
    > and if possible
    > stdout/stderr -> screen and log.txt
    >
    > 3 files from stdout/stderr


    I'd derive a class from file, overwrite it's write() method to send a
    copy to the log, and then assign sys.stdout = newFile(sys.stdout).
    Same for stderr.
    Bart Ogryczak, Feb 16, 2007
    #11
  12. En Fri, 16 Feb 2007 14:04:33 -0300, Bart Ogryczak <>
    escribió:

    > On Feb 14, 11:28 pm, "" <>
    > wrote:


    >> I'm looking for a way to output stdout/stderr (from a subprocess or
    >> spawn) to screen and to at least two different files.

    >
    > I'd derive a class from file, overwrite it's write() method to send a
    > copy to the log, and then assign sys.stdout = newFile(sys.stdout).
    > Same for stderr.


    That's ok inside the same process, but the OP needs to use it "from a
    subprocess or spawn".
    You have to use something like tee, working with real file handles.

    --
    Gabriel Genellina
    Gabriel Genellina, Feb 16, 2007
    #12
  13. Guest

    On Feb 16, 3:28 pm, "Gabriel Genellina" <> wrote:

    >
    > That's ok inside the same process, but the OP needs to use it "from a
    > subprocess or spawn".
    > You have to use something like tee, working with real file handles.
    >


    I'm not particularly familiar with this, but it seems to me that if
    you're trying to catch stdout/stderr from a program you can call with
    (say) popen2, you could just read from the returned stdout/stderr
    pipe, and then write to a series of file handles (including
    sys.stdout).

    Or am I missing something? =)

    ~G
    , Feb 16, 2007
    #13
  14. Guest

    On Feb 16, 4:07 pm, wrote:
    > On Feb 16, 3:28 pm, "Gabriel Genellina" <> wrote:
    >
    >
    >
    > > That's ok inside the same process, but the OP needs to use it "from a
    > > subprocess or spawn".
    > > You have to use something like tee, working with real file handles.

    >
    > I'm not particularly familiar with this, but it seems to me that if
    > you're trying to catch stdout/stderr from a program you can call with
    > (say) popen2, you could just read from the returned stdout/stderr
    > pipe, and then write to a series of file handles (including
    > sys.stdout).
    >
    > Or am I missing something? =)
    >
    > ~G


    That works, but it isn't live streaming of stdout/stderr. Most of the
    time, if you stream both, one could lock the process, or have the
    stdout/stderr printed in the wrong order.
    , Feb 16, 2007
    #14
  15. Fuzzyman Guest

    On Feb 16, 11:37 pm, "" <>
    wrote:
    > On Feb 16, 4:07 pm, wrote:
    >
    >
    >
    > > On Feb 16, 3:28 pm, "Gabriel Genellina" <> wrote:

    >
    > > > That's ok inside the same process, but the OP needs to use it "from a
    > > > subprocess or spawn".
    > > > You have to use something like tee, working with real file handles.

    >
    > > I'm not particularly familiar with this, but it seems to me that if
    > > you're trying to catch stdout/stderr from a program you can call with
    > > (say) popen2, you could just read from the returned stdout/stderr
    > > pipe, and then write to a series of file handles (including
    > > sys.stdout).

    >
    > > Or am I missing something? =)

    >
    > > ~G

    >
    > That works, but it isn't live streaming of stdout/stderr. Most of the
    > time, if you stream both, one could lock the process, or have the
    > stdout/stderr printed in the wrong order.


    Everytime I've looked to do something like this (non-blocking read on
    the stdout of a subprocess) I've always come back to the conclusion
    that threads and queues are the only reasonable way (particularly on
    windows). There may be a better solution using select.

    Fuzzyman
    http://www.voidspace.org.uk/python/articles.shtml
    Fuzzyman, Feb 17, 2007
    #15
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ben Finney
    Replies:
    2
    Views:
    979
    Egor Bolonev
    Jun 30, 2003
  2. Replies:
    4
    Views:
    938
    M.E.Farmer
    Feb 13, 2005
  3. Yoav
    Replies:
    1
    Views:
    263
    Diez B. Roggisch
    Aug 22, 2005
  4. pratap
    Replies:
    3
    Views:
    444
    James Kanze
    Jul 19, 2007
  5. Replies:
    5
    Views:
    3,416
    James Kanze
    Mar 4, 2008
Loading...

Share This Page