Re: Help replacing os.system call with subprocess call

Discussion in 'Python' started by David Pratt, Apr 7, 2008.

  1. David Pratt

    David Pratt Guest

    Hi David and Matt. I appreciate your help which has got me moving
    forward again so many thanks for your reply. I have been using
    subprocess.Popen a fair bit but this was the first time I had to use
    subprocess to capture large file output. The trouble I was having was
    with the process would just hang. Chunking was the solution. I guess I
    assumed this would be taken care of in the internals.

    Overall, I wish subprocess had some better documentation since it is
    definitely not a drop in replacement for os.system. In other
    circumstances I am using for simple calls which works

    The speed of this solution is slower than os.system. Would a queue of
    some kind be needed to speed this up? Has anyone implemented something
    like this? Many thanks.


    Matt Nordhoff wrote:
    > David Pratt wrote:
    >> Hi. I am trying to replace a system call with a subprocess call. I have
    >> tried subprocess.Popen and with but have not been
    >> successful. The command line would be:
    >> svnadmin dump /my/repository > svndump.db
    >> This is what I am using currently:
    >> os.system('svnadmin dump %s > %s' % (svn_dir,
    >> os.path.join(backup_dir, 'svndump.db')))
    >> Many thanks.

    > Try this:
    > import os.path
    > import subprocess
    > p = subprocess.Popen(
    > ['svnadmin', 'dump', svndir],
    > stdout=subprocess.PIPE,
    > )
    > fh = open(os.path.join(backup_dir, 'svndump.db'), 'wb')
    > while True:
    > chunk =**20) # 1 MB
    > if not chunk:
    > break
    > fh.write(chunk)
    > fh.close()
    > It reads svnadmin's stdout in 1 MB chunks, in case it's large enough
    > that reading the whole thing into RAM at once would be a bad idea.
    > No error handling. For one, you might want to add a try...finally to
    > ensure that fh will get closed. (Or if you have Python 2.5, use a with
    > statement! :) ) Also, Popen will raise an OSError if svnadmin can't be
    > found or something. And this isn't even considering svnadmin erroring out...
    > svnadmin's stderr will go to your stderr.
    > I didn't test it, but I'm pretty sure it will work. (I spotted a syntax
    > error while writing that though.) I don't have much experience with
    > Popen's stdio objects, so it's possible you'd need to do something like
    > call p.wait() to wait for it to exit before being able to read its stdout.
    > It could be slower than the os.system version, since now Python is doing
    > all of the I/O, instead of your shell, but I doubt that'll be a big problem.
    > (Also, insert suggestion about using a good VCS. ;-) )
    David Pratt, Apr 7, 2008
    1. Advertisements

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Tom Brown
    Tom Brown
    Sep 22, 2005
  2. David Pratt
    David Pratt
    Apr 7, 2008
  3. hiral
    Jean-Michel Pichavant
    May 5, 2010
  4. goodman
    Lawrence D'Oliveiro
    Nov 5, 2010
  5. Rob Meade

    Replacing - and not Replacing...

    Rob Meade, Apr 5, 2005, in forum: ASP General
    Chris Hohmann
    Apr 11, 2005

Share This Page