Re: Help replacing os.system call with subprocess call

Discussion in 'Python' started by David Pratt, Apr 7, 2008.

  1. David Pratt

    David Pratt Guest

    Hi David and Matt. I appreciate your help which has got me moving
    forward again so many thanks for your reply. I have been using
    subprocess.Popen a fair bit but this was the first time I had to use
    subprocess to capture large file output. The trouble I was having was
    with the process would just hang. Chunking was the solution. I guess I
    assumed this would be taken care of in the internals.

    Overall, I wish subprocess had some better documentation since it is
    definitely not a drop in replacement for os.system. In other
    circumstances I am using subprocess.call() for simple calls which works
    fine.

    The speed of this solution is slower than os.system. Would a queue of
    some kind be needed to speed this up? Has anyone implemented something
    like this? Many thanks.

    Regards,
    David

    Matt Nordhoff wrote:
    > David Pratt wrote:
    >> Hi. I am trying to replace a system call with a subprocess call. I have
    >> tried subprocess.Popen and subprocess.call with but have not been
    >> successful. The command line would be:
    >>
    >> svnadmin dump /my/repository > svndump.db
    >>
    >> This is what I am using currently:
    >>
    >> os.system('svnadmin dump %s > %s' % (svn_dir,
    >> os.path.join(backup_dir, 'svndump.db')))
    >>
    >> Many thanks.

    >
    > Try this:
    >
    > import os.path
    > import subprocess
    >
    > p = subprocess.Popen(
    > ['svnadmin', 'dump', svndir],
    > stdout=subprocess.PIPE,
    > )
    >
    > fh = open(os.path.join(backup_dir, 'svndump.db'), 'wb')
    >
    > while True:
    > chunk = p.stdout.read(2**20) # 1 MB
    > if not chunk:
    > break
    > fh.write(chunk)
    >
    > fh.close()
    >
    > It reads svnadmin's stdout in 1 MB chunks, in case it's large enough
    > that reading the whole thing into RAM at once would be a bad idea.
    >
    > No error handling. For one, you might want to add a try...finally to
    > ensure that fh will get closed. (Or if you have Python 2.5, use a with
    > statement! :) ) Also, Popen will raise an OSError if svnadmin can't be
    > found or something. And this isn't even considering svnadmin erroring out...
    >
    > svnadmin's stderr will go to your stderr.
    >
    > I didn't test it, but I'm pretty sure it will work. (I spotted a syntax
    > error while writing that though.) I don't have much experience with
    > Popen's stdio objects, so it's possible you'd need to do something like
    > call p.wait() to wait for it to exit before being able to read its stdout.
    >
    > It could be slower than the os.system version, since now Python is doing
    > all of the I/O, instead of your shell, but I doubt that'll be a big problem.
    >
    > (Also, insert suggestion about using a good VCS. ;-) )
    David Pratt, Apr 7, 2008
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Tom Brown
    Replies:
    0
    Views:
    464
    Tom Brown
    Sep 22, 2005
  2. David Pratt
    Replies:
    0
    Views:
    259
    David Pratt
    Apr 7, 2008
  3. hiral
    Replies:
    2
    Views:
    595
    Jean-Michel Pichavant
    May 5, 2010
  4. goodman
    Replies:
    4
    Views:
    636
    Lawrence D'Oliveiro
    Nov 5, 2010
  5. Rob Meade

    Replacing - and not Replacing...

    Rob Meade, Apr 5, 2005, in forum: ASP General
    Replies:
    5
    Views:
    271
    Chris Hohmann
    Apr 11, 2005
Loading...

Share This Page