popen pipe limit

Discussion in 'Python' started by skunkwerk, Apr 8, 2008.

  1. skunkwerk

    skunkwerk Guest

    I'm getting errors when reading from/writing to pipes that are fairly
    large in size. To bypass this, I wanted to redirect output to a file
    in the subprocess.Popen function, but couldn't get it to work (even
    after setting Shell=True). I tried adding ">","temp.sql" after the
    password field but mysqldump gave me an error.

    the code:
    p1 = subprocess.Popen(["mysqldump","--all-databases","--user=user","--
    password=password"], shell=True)
    p2 = subprocess.Popen(["gzip","-9"], stdin=p1.stdout)
    output = p2.communicate()[0]
    file=open('test.sql.gz','w')
    file.write(str(output))
    file.close()

    the output:
    gzip: compressed data not written to a terminal. Use -f to force
    compression.
    For help, type: gzip -h
    mysqldump: Got errno 32 on write

    I'm using python rather than a shell script for this because I need to
    upload the resulting file to a server as soon as it's done.

    thanks
    skunkwerk, Apr 8, 2008
    #1
    1. Advertising

  2. En Mon, 07 Apr 2008 20:52:54 -0300, skunkwerk <>
    escribió:

    > I'm getting errors when reading from/writing to pipes that are fairly
    > large in size. To bypass this, I wanted to redirect output to a file
    > in the subprocess.Popen function, but couldn't get it to work (even
    > after setting Shell=True). I tried adding ">","temp.sql" after the
    > password field but mysqldump gave me an error.
    >
    > the code:
    > p1 = subprocess.Popen(["mysqldump","--all-databases","--user=user","--
    > password=password"], shell=True)
    > p2 = subprocess.Popen(["gzip","-9"], stdin=p1.stdout)
    > output = p2.communicate()[0]
    > file=open('test.sql.gz','w')
    > file.write(str(output))
    > file.close()


    You need a pipe to chain subprocesses:

    import subprocess
    p1 =
    subprocess.Popen(["mysqldump","--all-databases","--user=user","--password=password"],
    stdout=subprocess.PIPE)
    ofile = open("test.sql.gz", "wb")
    p2 = subprocess.Popen(["gzip","-9"], stdin=p1.stdout, stdout=ofile)
    p1.wait()
    p2.wait()
    ofile.close()

    If you don't want the final file on disk:

    p1 =
    subprocess.Popen(["mysqldump","--all-databases","--user=user","--password=password"],
    stdout=subprocess.PIPE)
    p2 = subprocess.Popen(["gzip","-9"], stdin=p1.stdout,
    stdout=subprocess.PIPE)
    while True:
    chunk = p2.stdout.read(4192)
    if not chunk: break
    # do something with read chunk

    p1.wait()
    p2.wait()

    --
    Gabriel Genellina
    Gabriel Genellina, Apr 8, 2008
    #2
    1. Advertising

  3. skunkwerk

    skunkwerk Guest

    On Apr 7, 6:17 pm, "Gabriel Genellina" <> wrote:
    > En Mon, 07 Apr 2008 20:52:54 -0300,skunkwerk<>  
    > escribió:
    >
    > > I'm getting errors when reading from/writing to pipes that are fairly
    > > large in size.  To bypass this, I wanted to redirect output to a file
    > > in the subprocess.Popen function, but couldn't get it to work (even
    > > after setting Shell=True).  I tried adding ">","temp.sql" after the
    > > password field but mysqldump gave me an error.

    >
    > > the code:
    > > p1 = subprocess.Popen(["mysqldump","--all-databases","--user=user","--
    > > password=password"], shell=True)
    > > p2 = subprocess.Popen(["gzip","-9"], stdin=p1.stdout)
    > > output = p2.communicate()[0]
    > > file=open('test.sql.gz','w')
    > > file.write(str(output))
    > > file.close()

    >
    > You need a pipe to chain subprocesses:
    >
    > import subprocess
    > p1 =  
    > subprocess.Popen(["mysqldump","--all-databases","--user=user","--password=password"],  
    > stdout=subprocess.PIPE)
    > ofile = open("test.sql.gz", "wb")
    > p2 = subprocess.Popen(["gzip","-9"], stdin=p1.stdout, stdout=ofile)
    > p1.wait()
    > p2.wait()
    > ofile.close()
    >
    > If you don't want the final file on disk:
    >
    > p1 =  
    > subprocess.Popen(["mysqldump","--all-databases","--user=user","--password=password"],  
    > stdout=subprocess.PIPE)
    > p2 = subprocess.Popen(["gzip","-9"], stdin=p1.stdout,  
    > stdout=subprocess.PIPE)
    > while True:
    >    chunk = p2.stdout.read(4192)
    >    if not chunk: break
    >    # do something with read chunk
    >
    > p1.wait()
    > p2.wait()
    >
    > --
    > Gabriel Genellina


    thanks Gabriel - tried the first one and it worked great!
    skunkwerk, Apr 10, 2008
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. lee, wonsun
    Replies:
    1
    Views:
    477
    Jack Klein
    Nov 2, 2004
  2. Tom Brown
    Replies:
    0
    Views:
    453
    Tom Brown
    Sep 22, 2005
  3. schickb

    Popen pipe hang

    schickb, May 13, 2008, in forum: Python
    Replies:
    0
    Views:
    288
    schickb
    May 13, 2008
  4. Mathieu Prevot
    Replies:
    3
    Views:
    357
    Mathieu Prevot
    May 24, 2008
  5. Replies:
    1
    Views:
    212
    Ben Morrow
    Jun 2, 2004
Loading...

Share This Page