subprocess.popen function with quotes

Discussion in 'Python' started by skunkwerk, Mar 26, 2008.

  1. skunkwerk

    skunkwerk Guest

    Hi,
    i'm trying to call subprocess.popen on the 'rename' function in
    linux. When I run the command from the shell, like so:

    rename -vn 's/\.htm$/\.html/' *.htm

    it works fine... however when I try to do it in python like so:
    p = subprocess.Popen(["rename","-vn","'s/\.htm$/
    \.html/'","*.htm"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)

    print p.communicate()[0]

    nothing gets printed out (even for p.communicate()[1])

    I think the problem is the quoted string the rename command wants -
    when I put it in triple quotes like """s/\.htm$/\.html/""" I get some
    output, but not the correct output. I've also tried escaping the
    single quotes with \' and putting it in regular double quotes but that
    didn't work either.

    i'd appreciate any help
     
    skunkwerk, Mar 26, 2008
    #1
    1. Advertising

  2. skunkwerk

    skunkwerk Guest

    also, i've tried the Shell=True parameter for Popen, but that didn't
    seem to make a difference

    On Mar 25, 8:31 pm, skunkwerk <> wrote:
    > Hi,
    > i'm trying to call subprocess.popen on the 'rename' function in
    > linux. When I run the command from the shell, like so:
    >
    > rename -vn 's/\.htm$/\.html/' *.htm
    >
    > it works fine... however when I try to do it in python like so:
    > p = subprocess.Popen(["rename","-vn","'s/\.htm$/
    > \.html/'","*.htm"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    >
    > print p.communicate()[0]
    >
    > nothing gets printed out (even for p.communicate()[1])
    >
    > I think the problem is the quoted string the rename command wants -
    > when I put it in triple quotes like """s/\.htm$/\.html/""" I get some
    > output, but not the correct output. I've also tried escaping the
    > single quotes with \' and putting it in regular double quotes but that
    > didn't work either.
    >
    > i'd appreciate any help
     
    skunkwerk, Mar 26, 2008
    #2
    1. Advertising

  3. En Wed, 26 Mar 2008 00:39:05 -0300, skunkwerk <>
    escribió:

    >> i'm trying to call subprocess.popen on the 'rename' function in
    >> linux. When I run the command from the shell, like so:
    >>
    >> rename -vn 's/\.htm$/\.html/' *.htm
    >>
    >> it works fine... however when I try to do it in python like so:
    >> p = subprocess.Popen(["rename","-vn","'s/\.htm$/
    >> \.html/'","*.htm"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    >>
    >> print p.communicate()[0]
    >>
    >> nothing gets printed out (even for p.communicate()[1])


    I'd try with:

    p = subprocess.Popen(["rename", "-vn", r"'s/\.htm$/\.html/'", "*.htm"],
    stdout=subprocess.PIPE, stderr=subprocess.PIPE,
    shell=True)

    (note that I added shell=True and I'm using a raw string to specify the
    reg.expr.)

    --
    Gabriel Genellina
     
    Gabriel Genellina, Mar 26, 2008
    #3
  4. skunkwerk

    skunkwerk Guest

    On Mar 25, 9:25 pm, "Gabriel Genellina" <>
    wrote:
    > En Wed, 26 Mar 2008 00:39:05 -0300, skunkwerk <>  
    > escribió:
    >
    > >>    i'm trying to call subprocess.popen on the 'rename' function in
    > >> linux.  When I run the command from the shell, like so:

    >
    > >> rename -vn 's/\.htm$/\.html/' *.htm

    >
    > >> it works fine... however when I try to do it in python like so:
    > >> p = subprocess.Popen(["rename","-vn","'s/\.htm$/
    > >> \.html/'","*.htm"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)

    >
    > >> print p.communicate()[0]

    >
    > >> nothing gets printed out (even for p.communicate()[1])

    >
    > I'd try with:
    >
    > p = subprocess.Popen(["rename", "-vn", r"'s/\.htm$/\.html/'", "*.htm"],
    >        stdout=subprocess.PIPE, stderr=subprocess.PIPE,
    >        shell=True)
    >
    > (note that I added shell=True and I'm using a raw string to specify the  
    > reg.expr.)
    >
    > --
    > Gabriel Genellina


    Thanks Gabriel,
    I tried the new command and one with the raw string and single
    quotes, but it is still giving me the same results (no output). any
    other suggestions?

    cheers
     
    skunkwerk, Mar 26, 2008
    #4
  5. skunkwerk

    Kurt Smith Guest

    On Wed, Mar 26, 2008 at 12:15 AM, skunkwerk <> wrote:
    > On Mar 25, 9:25 pm, "Gabriel Genellina" <>
    > wrote:
    > > En Wed, 26 Mar 2008 00:39:05 -0300, skunkwerk <>
    > > escribió:

    >
    > >
    > > >> i'm trying to call subprocess.popen on the 'rename' function in
    > > >> linux. When I run the command from the shell, like so:

    > >
    > > >> rename -vn 's/\.htm$/\.html/' *.htm

    > >
    > > >> it works fine... however when I try to do it in python like so:
    > > >> p = subprocess.Popen(["rename","-vn","'s/\.htm$/
    > > >> \.html/'","*.htm"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)

    > >
    > > >> print p.communicate()[0]

    > >
    > > >> nothing gets printed out (even for p.communicate()[1])

    > >

    >
    > > I'd try with:
    > >
    > > p = subprocess.Popen(["rename", "-vn", r"'s/\.htm$/\.html/'", "*.htm"],
    > > stdout=subprocess.PIPE, stderr=subprocess.PIPE,
    > > shell=True)
    > >
    > > (note that I added shell=True and I'm using a raw string to specify the
    > > reg.expr.)
    > >
    > > --
    > > Gabriel Genellina

    >
    > Thanks Gabriel,
    > I tried the new command and one with the raw string and single
    > quotes, but it is still giving me the same results (no output). any
    > other suggestions?


    I had similar problems passing quoted arguments to grep -- I don't
    have rename on my system so I can't do an exact translation.

    First version, passing argument to grep that would normally have to be
    quoted if typed in shell:

    In [1]: from subprocess import *

    In [2]: p1 = Popen(['ls'], stdout=PIPE)

    In [3]: p2 = Popen(['grep', '[0-9]\{7\}'], stdin=p1.stdout,
    stdout=PIPE) # note that the grep regex isn't double quoted...

    In [4]: output = p2.communicate()[0]

    In [5]: print output
    cur0046700.png
    cur0046700_1.png
    cur0046750.png
    dendat0046700.png
    dendat0046700_1.png
    dendat0046750.png

    And we see that everything is hunky dory.

    Now, trying to pass grep a quoted argument:

    In [10]: p1 = Popen(['ls'], stdout=PIPE)

    In [11]: p2 = Popen(['grep', '"[0-9]\{7\}"'], stdin=p1.stdout, stdout=PIPE)

    In [12]: output = p2.communicate()[0]

    In [13]: print output


    In [14]:

    And we get nothing. N.B. that's a single-quote double-quote string
    argument to grep in the second example, not a triple single quote.

    Incidentally, a triple quoted string will work as well.

    Moral from this example: when passing arguments that would normally
    be quoted to be safe from the shell's expansion, etc, don't. Just
    pass it using Popen(['cmd', 'arg-that-would-normally-be-quoted']) and
    it will be passed directly to the function without the shell's
    intervention.

    Since the argument is passed directly to the command (rename in your
    case, grep in this one) quoting to preserve special characters isn't
    needed, and the quotes will be passed as part of the argument, giving
    the null results you got above.

    Kurt
     
    Kurt Smith, Mar 26, 2008
    #5
  6. En Wed, 26 Mar 2008 02:15:28 -0300, skunkwerk <>
    escribió:

    > On Mar 25, 9:25 pm, "Gabriel Genellina" <>
    > wrote:
    >> En Wed, 26 Mar 2008 00:39:05 -0300, skunkwerk <>  
    >> escribió:
    >>
    >> >>    i'm trying to call subprocess.popen on the 'rename' function in
    >> >> linux.  When I run the command from the shell, like so:

    >>
    >> >> rename -vn 's/\.htm$/\.html/' *.htm

    >>
    >> >> it works fine... however when I try to do it in python like so:
    >> >> p = subprocess.Popen(["rename","-vn","'s/\.htm$/
    >> >> \.html/'","*.htm"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)

    >>
    >> >> print p.communicate()[0]

    >>
    >> >> nothing gets printed out (even for p.communicate()[1])

    >>
    >> I'd try with:
    >>
    >> p = subprocess.Popen(["rename", "-vn", r"'s/\.htm$/\.html/'", "*.htm"],
    >>        stdout=subprocess.PIPE, stderr=subprocess.PIPE,
    >>        shell=True)
    >>
    >> (note that I added shell=True and I'm using a raw string to specify the
    >> reg.expr.)

    >
    > Thanks Gabriel,
    > I tried the new command and one with the raw string and single
    > quotes, but it is still giving me the same results (no output). any
    > other suggestions?


    My next try would be without the single quotes...

    --
    Gabriel Genellina
     
    Gabriel Genellina, Mar 26, 2008
    #6
  7. skunkwerk

    skunkwerk Guest

    On Mar 25, 11:04 pm, "Gabriel Genellina" <>
    wrote:
    > En Wed, 26 Mar 2008 02:15:28 -0300, skunkwerk <>  
    > escribió:
    >
    >
    >
    > > On Mar 25, 9:25 pm, "Gabriel Genellina" <>
    > > wrote:
    > >> En Wed, 26 Mar 2008 00:39:05 -0300, skunkwerk <>  
    > >> escribió:

    >
    > >> >>    i'm trying to call subprocess.popen on the 'rename' function in
    > >> >> linux.  When I run the command from the shell, like so:

    >
    > >> >> rename -vn 's/\.htm$/\.html/' *.htm

    >
    > >> >> it works fine... however when I try to do it in python like so:
    > >> >> p = subprocess.Popen(["rename","-vn","'s/\.htm$/
    > >> >> \.html/'","*.htm"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)

    >
    > >> >> print p.communicate()[0]

    >
    > >> >> nothing gets printed out (even for p.communicate()[1])

    >
    > >> I'd try with:

    >
    > >> p = subprocess.Popen(["rename", "-vn", r"'s/\.htm$/\.html/'", "*.htm"],
    > >>        stdout=subprocess.PIPE, stderr=subprocess.PIPE,
    > >>        shell=True)

    >
    > >> (note that I added shell=True and I'm using a raw string to specify the  
    > >> reg.expr.)

    >
    > > Thanks Gabriel,
    > >    I tried the new command and one with the raw string and single
    > > quotes, but it is still giving me the same results (no output).  any
    > > other suggestions?

    >
    > My next try would be without the single quotes...
    >
    > --
    > Gabriel Genellina


    thanks for the input guys,
    I've tried the suggestions but can't get it to work. I have a file
    named test.htm in my directory, and when I run the following command:

    rename -vn 's/(.*)\.htm$/model.html/' *.htm

    from the shell in that directory I get the following output:
    test.htm renamed as model.html

    now my python script is called test.py, is located in the same
    directory, and is called from the shell with 'python test.py'
    the contents of test.py:
    import subprocess

    p = subprocess.Popen(['rename','-vn','s/(.*)\.htm$/
    model.html/','*.htm'],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    print p.communicate()[0]

    i change to print p.communicate()[1] in case the output is blank the
    first time

    this is the output:
    *.htm renamed as model.html

    when I add shell=True to the subprocess command, I get the following
    output:
    Usage: rename [-v] [-n] [-f] perlexpr [filenames]

    am i doing something wrong?
     
    skunkwerk, Mar 26, 2008
    #7
  8. skunkwerk

    skunkwerk Guest

    On Mar 26, 6:44 am, skunkwerk <> wrote:
    > On Mar 25, 11:04 pm, "Gabriel Genellina" <>
    > wrote:
    >
    >
    >
    > > En Wed, 26 Mar 2008 02:15:28 -0300, skunkwerk <>  
    > > escribió:

    >
    > > > On Mar 25, 9:25 pm, "Gabriel Genellina" <>
    > > > wrote:
    > > >> En Wed, 26 Mar 2008 00:39:05 -0300, skunkwerk <>  
    > > >> escribió:

    >
    > > >> >>    i'm trying to call subprocess.popen on the 'rename' function in
    > > >> >> linux.  When I run the command from the shell, like so:

    >
    > > >> >> rename -vn 's/\.htm$/\.html/' *.htm

    >
    > > >> >> it works fine... however when I try to do it in python like so:
    > > >> >> p = subprocess.Popen(["rename","-vn","'s/\.htm$/
    > > >> >> \.html/'","*.htm"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)

    >
    > > >> >> print p.communicate()[0]

    >
    > > >> >> nothing gets printed out (even for p.communicate()[1])

    >
    > > >> I'd try with:

    >
    > > >> p = subprocess.Popen(["rename", "-vn", r"'s/\.htm$/\.html/'", "*.htm"],
    > > >>        stdout=subprocess.PIPE, stderr=subprocess.PIPE,
    > > >>        shell=True)

    >
    > > >> (note that I added shell=True and I'm using a raw string to specify the  
    > > >> reg.expr.)

    >
    > > > Thanks Gabriel,
    > > >    I tried the new command and one with the raw string and single
    > > > quotes, but it is still giving me the same results (no output).  any
    > > > other suggestions?

    >
    > > My next try would be without the single quotes...

    >
    > > --
    > > Gabriel Genellina

    >
    > thanks for the input guys,
    >   I've tried the suggestions but can't get it to work.  I have a file
    > named test.htm in my directory, and when I run the following command:
    >
    > rename -vn 's/(.*)\.htm$/model.html/' *.htm
    >
    > from the shell in that directory I get the following output:
    > test.htm renamed as model.html
    >
    > now my python script is called test.py, is located in the same
    > directory, and is called from the shell with 'python test.py'
    > the contents of test.py:
    > import subprocess
    >
    > p = subprocess.Popen(['rename','-vn','s/(.*)\.htm$/
    > model.html/','*.htm'],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    > print p.communicate()[0]
    >
    > i change to print p.communicate()[1] in case the output is blank the
    > first time
    >
    > this is the output:
    > *.htm renamed as model.html
    >
    > when I add shell=True to the subprocess command, I get the following
    > output:
    > Usage: rename [-v] [-n] [-f] perlexpr [filenames]
    >
    > am i doing something wrong?


    in addition, when I use Popen without any quotes, or without quotes
    for the regular expression, I get an exception.

    I'm running ubuntu linux 7.10 with python 2.5.1

    thanks
     
    skunkwerk, Mar 26, 2008
    #8
  9. skunkwerk wrote:

    > p = subprocess.Popen(['rename','-vn','s/(.*)\.htm$/
    > model.html/','*.htm'],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    > print p.communicate()[0]
    >
    > i change to print p.communicate()[1] in case the output is blank the
    > first time
    >
    > this is the output:
    > *.htm renamed as model.html


    Without shell=True, your glob characters will not be expanded. Hence, the
    command looks for a file actually named "*.htm"

    > when I add shell=True to the subprocess command, I get the following
    > output:
    > Usage: rename [-v] [-n] [-f] perlexpr [filenames]


    Here the use of the shell may be confounding the arguments passed. Your
    command will probably work better if you avoid using shell=True. However,
    you will need to perform your own globbing:

    # Untested (no perl-rename here):

    command = ['rename','-vn', 's/(.*)\.htm$/model.html/']
    files = glob.glob('*.htm')
    command.extend(files)
    p = subprocess.Popen(
    command,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    )


    Jeffrey
     
    Jeffrey Froman, Mar 26, 2008
    #9
  10. skunkwerk

    skunkwerk Guest

    On Mar 26, 8:05 am, Jeffrey Froman <> wrote:
    > skunkwerk wrote:
    > > p = subprocess.Popen(['rename','-vn','s/(.*)\.htm$/
    > > model.html/','*.htm'],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    > > print p.communicate()[0]

    >
    > > i change to print p.communicate()[1] in case the output is blank the
    > > first time

    >
    > > this is the output:
    > > *.htm renamed as model.html

    >
    > Without shell=True, your glob characters will not be expanded. Hence, the
    > command looks for a file actually named "*.htm"
    >
    > > when I add shell=True to the subprocess command, I get the following
    > > output:
    > > Usage: rename [-v] [-n] [-f] perlexpr [filenames]

    >
    > Here the use of the shell may be confounding the arguments passed. Your
    > command will probably work better if you avoid using shell=True. However,
    > you will need to perform your own globbing:
    >
    > # Untested (no perl-rename here):
    >
    > command = ['rename','-vn', 's/(.*)\.htm$/model.html/']
    > files = glob.glob('*.htm')
    > command.extend(files)
    > p = subprocess.Popen(
    >     command,
    >     stdout=subprocess.PIPE,
    >     stderr=subprocess.PIPE,
    >     )
    >
    > Jeffrey


    thanks Jeffrey, that worked like a charm!
     
    skunkwerk, Mar 27, 2008
    #10
  11. skunkwerk

    skunkwerk Guest

    On Mar 26, 10:33 pm, skunkwerk <> wrote:
    > On Mar 26, 8:05 am, Jeffrey Froman <> wrote:
    >
    >
    >
    > >skunkwerkwrote:
    > > > p = subprocess.Popen(['rename','-vn','s/(.*)\.htm$/
    > > > model.html/','*.htm'],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    > > > print p.communicate()[0]

    >
    > > > i change to print p.communicate()[1] in case the output is blank the
    > > > first time

    >
    > > > this is the output:
    > > > *.htm renamed as model.html

    >
    > > Without shell=True, your glob characters will not be expanded. Hence, the
    > > command looks for a file actually named "*.htm"

    >
    > > > when I add shell=True to the subprocess command, I get the following
    > > > output:
    > > > Usage: rename [-v] [-n] [-f] perlexpr [filenames]

    >
    > > Here the use of the shell may be confounding the arguments passed. Your
    > > command will probably work better if you avoid using shell=True. However,
    > > you will need to perform your own globbing:

    >
    > > # Untested (no perl-rename here):

    >
    > > command = ['rename','-vn', 's/(.*)\.htm$/model.html/']
    > > files = glob.glob('*.htm')
    > > command.extend(files)
    > > p = subprocess.Popen(
    > >     command,
    > >     stdout=subprocess.PIPE,
    > >     stderr=subprocess.PIPE,
    > >     )

    >
    > > Jeffrey

    >
    > thanks Jeffrey, that worked like a charm!


    I'm trying to detect when the subprocess has terminated using the
    wait() function - but when there is an error with the call to rename
    (ie the file doesn't exist) rename (when run from the command line
    just terminates and displays the error). In the code above, though,
    my call to p.wait() just hangs when rename should throw an error...
    I've tried adding shell=True but that stops the rename from working.
    any ideas?

    thanks
     
    skunkwerk, Apr 13, 2008
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Chris White

    Quotes/Double Quotes in Image Control

    Chris White, Sep 22, 2004, in forum: ASP .Net
    Replies:
    1
    Views:
    4,905
    Hermit Dave
    Sep 22, 2004
  2. Chris
    Replies:
    1
    Views:
    13,744
    Oisin
    Mar 24, 2006
  3. Lawrence Tierney

    Multiline quotes - escaping quotes - et al

    Lawrence Tierney, Dec 24, 2003, in forum: Java
    Replies:
    3
    Views:
    4,541
    Andrew Thompson
    Dec 24, 2003
  4. banu
    Replies:
    6
    Views:
    1,032
    Sean DiZazzo
    Oct 29, 2009
  5. jOhn
    Replies:
    1
    Views:
    266
    Phlip
    Jan 29, 2008
Loading...

Share This Page