subprocess.popen function with quotes

[skunkwerk]

Hi,
i'm trying to call subprocess.popen on the 'rename' function in
linux. When I run the command from the shell, like so:

rename -vn 's/\.htm$/\.html/' *.htm

it works fine... however when I try to do it in python like so:
p = subprocess.Popen(["rename","-vn","'s/\.htm$/
\.html/'","*.htm"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)

print p.communicate()[0]

nothing gets printed out (even for p.communicate()[1])

I think the problem is the quoted string the rename command wants -
when I put it in triple quotes like """s/\.htm$/\.html/""" I get some
output, but not the correct output. I've also tried escaping the
single quotes with \' and putting it in regular double quotes but that
didn't work either.

i'd appreciate any help

 

[skunkwerk]

also, i've tried the Shell=True parameter for Popen, but that didn't
seem to make a difference

 

[Gabriel Genellina]

i'm trying to call subprocess.popen on the 'rename' function in
linux. When I run the command from the shell, like so:

rename -vn 's/\.htm$/\.html/' *.htm

it works fine... however when I try to do it in python like so:
p = subprocess.Popen(["rename","-vn","'s/\.htm$/
\.html/'","*.htm"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)

print p.communicate()[0]

nothing gets printed out (even for p.communicate()[1])

I'd try with:

p = subprocess.Popen(["rename", "-vn", r"'s/\.htm$/\.html/'", "*.htm"],
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
shell=True)

(note that I added shell=True and I'm using a raw string to specify the
reg.expr.)

 

[skunkwerk]

   i'm trying to call subprocess.popen on the 'rename' function in
linux.  When I run the command from the shell, like so:
rename -vn 's/\.htm$/\.html/' *.htm
it works fine... however when I try to do it in python like so:
p = subprocess.Popen(["rename","-vn","'s/\.htm$/
\.html/'","*.htm"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
print p.communicate()[0]
nothing gets printed out (even for p.communicate()[1])

I'd try with:

p = subprocess.Popen(["rename", "-vn", r"'s/\.htm$/\.html/'", "*.htm"],
       stdout=subprocess.PIPE, stderr=subprocess.PIPE,
       shell=True)

(note that I added shell=True and I'm using a raw string to specify the  
reg.expr.)

Thanks Gabriel,
I tried the new command and one with the raw string and single
quotes, but it is still giving me the same results (no output). any
other suggestions?

cheers

 

[Kurt Smith]

En Wed, 26 Mar 2008 00:39:05 -0300, skunkwerk <[email protected]>
escribió:

i'm trying to call subprocess.popen on the 'rename' function in
linux. When I run the command from the shell, like so:

rename -vn 's/\.htm$/\.html/' *.htm

it works fine... however when I try to do it in python like so:
p = subprocess.Popen(["rename","-vn","'s/\.htm$/
\.html/'","*.htm"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)

print p.communicate()[0]

nothing gets printed out (even for p.communicate()[1])

I'd try with:

p = subprocess.Popen(["rename", "-vn", r"'s/\.htm$/\.html/'", "*.htm"],
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
shell=True)

(note that I added shell=True and I'm using a raw string to specify the
reg.expr.)

Thanks Gabriel,
I tried the new command and one with the raw string and single
quotes, but it is still giving me the same results (no output). any
other suggestions?

I had similar problems passing quoted arguments to grep -- I don't
have rename on my system so I can't do an exact translation.

First version, passing argument to grep that would normally have to be
quoted if typed in shell:

In [1]: from subprocess import *

In [2]: p1 = Popen(['ls'], stdout=PIPE)

In [3]: p2 = Popen(['grep', '[0-9]\{7\}'], stdin=p1.stdout,
stdout=PIPE) # note that the grep regex isn't double quoted...

In [4]: output = p2.communicate()[0]

In [5]: print output
cur0046700.png
cur0046700_1.png
cur0046750.png
dendat0046700.png
dendat0046700_1.png
dendat0046750.png

And we see that everything is hunky dory.

Now, trying to pass grep a quoted argument:

In [10]: p1 = Popen(['ls'], stdout=PIPE)

In [11]: p2 = Popen(['grep', '"[0-9]\{7\}"'], stdin=p1.stdout, stdout=PIPE)

In [12]: output = p2.communicate()[0]

In [13]: print output


In [14]:

And we get nothing. N.B. that's a single-quote double-quote string
argument to grep in the second example, not a triple single quote.

Incidentally, a triple quoted string will work as well.

Moral from this example: when passing arguments that would normally
be quoted to be safe from the shell's expansion, etc, don't. Just
pass it using Popen(['cmd', 'arg-that-would-normally-be-quoted']) and
it will be passed directly to the function without the shell's
intervention.

Since the argument is passed directly to the command (rename in your
case, grep in this one) quoting to preserve special characters isn't
needed, and the quotes will be passed as part of the argument, giving
the null results you got above.

Kurt

 

[Gabriel Genellina]

   i'm trying to call subprocess.popen on the 'rename' function in
linux.  When I run the command from the shell, like so:

rename -vn 's/\.htm$/\.html/' *.htm

it works fine... however when I try to do it in python like so:
p = subprocess.Popen(["rename","-vn","'s/\.htm$/
\.html/'","*.htm"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)

print p.communicate()[0]

nothing gets printed out (even for p.communicate()[1])

I'd try with:

p = subprocess.Popen(["rename", "-vn", r"'s/\.htm$/\.html/'", "*.htm"],
       stdout=subprocess.PIPE, stderr=subprocess.PIPE,
       shell=True)

(note that I added shell=True and I'm using a raw string to specify the
reg.expr.)

Thanks Gabriel,
I tried the new command and one with the raw string and single
quotes, but it is still giving me the same results (no output). any
other suggestions?

My next try would be without the single quotes...

 

[skunkwerk]

En Wed, 26 Mar 2008 02:15:28 -0300, skunkwerk <[email protected]>  
escribió:

En Wed, 26 Mar 2008 00:39:05 -0300, skunkwerk <[email protected]>  
escribió:
   i'm trying to call subprocess.popen on the 'rename' function in
linux.  When I run the command from the shell, like so:
rename -vn 's/\.htm$/\.html/' *.htm
it works fine... however when I try to do it in python like so:
p = subprocess.Popen(["rename","-vn","'s/\.htm$/
\.html/'","*.htm"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
print p.communicate()[0]
nothing gets printed out (even for p.communicate()[1])
I'd try with:
p = subprocess.Popen(["rename", "-vn", r"'s/\.htm$/\.html/'", "*.htm"],
       stdout=subprocess.PIPE, stderr=subprocess.PIPE,
       shell=True)
(note that I added shell=True and I'm using a raw string to specify the  
reg.expr.)

Thanks Gabriel,
   I tried the new command and one with the raw string and single
quotes, but it is still giving me the same results (no output).  any
other suggestions?

My next try would be without the single quotes...

thanks for the input guys,
I've tried the suggestions but can't get it to work. I have a file
named test.htm in my directory, and when I run the following command:

rename -vn 's/(.*)\.htm$/model.html/' *.htm

from the shell in that directory I get the following output:
test.htm renamed as model.html

now my python script is called test.py, is located in the same
directory, and is called from the shell with 'python test.py'
the contents of test.py:
import subprocess

p = subprocess.Popen(['rename','-vn','s/(.*)\.htm$/
model.html/','*.htm'],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
print p.communicate()[0]

i change to print p.communicate()[1] in case the output is blank the
first time

this is the output:
*.htm renamed as model.html

when I add shell=True to the subprocess command, I get the following
output:
Usage: rename [-v] [-n] [-f] perlexpr [filenames]

am i doing something wrong?

 

[skunkwerk]

En Wed, 26 Mar 2008 02:15:28 -0300, skunkwerk <[email protected]>  
escribió:

On Mar 25, 9:25 pm, "Gabriel Genellina" <[email protected]>
wrote:
En Wed, 26 Mar 2008 00:39:05 -0300, skunkwerk <[email protected]>  
escribió:
   i'm trying to call subprocess.popen on the 'rename' function in
linux.  When I run the command from the shell, like so:
rename -vn 's/\.htm$/\.html/' *.htm
it works fine... however when I try to do it in python like so:
p = subprocess.Popen(["rename","-vn","'s/\.htm$/
\.html/'","*.htm"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
print p.communicate()[0]
nothing gets printed out (even for p.communicate()[1])
I'd try with:
p = subprocess.Popen(["rename", "-vn", r"'s/\.htm$/\.html/'", "*.htm"],
       stdout=subprocess.PIPE, stderr=subprocess.PIPE,
       shell=True)
(note that I added shell=True and I'm using a raw string to specify the  
reg.expr.)
Thanks Gabriel,
   I tried the new command and one with the raw string and single
quotes, but it is still giving me the same results (no output).  any
other suggestions?

My next try would be without the single quotes...

thanks for the input guys,
  I've tried the suggestions but can't get it to work.  I have a file
named test.htm in my directory, and when I run the following command:

rename -vn 's/(.*)\.htm$/model.html/' *.htm

from the shell in that directory I get the following output:
test.htm renamed as model.html

now my python script is called test.py, is located in the same
directory, and is called from the shell with 'python test.py'
the contents of test.py:
import subprocess

p = subprocess.Popen(['rename','-vn','s/(.*)\.htm$/
model.html/','*.htm'],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
print p.communicate()[0]

i change to print p.communicate()[1] in case the output is blank the
first time

this is the output:
*.htm renamed as model.html

when I add shell=True to the subprocess command, I get the following
output:
Usage: rename [-v] [-n] [-f] perlexpr [filenames]

am i doing something wrong?

in addition, when I use Popen without any quotes, or without quotes
for the regular expression, I get an exception.

I'm running ubuntu linux 7.10 with python 2.5.1

thanks

 

[Jeffrey Froman]

p = subprocess.Popen(['rename','-vn','s/(.*)\.htm$/
model.html/','*.htm'],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
print p.communicate()[0]

i change to print p.communicate()[1] in case the output is blank the
first time

this is the output:
*.htm renamed as model.html

Without shell=True, your glob characters will not be expanded. Hence, the
command looks for a file actually named "*.htm"

when I add shell=True to the subprocess command, I get the following
output:
Usage: rename [-v] [-n] [-f] perlexpr [filenames]

Here the use of the shell may be confounding the arguments passed. Your
command will probably work better if you avoid using shell=True. However,
you will need to perform your own globbing:

# Untested (no perl-rename here):

command = ['rename','-vn', 's/(.*)\.htm$/model.html/']
files = glob.glob('*.htm')
command.extend(files)
p = subprocess.Popen(
command,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)


Jeffrey

 

[skunkwerk]

p = subprocess.Popen(['rename','-vn','s/(.*)\.htm$/
model.html/','*.htm'],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
print p.communicate()[0]

i change to print p.communicate()[1] in case the output is blank the
first time

this is the output:
*.htm renamed as model.html

Without shell=True, your glob characters will not be expanded. Hence, the
command looks for a file actually named "*.htm"

when I add shell=True to the subprocess command, I get the following
output:
Usage: rename [-v] [-n] [-f] perlexpr [filenames]

Here the use of the shell may be confounding the arguments passed. Your
command will probably work better if you avoid using shell=True. However,
you will need to perform your own globbing:

# Untested (no perl-rename here):

command = ['rename','-vn', 's/(.*)\.htm$/model.html/']
files = glob.glob('*.htm')
command.extend(files)
p = subprocess.Popen(
    command,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    )

Jeffrey

thanks Jeffrey, that worked like a charm!

 

[skunkwerk]

skunkwerkwrote:

p = subprocess.Popen(['rename','-vn','s/(.*)\.htm$/
model.html/','*.htm'],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
print p.communicate()[0]
i change to print p.communicate()[1] in case the output is blank the
first time
this is the output:
*.htm renamed as model.html

Without shell=True, your glob characters will not be expanded. Hence, the
command looks for a file actually named "*.htm"

when I add shell=True to the subprocess command, I get the following
output:
Usage: rename [-v] [-n] [-f] perlexpr [filenames]

Here the use of the shell may be confounding the arguments passed. Your
command will probably work better if you avoid using shell=True. However,
you will need to perform your own globbing:

# Untested (no perl-rename here):

command = ['rename','-vn', 's/(.*)\.htm$/model.html/']
files = glob.glob('*.htm')
command.extend(files)
p = subprocess.Popen(
    command,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    )

Jeffrey

thanks Jeffrey, that worked like a charm!

I'm trying to detect when the subprocess has terminated using the
wait() function - but when there is an error with the call to rename
(ie the file doesn't exist) rename (when run from the command line
just terminates and displays the error). In the code above, though,
my call to p.wait() just hangs when rename should throw an error...
I've tried adding shell=True but that stops the rename from working.
any ideas?

thanks