Popen and wget, problems

[Jesse]

Hi all, I have a problem using wget and Popen. I hope someone can help.


-- Problem --
I want to use the command:
wget -nv -O "dir/cpan.txt" "http://search.cpan.org"
and capture all it's stdout+stderr.
(Note that option -O requires 'dir' to be existing before wget is executed)

Popen doesn't work, while os.system and shell do. Popen will give the error:
dir/cpan.txt: No such file or directory

While os.system and shell will give the correct result:
06:52:40 URL:http://search.cpan.org/ [3657/3657] -> "dir1/cpan.txt" [1]



-- Background info about wget --
-Option -nv: -nv, --no-verbose turn off verboseness, without
being quiet.
-Option -O: -O, --output-document=FILE write documents to FILE.

Note that wget requires any directories in the file-path of option -O to be
existing before the wget command is executed.


-- Python Code using Popen with cmd arg list --
# imports
import os
from subprocess import Popen, PIPE

# vars and create dir
cmd_set = ['wget', '-nv', '-O dir/cpan.txt', 'http://search.span.org']
cmd = ' '.join(cmd_set)
print "cmd: " + cmd
try:
os.makedirs('dir')
except:
print 'dir already exists'


# execute using Popen (does NOT work)
proc = Popen(cmd_set, stdout=PIPE, stderr=PIPE)
return_code = proc.wait()
if return_code == 0:
print "Success:\n%s" % (proc.stdout.read() + proc.stderr.read())
else:
print "Failure %s:\n%s" % (return_code, proc.stderr.read() +
proc.stdout.read())


# execute using os.system (does work)
os.system(cmd)


-- Python code output of Popen --
Failure 1:
dir/cpan.txt: No such file or directory


-- Question --
Why is Popen unable to correctly execute the wget, while os.system can?

 

[Rob Wolfe]

Hi all, I have a problem using wget and Popen. I hope someone can help.


-- Problem --
I want to use the command:
wget -nv -O "dir/cpan.txt" "http://search.cpan.org"
and capture all it's stdout+stderr.
(Note that option -O requires 'dir' to be existing before wget is executed)

Popen doesn't work, while os.system and shell do. Popen will give the error:
dir/cpan.txt: No such file or directory

While os.system and shell will give the correct result:
06:52:40 URL:http://search.cpan.org/ [3657/3657] -> "dir1/cpan.txt" [1]
[...]

-- Python Code using Popen with cmd arg list --
# imports
import os
from subprocess import Popen, PIPE

# vars and create dir
cmd_set = ['wget', '-nv', '-O dir/cpan.txt', 'http://search.span.org']
cmd = ' '.join(cmd_set)
print "cmd: " + cmd
try:
os.makedirs('dir')
except:
print 'dir already exists'


# execute using Popen (does NOT work)
proc = Popen(cmd_set, stdout=PIPE, stderr=PIPE)
return_code = proc.wait()
if return_code == 0:
print "Success:\n%s" % (proc.stdout.read() + proc.stderr.read())
else:
print "Failure %s:\n%s" % (return_code, proc.stderr.read() +
proc.stdout.read())


# execute using os.system (does work)
os.system(cmd)


-- Python code output of Popen --
Failure 1:
dir/cpan.txt: No such file or directory


-- Question --
Why is Popen unable to correctly execute the wget, while os.system can?

I don't know exactly why in this case Popen doesn't work,
but the counterpart of os.system is Popen with option shell=True
and the first parameter should be a string instead of list.
That seems to work:
proc = Popen("wget -nv -O dir/cpan.txt http://search.span.org",
shell=True, stdout=PIPE, stderr=PIPE)

and this variant seems to work too:
cmd_set = ['wget', '-nv', '-O', 'dir/cpan.txt', 'http://search.span.org']

 

[Jesse]

Thx Rob!

Your solution works perfect!

Hi all, I have a problem using wget and Popen. I hope someone can help.


-- Problem --
I want to use the command:
wget -nv -O "dir/cpan.txt" "http://search.cpan.org"
and capture all it's stdout+stderr.
(Note that option -O requires 'dir' to be existing before wget is
executed)

Popen doesn't work, while os.system and shell do. Popen will give the
error:
dir/cpan.txt: No such file or directory

While os.system and shell will give the correct result:
06:52:40 URL:http://search.cpan.org/ [3657/3657] -> "dir1/cpan.txt" [1]
[...]

-- Python Code using Popen with cmd arg list --
# imports
import os
from subprocess import Popen, PIPE

# vars and create dir
cmd_set = ['wget', '-nv', '-O dir/cpan.txt', 'http://search.span.org']
cmd = ' '.join(cmd_set)
print "cmd: " + cmd
try:
os.makedirs('dir')
except:
print 'dir already exists'


# execute using Popen (does NOT work)
proc = Popen(cmd_set, stdout=PIPE, stderr=PIPE)
return_code = proc.wait()
if return_code == 0:
print "Success:\n%s" % (proc.stdout.read() + proc.stderr.read())
else:
print "Failure %s:\n%s" % (return_code, proc.stderr.read() +
proc.stdout.read())


# execute using os.system (does work)
os.system(cmd)


-- Python code output of Popen --
Failure 1:
dir/cpan.txt: No such file or directory


-- Question --
Why is Popen unable to correctly execute the wget, while os.system can?

I don't know exactly why in this case Popen doesn't work,
but the counterpart of os.system is Popen with option shell=True
and the first parameter should be a string instead of list.
That seems to work:
proc = Popen("wget -nv -O dir/cpan.txt http://search.span.org",
shell=True, stdout=PIPE, stderr=PIPE)

and this variant seems to work too:
cmd_set = ['wget', '-nv', '-O', 'dir/cpan.txt', 'http://search.span.org']

 

[js]

Hi Jesse.

cmd_set = ['wget', '-nv', '-O dir/cpan.txt', 'http://search.span.org'] [snip]
proc = Popen(cmd_set, stdout=PIPE, stderr=PIPE)

wget will treat this as
$ wget -nv '-O dir/cpan.txt' "http://search.cpan.org"

And will emit the following error because there's no pathname ' dir/cpan.txt'.
(Note the pathname has a trailing space.)

dir/cpan.txt: No such file or directory

Replace '-O dir/cpan.txt" with '-Odir/cpan.txt' or '-O', 'dir/cpan.txt'
and it should work.

Hope this helps