subprocess and & (ampersand)

S

Steven Bethard

I'm having trouble using the subprocess module on Windows when my
command line includes special characters like "&" (ampersand)::
.... stdout=subprocess.PIPE,
.... stderr=subprocess.PIPE)"'y' is not recognized as an internal or external command,\r\noperable
program or batch file.\r\n"

As you can see, Windows is interpreting that "&" as separating two
commands, instead of being part of the single argument as I intend it to
be above. Is there any workaround for this? How do I get "&" treated
like a regular character using the subprocess module?

Thanks,

STeVe
 
S

Steven D'Aprano

I'm having trouble using the subprocess module on Windows when my
command line includes special characters like "&" (ampersand)::

... stdout=subprocess.PIPE, ...
stderr=subprocess.PIPE)
"'y' is not recognized as an internal or external command,\r\noperable
program or batch file.\r\n"

As you can see, Windows is interpreting that "&" as separating two
commands, instead of being part of the single argument as I intend it to
be above. Is there any workaround for this? How do I get "&" treated
like a regular character using the subprocess module?


That's nothing to do with the subprocess module. As you say, it is
Windows interpreting the ampersand as a special character, so you need to
escape the character to the Windows shell.

Under Windows, the escape character is ^, or you can put the string in
double quotes:

# untested
command = 'lynx.bat -dump http://www.example.com/?x=1^&y=2'
command = 'lynx.bat -dump "http://www.example.com/?x=1&y=2"'

In Linux land, you would use a backslash or quotes.

To find the answer to this question, I googled for "windows how to escape
special characters shell" and found these two pages:


http://www.microsoft.com/technet/archive/winntas/deploy/prodspecs/shellscr.mspx

http://technet2.microsoft.com/WindowsServer/en/library/44500063-fdaf-4e4f-8dac-476c497a166f1033.mspx


Hope this helps,
 
S

Steven Bethard

Steven said:
That's nothing to do with the subprocess module. As you say, it is
Windows interpreting the ampersand as a special character, so you need to
escape the character to the Windows shell.

Under Windows, the escape character is ^, or you can put the string in
double quotes:

# untested
command = 'lynx.bat -dump http://www.example.com/?x=1^&y=2'
command = 'lynx.bat -dump "http://www.example.com/?x=1&y=2"'

Sorry, I should have mentioned that I already tried that. You get the
same result::
... stdin=subprocess.PIPE,
... stdout=subprocess.PIPE,
... stderr=subprocess.PIPE) "'y' is not recognized as an internal or external command,\r\noperable
program or batch file.\r\n"

In fact, the "^" doesn't seem to work at the command line either::

Can't Access `file://localhost/C:/PROGRA~1/lynx/1'
Alert!: Unable to access document.

lynx: Can't access startfile
'y' is not recognized as an internal or external command,
operable program or batch file.

Using quotes does work at the command line::

C:\PROGRA~1\lynx>lynx.bat -dump "http://www.example.com/?x=1&y=2"
You have reached this web page by typing "example.com",
"example.net", or "example.org" into your web browser.

These domain names are reserved for use in documentation and are
not available for registration. See [1]RFC 2606, Section 3.

References

1. http://www.rfc-editor.org/rfc/rfc2606.txt

But I get no output at all when using quotes with subprocess::
... stdin=subprocess.PIPE,
... stdout=subprocess.PIPE,
... stderr=subprocess.PIPE) ''

Any other ideas?

STeVe
 
T

Tim Golden

Steven said:
I'm having trouble using the subprocess module on Windows when my
command line includes special characters like "&" (ampersand)::

... stdout=subprocess.PIPE,
... stderr=subprocess.PIPE)
"'y' is not recognized as an internal or external command,\r\noperable
program or batch file.\r\n"

As you can see, Windows is interpreting that "&" as separating two
commands, instead of being part of the single argument as I intend it to
be above. Is there any workaround for this? How do I get "&" treated
like a regular character using the subprocess module?

A little experimentation suggests that the problem's somehow
tied up with the .bat file. ie this works for me (doubly
complicated because of the long firefox path:

<code>
import subprocess

cmd = [
r"c:\Program Files\Mozilla Firefox\firefox.exe",
"http://local.goodtoread.org/search?word=tim&cached=0"
]
subprocess.Popen (cmd)

</code>

but this doesn't:

<c:/temp/firefox.bat>
"c:\Program Files\Mozilla Firefox\firefox.exe" "%*"
</c:/temp/firefox.bat>

<code>
import subprocess

cmd = [
r"c:\temp\firefox.bat",
"http://local.goodtoread.org/search?word=tim&cached=0"
]
subprocess.Popen (cmd)

</code>

although, interestingly, it seems to cut off at the
"=" before the "&". Not sure how significant that is.

So, even assuming we're looking at the same situation,
I suppose one solution for you is to break out the
..bat file. But that may not be a possibility.

TJG
 
T

Tim Golden

Steven said:
I'm having trouble using the subprocess module on Windows when my
command line includes special characters like "&" (ampersand)::

... stdout=subprocess.PIPE,
... stderr=subprocess.PIPE)
"'y' is not recognized as an internal or external command,\r\noperable
program or batch file.\r\n"

As you can see, Windows is interpreting that "&" as separating two
commands, instead of being part of the single argument as I intend it to
be above. Is there any workaround for this? How do I get "&" treated
like a regular character using the subprocess module?

Although I'm sure you'll have looked into this already, the
subprocess module on Windows is using CreateProcess pretty
straightforwardly:

http://msdn2.microsoft.com/en-us/library/ms682425(VS.85).aspx

The docs there say that, to call a batch file, you need to
specify the command interpreter with /c and pass the batch
file, but as far as I can see it makes no difference! (Probably
means there's a special-caser behind the scenes of CreateProcess).

TJG
 
T

Tim Golden

Ross said:
Tim Golden said:
but this doesn't:

<c:/temp/firefox.bat>
"c:\Program Files\Mozilla Firefox\firefox.exe" "%*"
</c:/temp/firefox.bat>

<code>
import subprocess

cmd = [
r"c:\temp\firefox.bat",
"http://local.goodtoread.org/search?word=tim&cached=0"
]
subprocess.Popen (cmd)

</code>

You need to use double quotes both in the .BAT file and in the string
you pass to subprocess.Popen().

Ross Ridge

In the context of my example above, could you just
say which bit you thing should be quoted and isn't?
(That sounds sarcastic, but isn't; I just want to
understand if I've missed something). If you simply
requote the second element in the cmd list
('"http:/....."') then the internal quotes are escaped
by some part of the mechanism and it still doesn't work.

TJG
 
R

Ross Ridge

Tim Golden said:
but this doesn't:

<c:/temp/firefox.bat>
"c:\Program Files\Mozilla Firefox\firefox.exe" "%*"
</c:/temp/firefox.bat>

<code>
import subprocess

cmd = [
r"c:\temp\firefox.bat",
"http://local.goodtoread.org/search?word=tim&cached=0"
]
subprocess.Popen (cmd)

</code>

Ross said:
You need to use double quotes both in the .BAT file and in the string
you pass to subprocess.Popen().

Tim Golden said:
... If you simply requote the second element in the cmd list
('"http:/....."') then the internal quotes are escaped by some part of
the mechanism and it still doesn't work.

Hmm... I guess things are much more messy than that. CMD doesn't do
standard quote processing of it's arguments or .BAT file arguments, and
so is incompatible with how subprocess quotes args lists. It looks like
you need use a string instead of list with subprocess.Popen and not use
quotes in the batch file. So something like:

firefox.bat:
"c:\Program Files\Mozilla Firefox\firefox.exe" %1

code:
subprocess.Popen(r'c:\temp\firefox.bat "http://local.goodtoread.org/search?word=tim&cached=0"')

Ross Ridge
 
S

Steven Bethard

Ross said:
Tim Golden said:
but this doesn't:

<c:/temp/firefox.bat>
"c:\Program Files\Mozilla Firefox\firefox.exe" "%*"
</c:/temp/firefox.bat>

<code>
import subprocess

cmd = [
r"c:\temp\firefox.bat",
"http://local.goodtoread.org/search?word=tim&cached=0"
]
subprocess.Popen (cmd)

</code>

Ross said:
You need to use double quotes both in the .BAT file and in the string
you pass to subprocess.Popen().

Tim Golden said:
... If you simply requote the second element in the cmd list
('"http:/....."') then the internal quotes are escaped by some part of
the mechanism and it still doesn't work.

Hmm... I guess things are much more messy than that. CMD doesn't do
standard quote processing of it's arguments or .BAT file arguments, and
so is incompatible with how subprocess quotes args lists. It looks like
you need use a string instead of list with subprocess.Popen and not use
quotes in the batch file. So something like:

firefox.bat:
"c:\Program Files\Mozilla Firefox\firefox.exe" %1

code:
subprocess.Popen(r'c:\temp\firefox.bat "http://local.goodtoread.org/search?word=tim&cached=0"')

This works. The other thing that works is to put extra quotes around
the URL when you pass in the list::

... stdin=subprocess.PIPE,
... stdout=subprocess.PIPE,
... stderr=subprocess.PIPE) ' You have reached this web page by typing "example.com",
"example.net",\n or "example.org" into your web browser.\n\n These
domain names are reserved for use in documentation and are not\n
available for registration. See [1]RFC 2606, Section
3.\n\nReferences\n\n 1. http://www.rfc-editor.org/rfc/rfc2606.txt\n'


I tried this before, and posted it, but I had mistakenly read from
stderr instead of stdout. Thanks for making me double-check that code!

STeVe
 
A

Albert van der Horst

Tim Golden said:
So, even assuming we're looking at the same situation,
I suppose one solution for you is to break out the
.bat file. But that may not be a possibility.

Unless you need compatibility with Windows 3.1., I suggest you
stay away from .bat files. Better call them .cmd, such that they
are executed by cmd.exe (32 bits).
It could be part of your problem.

Groetjes Albert
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,906
Latest member
SkinfixSkintag

Latest Threads

Top