subprocess and & (ampersand)

Discussion in 'Python' started by Steven Bethard, Jan 23, 2008.

  1. I'm having trouble using the subprocess module on Windows when my
    command line includes special characters like "&" (ampersand)::
    .... stdout=subprocess.PIPE,
    .... stderr=subprocess.PIPE)"'y' is not recognized as an internal or external command,\r\noperable
    program or batch file.\r\n"

    As you can see, Windows is interpreting that "&" as separating two
    commands, instead of being part of the single argument as I intend it to
    be above. Is there any workaround for this? How do I get "&" treated
    like a regular character using the subprocess module?

    Thanks,

    STeVe
     
    Steven Bethard, Jan 23, 2008
    #1
    1. Advertisements


  2. That's nothing to do with the subprocess module. As you say, it is
    Windows interpreting the ampersand as a special character, so you need to
    escape the character to the Windows shell.

    Under Windows, the escape character is ^, or you can put the string in
    double quotes:

    # untested
    command = 'lynx.bat -dump http://www.example.com/?x=1^&y=2'
    command = 'lynx.bat -dump "http://www.example.com/?x=1&y=2"'

    In Linux land, you would use a backslash or quotes.

    To find the answer to this question, I googled for "windows how to escape
    special characters shell" and found these two pages:


    http://www.microsoft.com/technet/archive/winntas/deploy/prodspecs/shellscr.mspx

    http://technet2.microsoft.com/WindowsServer/en/library/44500063-fdaf-4e4f-8dac-476c497a166f1033.mspx


    Hope this helps,
     
    Steven D'Aprano, Jan 23, 2008
    #2
    1. Advertisements

  3. Sorry, I should have mentioned that I already tried that. You get the
    same result::
    ... stdin=subprocess.PIPE,
    ... stdout=subprocess.PIPE,
    ... stderr=subprocess.PIPE) "'y' is not recognized as an internal or external command,\r\noperable
    program or batch file.\r\n"

    In fact, the "^" doesn't seem to work at the command line either::
    Can't Access `file://localhost/C:/PROGRA~1/lynx/1'
    Alert!: Unable to access document.

    lynx: Can't access startfile
    'y' is not recognized as an internal or external command,
    operable program or batch file.

    Using quotes does work at the command line::

    C:\PROGRA~1\lynx>lynx.bat -dump "http://www.example.com/?x=1&y=2"
    You have reached this web page by typing "example.com",
    "example.net", or "example.org" into your web browser.

    These domain names are reserved for use in documentation and are
    not available for registration. See [1]RFC 2606, Section 3.

    References

    1. http://www.rfc-editor.org/rfc/rfc2606.txt

    But I get no output at all when using quotes with subprocess::
    ... stdin=subprocess.PIPE,
    ... stdout=subprocess.PIPE,
    ... stderr=subprocess.PIPE) ''

    Any other ideas?

    STeVe
     
    Steven Bethard, Jan 23, 2008
    #3
  4. Steven Bethard

    Tim Golden Guest

    A little experimentation suggests that the problem's somehow
    tied up with the .bat file. ie this works for me (doubly
    complicated because of the long firefox path:

    <code>
    import subprocess

    cmd = [
    r"c:\Program Files\Mozilla Firefox\firefox.exe",
    "http://local.goodtoread.org/search?word=tim&cached=0"
    ]
    subprocess.Popen (cmd)

    </code>

    but this doesn't:

    <c:/temp/firefox.bat>
    "c:\Program Files\Mozilla Firefox\firefox.exe" "%*"
    </c:/temp/firefox.bat>

    <code>
    import subprocess

    cmd = [
    r"c:\temp\firefox.bat",
    "http://local.goodtoread.org/search?word=tim&cached=0"
    ]
    subprocess.Popen (cmd)

    </code>

    although, interestingly, it seems to cut off at the
    "=" before the "&". Not sure how significant that is.

    So, even assuming we're looking at the same situation,
    I suppose one solution for you is to break out the
    ..bat file. But that may not be a possibility.

    TJG
     
    Tim Golden, Jan 23, 2008
    #4
  5. Steven Bethard

    Tim Golden Guest

    Although I'm sure you'll have looked into this already, the
    subprocess module on Windows is using CreateProcess pretty
    straightforwardly:

    http://msdn2.microsoft.com/en-us/library/ms682425(VS.85).aspx

    The docs there say that, to call a batch file, you need to
    specify the command interpreter with /c and pass the batch
    file, but as far as I can see it makes no difference! (Probably
    means there's a special-caser behind the scenes of CreateProcess).

    TJG
     
    Tim Golden, Jan 23, 2008
    #5
  6. Steven Bethard

    Ross Ridge Guest

    You need to use double quotes both in the .BAT file and in the string
    you pass to subprocess.Popen().

    Ross Ridge
     
    Ross Ridge, Jan 23, 2008
    #6
  7. Steven Bethard

    Tim Golden Guest

    In the context of my example above, could you just
    say which bit you thing should be quoted and isn't?
    (That sounds sarcastic, but isn't; I just want to
    understand if I've missed something). If you simply
    requote the second element in the cmd list
    ('"http:/....."') then the internal quotes are escaped
    by some part of the mechanism and it still doesn't work.

    TJG
     
    Tim Golden, Jan 23, 2008
    #7
  8. Steven Bethard

    Ross Ridge Guest

    Hmm... I guess things are much more messy than that. CMD doesn't do
    standard quote processing of it's arguments or .BAT file arguments, and
    so is incompatible with how subprocess quotes args lists. It looks like
    you need use a string instead of list with subprocess.Popen and not use
    quotes in the batch file. So something like:

    firefox.bat:
    "c:\Program Files\Mozilla Firefox\firefox.exe" %1

    code:
    subprocess.Popen(r'c:\temp\firefox.bat "http://local.goodtoread.org/search?word=tim&cached=0"')

    Ross Ridge
     
    Ross Ridge, Jan 23, 2008
    #8
  9. This works. The other thing that works is to put extra quotes around
    the URL when you pass in the list::

    ... stdin=subprocess.PIPE,
    ... stdout=subprocess.PIPE,
    ... stderr=subprocess.PIPE) ' You have reached this web page by typing "example.com",
    "example.net",\n or "example.org" into your web browser.\n\n These
    domain names are reserved for use in documentation and are not\n
    available for registration. See [1]RFC 2606, Section
    3.\n\nReferences\n\n 1. http://www.rfc-editor.org/rfc/rfc2606.txt\n'


    I tried this before, and posted it, but I had mistakenly read from
    stderr instead of stdout. Thanks for making me double-check that code!

    STeVe
     
    Steven Bethard, Jan 23, 2008
    #9
  10. Unless you need compatibility with Windows 3.1., I suggest you
    stay away from .bat files. Better call them .cmd, such that they
    are executed by cmd.exe (32 bits).
    It could be part of your problem.
    Groetjes Albert
     
    Albert van der Horst, Feb 1, 2008
    #10
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.