Need help running external program

Discussion in 'Python' started by Rigga, Feb 27, 2005.

  1. Rigga

    Rigga Guest

    Hi,

    I am running the line of code below from a shell script and it works fine,
    however I am at a total loss on how i can run it from within a Python
    script as every option I have tried fails and it appears to be down to the
    escaping of certain characters.

    wget -q www.anywebpage.com -O - | tr '\r' '\n' | tr \' \" | sed -n
    's/.*url="\([^"]*\)".*/\1/p'

    I want to use the above code in my program by using popen2 so i can query
    the results i.e.

    output, input = popen2("the code here")

    print output

    Any help would be appreciated.

    Many thanks

    RiGGa
    Rigga, Feb 27, 2005
    #1
    1. Advertising

  2. Rigga

    Pink Guest

    Rigga wrote:

    > Hi,
    >
    > I am running the line of code below from a shell script and it works fine,
    > however I am at a total loss on how i can run it from within a Python
    > script as every option I have tried fails and it appears to be down to the
    > escaping of certain characters.
    >
    > wget -q www.anywebpage.com -O - | tr '\r' '\n' | tr \' \" | sed -n
    > 's/.*url="\([^"]*\)".*/\1/p'

    If your problem is getting a python string without worrying about how to
    escape the escape sequences, try:

    r"""wget -q www.anywebpage.com -O - | tr '\r' '\n' | tr \' \" | sed -n
    's/.*url="\([^"]*\)".*/\1/p'"""

    You should be able to pass this directly to a popen() function.
    Pink, Feb 27, 2005
    #2
    1. Advertising

  3. Rigga

    Rigga Guest

    Pink wrote:

    > Rigga wrote:
    >
    >> Hi,
    >>
    >> I am running the line of code below from a shell script and it works
    >> fine, however I am at a total loss on how i can run it from within a
    >> Python script as every option I have tried fails and it appears to be
    >> down to the escaping of certain characters.
    >>
    >> wget -q www.anywebpage.com -O - | tr '\r' '\n' | tr \' \" | sed -n
    >> 's/.*url="\([^"]*\)".*/\1/p'

    > If your problem is getting a python string without worrying about how to
    > escape the escape sequences, try:
    >
    > r"""wget -q www.anywebpage.com -O - | tr '\r' '\n' | tr \' \" | sed -n
    > 's/.*url="\([^"]*\)".*/\1/p'"""
    >
    > You should be able to pass this directly to a popen() function.


    Hi,

    Thanks for replying however I have just tried that and it does not seem to
    work, it doesnt return any results (i take it the r was a typo)

    Thanks

    RiGGa
    Rigga, Feb 27, 2005
    #3
  4. I'm using wget from Python to get extactly one line from a reports page.
    I made this function that works for me:

    def wgetline(exp): # see Python Cookbook p. 228
    print "Getting result from server ..."
    command = 'wget -q -O - \
    http://www.foobar.com/report.pl\?UserID=xxx\&UserPW=xxx \
    | grep ' + exp
    child = os.popen(command)
    data = child.read()
    return data

    I had to escape the ? and & in the url, or the CGI script at the other
    end would refuse to cooperate with "Invalid UserID or UserPW".
    --
    Leif Biberg Kristensen
    Leif B. Kristensen, Feb 27, 2005
    #4
  5. Rigga

    Tim Jarman Guest

    Rigga wrote:

    > Pink wrote:
    >
    >> Rigga wrote:
    >>
    >>> Hi,
    >>>
    >>> I am running the line of code below from a shell script and it works
    >>> fine, however I am at a total loss on how i can run it from within a
    >>> Python script as every option I have tried fails and it appears to be
    >>> down to the escaping of certain characters.
    >>>
    >>> wget -q www.anywebpage.com -O - | tr '\r' '\n' | tr \' \" | sed -n
    >>> 's/.*url="\([^"]*\)".*/\1/p'

    >> If your problem is getting a python string without worrying about how to
    >> escape the escape sequences, try:
    >>
    >> r"""wget -q www.anywebpage.com -O - | tr '\r' '\n' | tr \' \" | sed -n
    >> 's/.*url="\([^"]*\)".*/\1/p'"""
    >>
    >> You should be able to pass this directly to a popen() function.

    >
    > Hi,
    >
    > Thanks for replying however I have just tried that and it does not seem to
    > work, it doesnt return any results (i take it the r was a typo)
    >
    > Thanks
    >
    > RiGGa


    No, the r was the point - it's there to tell Python not to do any escaping
    on the string. Try it again with the r and see what happens.

    --
    Website: www DOT jarmania FULLSTOP com
    Tim Jarman, Feb 27, 2005
    #5
  6. Rigga

    Rigga Guest

    Tim Jarman wrote:

    > Rigga wrote:
    >
    >> Pink wrote:
    >>
    >>> Rigga wrote:
    >>>
    >>>> Hi,
    >>>>
    >>>> I am running the line of code below from a shell script and it works
    >>>> fine, however I am at a total loss on how i can run it from within a
    >>>> Python script as every option I have tried fails and it appears to be
    >>>> down to the escaping of certain characters.
    >>>>
    >>>> wget -q www.anywebpage.com -O - | tr '\r' '\n' | tr \' \" | sed -n
    >>>> 's/.*url="\([^"]*\)".*/\1/p'
    >>> If your problem is getting a python string without worrying about how to
    >>> escape the escape sequences, try:
    >>>
    >>> r"""wget -q www.anywebpage.com -O - | tr '\r' '\n' | tr \' \" | sed -n
    >>> 's/.*url="\([^"]*\)".*/\1/p'"""
    >>>
    >>> You should be able to pass this directly to a popen() function.

    >>
    >> Hi,
    >>
    >> Thanks for replying however I have just tried that and it does not seem
    >> to work, it doesnt return any results (i take it the r was a typo)
    >>
    >> Thanks
    >>
    >> RiGGa

    >
    > No, the r was the point - it's there to tell Python not to do any escaping
    > on the string. Try it again with the r and see what happens.
    >

    Brilliant!!! that works a treat thankyou!!, where on earth did you find out
    about the 'r' any pointers to documentation appreciated.

    Thanks

    RiGGa
    Rigga, Feb 27, 2005
    #6
  7. Rigga said unto the world upon 2005-02-27 15:04:
    > Tim Jarman wrote:


    <SNIP>

    >>No, the r was the point - it's there to tell Python not to do any escaping
    >>on the string. Try it again with the r and see what happens.
    >>

    >
    > Brilliant!!! that works a treat thankyou!!, where on earth did you find out
    > about the 'r' any pointers to documentation appreciated.
    >
    > Thanks
    >
    > RiGGa


    <http://www.python.org/doc/current/ref/strings.html>
    <http://www.python.org/doc/current/lib/module-re.html>

    Best,

    Brian vdB
    Brian van den Broek, Feb 27, 2005
    #7
  8. Rigga

    Pink Guest

    Rigga wrote:
    >> No, the r was the point - it's there to tell Python not to do any
    >> escaping on the string. Try it again with the r and see what happens.
    >>

    > Brilliant!!! that works a treat thankyou!!, where on earth did you find
    > out
    > about the 'r' any pointers to documentation appreciated.

    This is a pretty common problem when working with regular expression (which
    usually contain many backslashes) - that's where I saw this syntax for the
    first time (e.g. http://docs.python.org/lib/match-objects.html).
    The official reference for string literals is here:
    http://docs.python.org/ref/strings.html

    c ya
    Pink, Feb 27, 2005
    #8
  9. Rigga

    Rigga Guest

    Brian van den Broek wrote:

    > Rigga said unto the world upon 2005-02-27 15:04:
    >> Tim Jarman wrote:

    >
    > <SNIP>
    >
    >>>No, the r was the point - it's there to tell Python not to do any
    >>>escaping on the string. Try it again with the r and see what happens.
    >>>

    >>
    >> Brilliant!!! that works a treat thankyou!!, where on earth did you find
    >> out
    >> about the 'r' any pointers to documentation appreciated.
    >>
    >> Thanks
    >>
    >> RiGGa

    >
    > <http://www.python.org/doc/current/ref/strings.html>
    > <http://www.python.org/doc/current/lib/module-re.html>
    >
    > Best,
    >
    > Brian vdB

    Thanks for all your help with this it is appreciated, one further question
    though, how do I pass a variable to the external program while using the
    r"""

    Thanks

    RiGGa
    Rigga, Mar 2, 2005
    #9
  10. Rigga

    Tim Jarman Guest

    Rigga wrote:

    > Brian van den Broek wrote:
    >
    >> Rigga said unto the world upon 2005-02-27 15:04:


    (snip stuff about raw strings)

    > Thanks for all your help with this it is appreciated, one further question
    > though, how do I pass a variable to the external program while using the
    > r"""
    >
    > Thanks
    >
    > RiGGa


    I'm not sure I understand the question. Say you have:

    parameter = r"my \funky \text"

    then surely you just pass it to your external program using whichever method
    you like, e.g.

    import os
    os.execl("your_external_prog", parameter) # replaces the current process

    or some variant of:

    return_code = os.spawnl(os.P_WAIT, "your_external_prog", parameter)

    or you can build a command line:

    command = "your_external_prog %s" % parameter
    return_code = os.system(command)

    (see docs on the os module for more variations on this theme than you can
    shack a stick at)

    It's just a string, after all.



    --
    Website: www DOT jarmania FULLSTOP com
    Tim Jarman, Mar 2, 2005
    #10
  11. Rigga

    Rigga Guest

    Tim Jarman wrote:

    > Rigga wrote:
    >
    >> Brian van den Broek wrote:
    >>
    >>> Rigga said unto the world upon 2005-02-27 15:04:

    >
    > (snip stuff about raw strings)
    >
    >> Thanks for all your help with this it is appreciated, one further
    >> question though, how do I pass a variable to the external program while
    >> using the r"""
    >>
    >> Thanks
    >>
    >> RiGGa

    >
    > I'm not sure I understand the question. Say you have:
    >
    > parameter = r"my \funky \text"
    >
    > then surely you just pass it to your external program using whichever
    > method you like, e.g.
    >
    > import os
    > os.execl("your_external_prog", parameter) # replaces the current process
    >
    > or some variant of:
    >
    > return_code = os.spawnl(os.P_WAIT, "your_external_prog", parameter)
    >
    > or you can build a command line:
    >
    > command = "your_external_prog %s" % parameter
    > return_code = os.system(command)
    >
    > (see docs on the os module for more variations on this theme than you can
    > shack a stick at)
    >
    > It's just a string, after all.
    >
    >
    >

    This is the command I am trying to run:

    feed is a list of web addresses

    output, input = popen2("wget -q %s -O - | tr '\r' '\n' | tr \' \" | sed -n
    's/.*url="\([^"]*\)".*/\1/p'" % feed[counter])

    But it does not work, if I escape the string using r""" and hard code in the
    web address rather than use %s and feed[counter] it works, my question is
    how do I escape the string to get it to work with the %s and feed[counter]

    Im new to python as you can tell :)
    Rigga, Mar 2, 2005
    #11
  12. Rigga

    Tim Jarman Guest

    Rigga wrote:

    (snip)

    >>

    > This is the command I am trying to run:
    >
    > feed is a list of web addresses
    >
    > output, input = popen2("wget -q %s -O - | tr '\r' '\n' | tr \' \" | sed -n
    > 's/.*url="\([^"]*\)".*/\1/p'" % feed[counter])
    >
    > But it does not work, if I escape the string using r""" and hard code in
    > the web address rather than use %s and feed[counter] it works, my question
    > is how do I escape the string to get it to work with the %s and
    > feed[counter]
    >
    > Im new to python as you can tell :)


    Disclaimer: I know nothing about wget beyond what just having typed 'man
    wget' told me! ;)

    1. What *exactly* does "it does not work" mean? Do you get a traceback? If
    so, post it - that will help others to help you. What results are you
    expecting?

    2. Are you sure feed contains what you think it contains at this point in
    your program? What do you see if you do:

    for thing in feed: print thing

    ?

    A good strategy in these cases is to go in small steps. before you try
    getting fancy with popen2, have your program just print the command-line
    correctly. Then maybe try it with something like echo just to see that
    you're passing what you think you're passing. And so on.

    We were all new once - no blame! :)

    --
    Website: www DOT jarmania FULLSTOP com
    Tim Jarman, Mar 2, 2005
    #12
  13. Rigga

    Steve Holden Guest

    Rigga wrote:
    > Tim Jarman wrote:
    >
    >
    >>Rigga wrote:
    >>
    >>
    >>>Brian van den Broek wrote:
    >>>
    >>>
    >>>>Rigga said unto the world upon 2005-02-27 15:04:

    >>
    >>(snip stuff about raw strings)
    >>
    >>
    >>>Thanks for all your help with this it is appreciated, one further
    >>>question though, how do I pass a variable to the external program while
    >>>using the r"""
    >>>
    >>>Thanks
    >>>
    >>>RiGGa

    >>
    >>I'm not sure I understand the question. Say you have:
    >>
    >>parameter = r"my \funky \text"
    >>
    >>then surely you just pass it to your external program using whichever
    >>method you like, e.g.
    >>
    >>import os
    >>os.execl("your_external_prog", parameter) # replaces the current process
    >>
    >>or some variant of:
    >>
    >>return_code = os.spawnl(os.P_WAIT, "your_external_prog", parameter)
    >>
    >>or you can build a command line:
    >>
    >>command = "your_external_prog %s" % parameter
    >>return_code = os.system(command)
    >>
    >>(see docs on the os module for more variations on this theme than you can
    >>shack a stick at)
    >>
    >>It's just a string, after all.
    >>
    >>
    >>

    >
    > This is the command I am trying to run:
    >
    > feed is a list of web addresses
    >
    > output, input = popen2("wget -q %s -O - | tr '\r' '\n' | tr \' \" | sed -n
    > 's/.*url="\([^"]*\)".*/\1/p'" % feed[counter])
    >
    > But it does not work, if I escape the string using r""" and hard code in the
    > web address rather than use %s and feed[counter] it works, my question is
    > how do I escape the string to get it to work with the %s and feed[counter]
    >
    > Im new to python as you can tell :)


    Right, using raw strings (r" ... ") makes sure that backslashes in the
    literal are retained rather than used as escapes. Socould you show us an
    example where the expression (using ... % feed[counter]) gives you a
    different value from "hard coding the web address"?

    I suspect if you use a raw string for the format then that will be
    enough - in other words, does

    output, input = popen2(r"""wget -q %s -O - | tr '\r' '\n' | tr \' \" |
    sed -n 's/.*url="\([^"]*\)".*/\1/p'""" % feed[counter])

    work?

    regards
    Steve
    --
    Meet the Python developers and your c.l.py favorites March 23-25
    Come to PyCon DC 2005 http://www.pycon.org/
    Steve Holden http://www.holdenweb.com/
    Steve Holden, Mar 2, 2005
    #13
  14. Rigga

    Rigga Guest

    Thanks to all for your help it is now working, I rant he code through a
    debugger and found that the input file I was using to create my list of
    addresses to wget had newlines in them and were therefore breaking my
    command line.

    All your advice has been appreciated.

    RiGGa
    Rigga, Mar 3, 2005
    #14
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    2
    Views:
    460
  2. Replies:
    0
    Views:
    289
  3. Replies:
    1
    Views:
    243
    Dennis Lee Bieber
    Nov 25, 2006
  4. Heikki Toivonen
    Replies:
    2
    Views:
    275
    Heikki Toivonen
    Mar 30, 2007
  5. Zhao Yi
    Replies:
    4
    Views:
    131
    Brian Candler
    Jan 7, 2009
Loading...

Share This Page