using cgi form via http and extracting results

Discussion in 'Python' started by bmgx, Jan 4, 2004.

  1. bmgx

    bmgx Guest

    I would like to use an already existing online service (currency
    converter) basically consisting of a html form with a few options that
    is submitted and returns the results. I really don't know where to start
    but I have assumed the following steps need to be taken:

    1) Make an http connection to the remote script (http://server/script.cgi)

    2) Fill out the html form (1x TEXT and 2x SELECT input fields)

    3) Submit the form

    4) extract the actual returned result. (using regex or something..)

    Any help on which modules to use would be much appreciated!
     
    bmgx, Jan 4, 2004
    #1
    1. Advertising

  2. bmgx

    bmgx Guest

    I found a possible solution:
    http://wwwsearch.sourceforge.net/ClientForm

    bmgx wrote:

    > I would like to use an already existing online service (currency
    > converter) basically consisting of a html form with a few options that
    > is submitted and returns the results. I really don't know where to start
    > but I have assumed the following steps need to be taken:
    >
    > 1) Make an http connection to the remote script (http://server/script.cgi)
    >
    > 2) Fill out the html form (1x TEXT and 2x SELECT input fields)
    >
    > 3) Submit the form
    >
    > 4) extract the actual returned result. (using regex or something..)
    >
    > Any help on which modules to use would be much appreciated!
    >
     
    bmgx, Jan 4, 2004
    #2
    1. Advertising

  3. bmgx

    Tim Roberts Guest

    bmgx <> wrote:

    >I would like to use an already existing online service (currency
    >converter) basically consisting of a html form with a few options that
    >is submitted and returns the results. I really don't know where to start
    >but I have assumed the following steps need to be taken:
    >
    >1) Make an http connection to the remote script (http://server/script.cgi)
    >
    >2) Fill out the html form (1x TEXT and 2x SELECT input fields)
    >
    >3) Submit the form
    >
    >4) extract the actual returned result. (using regex or something..)


    You don't actually need steps 1 and 2 at all. HTTP transactions are all
    completely separate. The results of a form submission are just a URL. If
    the form has fields "FromCurrency", "ToCurrency", and "Amount", all you
    have to do is this:


    http://www.currencyservice.com?FromCurrency=dollars&ToCurrency=pounds&Amount=15

    You don't have to HAVE the form source in order to submit a request.
    --
    - Tim Roberts,
    Providenza & Boekelheide, Inc.
     
    Tim Roberts, Jan 6, 2004
    #3
  4. bmgx

    Ben Finney Guest

    On Mon, 05 Jan 2004 17:39:30 -0800, Tim Roberts wrote:
    > bmgx <> wrote:
    >>1) Make an http connection to the remote script (http://server/script.cgi)
    >>2) Fill out the html form (1x TEXT and 2x SELECT input fields)
    >>3) Submit the form
    >>4) extract the actual returned result. (using regex or something..)

    >
    > You don't actually need steps 1 and 2 at all.


    True.

    > HTTP transactions are all completely separate.


    True (ignoring cookies for now).

    > The results of a form submission are just a URL.


    Usually false.

    The result of most form submissions is an HTTP POST request, which
    contains the URL and form content as separate data.

    > If the form has fields "FromCurrency", "ToCurrency", and "Amount", all
    > you have to do is this:
    >
    > http://www.currencyservice.com?FromCurrency=dollars&ToCurrency=pounds&Amount=15


    This is only true if the form action is an HTTP GET request, which is a
    rather insecure and ugly way to submit form data, and makes it
    impossible to submit many kinds of data. HTTP POST is the recommended
    method for submitting data to a CGI script.

    > You don't have to HAVE the form source in order to submit a request.


    True. You just need to construct the HTTP POST request correctly.

    --
    \ "It's a good thing we have gravity or else when birds died |
    `\ they'd just stay right up there. Hunters would be all |
    _o__) confused." -- Steven Wright |
    Ben Finney <http://bignose.squidly.org/>
     
    Ben Finney, Jan 6, 2004
    #4
  5. bmgx

    bmgx Guest

    How would one "create" post data so that I can skip the html form and
    access the submission script (in <form method="POST" action="SCRIPT">)
    directly. This is simple if the script uses GET as mentioned, but
    unfortunately this is not the case which is the very reason for my dilemma..

    Ben Finney wrote:
    > On Mon, 05 Jan 2004 17:39:30 -0800, Tim Roberts wrote:
    >
    >>bmgx <> wrote:
    >>
    >>>1) Make an http connection to the remote script (http://server/script.cgi)
    >>>2) Fill out the html form (1x TEXT and 2x SELECT input fields)
    >>>3) Submit the form
    >>>4) extract the actual returned result. (using regex or something..)

    >>
    >>You don't actually need steps 1 and 2 at all.

    >
    >
    > True.
    >
    >
    >>HTTP transactions are all completely separate.

    >
    >
    > True (ignoring cookies for now).
    >
    >
    >>The results of a form submission are just a URL.

    >
    >
    > Usually false.
    >
    > The result of most form submissions is an HTTP POST request, which
    > contains the URL and form content as separate data.
    >
    >
    >>If the form has fields "FromCurrency", "ToCurrency", and "Amount", all
    >>you have to do is this:
    >>
    >>http://www.currencyservice.com?FromCurrency=dollars&ToCurrency=pounds&Amount=15

    >
    >
    > This is only true if the form action is an HTTP GET request, which is a
    > rather insecure and ugly way to submit form data, and makes it
    > impossible to submit many kinds of data. HTTP POST is the recommended
    > method for submitting data to a CGI script.
    >
    >
    >>You don't have to HAVE the form source in order to submit a request.

    >
    >
    > True. You just need to construct the HTTP POST request correctly.
    >
     
    bmgx, Jan 6, 2004
    #5
  6. bmgx

    bmgx Guest

    Ok it's done right..

    import urllib
    a = urllib.urlopen("http://www.suckerservice.com/convert.cgi",
    "Amount=1&From=EUR&To=USD")
    b = a.readlines()
    c = open("tempresult.html","w")

    i = 0
    d = "********* local copy ******\n\n"
    while i < len(b):
    d = d + b
    i = i + 1

    c.write(d)
    c.close()
    a.close()

    bmgx wrote:

    > How would one "create" post data so that I can skip the html form and
    > access the submission script (in <form method="POST" action="SCRIPT">)
    > directly. This is simple if the script uses GET as mentioned, but
    > unfortunately this is not the case which is the very reason for my
    > dilemma..
    >
    > Ben Finney wrote:
    >
    >> On Mon, 05 Jan 2004 17:39:30 -0800, Tim Roberts wrote:
    >>
    >>> bmgx <> wrote:
    >>>
    >>>> 1) Make an http connection to the remote script
    >>>> (http://server/script.cgi)
    >>>> 2) Fill out the html form (1x TEXT and 2x SELECT input fields)
    >>>> 3) Submit the form
    >>>> 4) extract the actual returned result. (using regex or something..)
    >>>
    >>>
    >>> You don't actually need steps 1 and 2 at all.

    >>
    >>
    >>
    >> True.
    >>
    >>
    >>> HTTP transactions are all completely separate.

    >>
    >>
    >>
    >> True (ignoring cookies for now).
    >>
    >>
    >>> The results of a form submission are just a URL.

    >>
    >>
    >>
    >> Usually false.
    >>
    >> The result of most form submissions is an HTTP POST request, which
    >> contains the URL and form content as separate data.
    >>
    >>
    >>> If the form has fields "FromCurrency", "ToCurrency", and "Amount", all
    >>> you have to do is this:
    >>>
    >>> http://www.currencyservice.com?FromCurrency=dollars&ToCurrency=pounds&Amount=15
    >>>

    >>
    >>
    >>
    >> This is only true if the form action is an HTTP GET request, which is a
    >> rather insecure and ugly way to submit form data, and makes it
    >> impossible to submit many kinds of data. HTTP POST is the recommended
    >> method for submitting data to a CGI script.
    >>
    >>
    >>> You don't have to HAVE the form source in order to submit a request.

    >>
    >>
    >>
    >> True. You just need to construct the HTTP POST request correctly.
    >>

    >
     
    bmgx, Jan 6, 2004
    #6
  7. bmgx

    Joe Francia Guest

    bmgx wrote:
    > Ok it's done right..
    >
    > import urllib
    > a = urllib.urlopen("http://www.suckerservice.com/convert.cgi",
    > "Amount=1&From=EUR&To=USD")
    > b = a.readlines()
    > c = open("tempresult.html","w")
    >
    > i = 0
    > d = "********* local copy ******\n\n"
    > while i < len(b):
    > d = d + b
    > i = i + 1
    >
    > c.write(d)
    > c.close()
    > a.close()
    >


    Instead of constructing the data string by hand, you can pass a
    dictionary or list of 2-item tuples to urllib.urlencode() and get a
    properly formatted string:

    my_data = urllib.urlencode({'Amount':1,'From':'EUR','To':'USD'}) # or...
    my_data = urllib.urlencode([('Amount',1),('From','EUR'),('To','USD')])
    a = urllib.urlopen('http://www.suckerservice.com/convert.cgi', my_data)
     
    Joe Francia, Jan 6, 2004
    #7
  8. bmgx

    Tim Roberts Guest

    Ben Finney <> wrote:

    >On Mon, 05 Jan 2004 17:39:30 -0800, Tim Roberts wrote:
    >> bmgx <> wrote:
    >>>1) Make an http connection to the remote script (http://server/script.cgi)
    >>>2) Fill out the html form (1x TEXT and 2x SELECT input fields)
    >>>3) Submit the form
    >>>4) extract the actual returned result. (using regex or something..)

    >>
    >> You don't actually need steps 1 and 2 at all.

    >
    >True.
    >
    >> HTTP transactions are all completely separate.

    >
    >True (ignoring cookies for now).
    >
    >> The results of a form submission are just a URL.

    >
    >Usually false.
    >
    >The result of most form submissions is an HTTP POST request, which
    >contains the URL and form content as separate data.


    You are correct. However, in my experience, most handlers that expect POST
    data will also accept the same information via GET. That is fortunate,
    since it makes it easy to do things the way I suggested.

    >> If the form has fields "FromCurrency", "ToCurrency", and "Amount", all
    >> you have to do is this:
    >>
    >> http://www.currencyservice.com?FromCurrency=dollars&ToCurrency=pounds&Amount=15

    >
    >This is only true if the form action is an HTTP GET request, which is a
    >rather insecure and ugly way to submit form data, and makes it
    >impossible to submit many kinds of data. HTTP POST is the recommended
    >method for submitting data to a CGI script.


    There is no such "recommendation". GET is entirely appropriate for many
    kinds of information. A currency conversion service such as the one the
    O.P. described is a perfect example. There is no need for security in such
    a service, and it makes it easy to do canned requests like the one above.
    --
    - Tim Roberts,
    Providenza & Boekelheide, Inc.
     
    Tim Roberts, Jan 7, 2004
    #8
  9. bmgx

    Paul Rubin Guest

    Basically you have to examine the html source for the page you want to
    automate, and figure out how the form works. To do that, you need to
    understand html and http, there is no way around it. Maybe someone
    here can suggest a good reference for that.

    Once you've figured out the form, just use the urllib module to build
    and send the appropriate http request.
     
    Paul Rubin, Jan 7, 2004
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Alpha
    Replies:
    4
    Views:
    362
    Mark Parnell
    Dec 14, 2005
  2. Brian Cole
    Replies:
    0
    Views:
    281
    Brian Cole
    Jan 16, 2006
  3. stew dean

    Email HTML form results via ASPX

    stew dean, Mar 2, 2007, in forum: ASP .Net
    Replies:
    2
    Views:
    947
    stew dean
    Mar 2, 2007
  4. nntp
    Replies:
    12
    Views:
    396
    Joe Smith
    Oct 23, 2004
  5. bpascal123

    c cgi script display results below html form

    bpascal123, Mar 17, 2013, in forum: C Programming
    Replies:
    3
    Views:
    305
    Joachim Schmitz
    Mar 19, 2013
Loading...

Share This Page