Perl code to fill-in online form

Discussion in 'Perl Misc' started by Babul, Mar 21, 2008.

  1. Babul

    Babul Guest

    Hello everyone,

    I am trying to write a program in perl (windows) that I will use to
    fill out online form.

    Here are the steps that I follow to get the online form:
    1. I go to the web site first and get a Login form
    2. I fill in the Login form and submit that form. Then a page open
    with a link "Choose a period". When I click on that link, a small
    window pops up with links for different weeks. I click on my desired
    week.
    3. Now a page display with some project names, dates of my choosen
    week and box field to enter time that I spent on that progect.

    I tried WWW::Mechanize to get the contents of the web page.

    Here are the steps I followed to grab the contents of the web page:
    #! /usr/local/bin/perl

    use Tk;
    use Cwd;
    use WWW::Mechanize;
    $currentDir = getcwd;

    $mech = WWW::Mechanize->new();
    $mech->get("web address from the 3rd step above");
    $contents = $mech->content();
    open WF, ">$currentDir/temp.txt";
    print WF "$contents\n";
    close WF;

    But the contents I get is the source code of the Login page (from step
    1 above). My purpose is to get the source code of the web page which
    is displayed in step 3.

    If any one can help me I would appreciate that.

    Note: I also noticed that If I go to View->Source on the Internet
    Explorer to view the HTML source code of that web page, it does not
    display. But I can view source code using View->Source for any other
    web sites.

    Regards,

    Babul
     
    Babul, Mar 21, 2008
    #1
    1. Advertising

  2. Babul <> writes:

    > Hello everyone,


    hi!

    > I am trying to write a program in perl (windows) that I will use to
    > fill out online form.


    > 2. I fill in the Login form and submit that form. Then a page open
    > with a link "Choose a period". When I click on that link, a small
    > window pops up with links for different weeks. I click on my desired
    > week.


    ok

    > I tried WWW::Mechanize to get the contents of the web page.


    That's usually the best module to use for this, yes. but it sounds like
    the popup is opened via javascript and WWW::Mechanize doesn't support
    javascript.

    You either have to figure out how to get at the popup's url via
    WWW::Mechanize (probably using some hand-written rules), and/or you'll
    need to "mechanize" a javascript capable browser. Since you're on
    windows, you may want to try Win32::IE::Mechanize.

    See <http://search.cpan.org/~abeltje/Win32-IE-Mechanize-0.009/lib/Win32/IE/Mechanize.pm>

    I think this is all mentioned in the docs for WWW::Mechanize too.

    [ ...]

    > open WF, ">$currentDir/temp.txt";


    You don't need to specify the current directory. That's what a current
    directory is for.

    --
    Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/
     
    Joost Diepenmaat, Mar 22, 2008
    #2
    1. Advertising

  3. Babul

    Ben Bullock Guest

    On Fri, 21 Mar 2008 16:57:02 -0700, Babul wrote:

    > Hello everyone,
    >
    > I am trying to write a program in perl (windows) that I will use to fill
    > out online form.
    >
    > Here are the steps that I follow to get the online form: 1. I go to the
    > web site first and get a Login form 2. I fill in the Login form and
    > submit that form. Then a page open with a link "Choose a period". When I
    > click on that link, a small window pops up with links for different
    > weeks. I click on my desired week.
    > 3. Now a page display with some project names, dates of my choosen week
    > and box field to enter time that I spent on that progect.
    >
    > I tried WWW::Mechanize to get the contents of the web page.
    >
    > Here are the steps I followed to grab the contents of the web page: #!
    > /usr/local/bin/perl
    >
    > use Tk;
    > use Cwd;
    > use WWW::Mechanize;
    > $currentDir = getcwd;
    >
    > $mech = WWW::Mechanize->new();
    > $mech->get("web address from the 3rd step above"); $contents =
    > $mech->content();
    > open WF, ">$currentDir/temp.txt";
    > print WF "$contents\n";
    > close WF;
    >
    > But the contents I get is the source code of the Login page (from step 1
    > above). My purpose is to get the source code of the web page which is
    > displayed in step 3.
    >
    > If any one can help me I would appreciate that.


    You probably need to actually log in via Perl and get the cookies (login
    cookie, session cookie) via Perl. Even if your browser is logged in, Perl
    isn't able to share the cookies from the browser to access the web page.

    Here are some code fragments from a similar automated login which might
    help get you started:

    # modules I used

    use LWP::UserAgent;
    use HTML::Form;
    use HTTP::Cookies;

    # from the generic user agent starter for my login:

    sub my_module::start_user_agent
    {
    my $ua = LWP::UserAgent->new;

    $ua->cookie_jar(HTTP::Cookies->new(file => "lwpcookies.txt",
    autosave => 1));

    return $ua;
    }

    # from the automated login script:

    my $name = $form->find_input ("Name", "text")->value;
    my $remember = $form->find_input ("Remember", "checkbox")->value;
    my $password;# = $form->find_input ("Password", "text")->value;

    $form->value ("Name", "myusername");
    $form->value ("Remember", "1");
    $form->value ("Password", "mypassword");

    $request = $form->click;
    $response = $ua->request($request);

    # end

    I ran the login script as a different script since once I had the cookies
    I didn't need to run it again.

    Alternatively, if you want to automatically fill in the form from your
    browser, don't use Perl, use greasemonkey instead.
     
    Ben Bullock, Mar 22, 2008
    #3
  4. Babul

    Ben Bullock Guest

    On Sat, 22 Mar 2008 01:05:32 +0100, Joost Diepenmaat wrote:

    > Babul <> writes:
    >
    >> Hello everyone,

    >
    > hi!
    >
    >> I am trying to write a program in perl (windows) that I will use to
    >> fill out online form.

    >
    >> 2. I fill in the Login form and submit that form. Then a page open with
    >> a link "Choose a period". When I click on that link, a small window
    >> pops up with links for different weeks. I click on my desired week.

    >
    > ok
    >
    >> I tried WWW::Mechanize to get the contents of the web page.

    >
    > That's usually the best module to use for this, yes. but it sounds like
    > the popup is opened via javascript and WWW::Mechanize doesn't support
    > javascript.


    It's possible but highly unlikely. I think you didn't read where the
    poster said that he keeps getting the source code for the login window
    when he tries to access the page he wants. The problem is lack of
    authentication. If he has the URL for the popup window, window 3, which
    he said that he does:

    $mech->get("web address from the 3rd step above");

    it doesn't matter how window 2 opens window 3. Being opened with
    JavaScript has nothing to do with it.

    > You either have to figure out how to get at the popup's url via
    > WWW::Mechanize (probably using some hand-written rules), and/or you'll
    > need to "mechanize" a javascript capable browser. Since you're on
    > windows, you may want to try Win32::IE::Mechanize.


    No, no, no, this is really bad advice. You didn't read the poster's
    question properly.
     
    Ben Bullock, Mar 22, 2008
    #4
  5. Ben Bullock <> writes:

    > It's possible but highly unlikely. I think you didn't read where the
    > poster said that he keeps getting the source code for the login window
    > when he tries to access the page he wants. The problem is lack of
    > authentication. If he has the URL for the popup window, window 3, which
    > he said that he does:
    >
    > $mech->get("web address from the 3rd step above");
    >
    > it doesn't matter how window 2 opens window 3. Being opened with
    > JavaScript has nothing to do with it.


    Although you state your case a bit confusingly, after re-reading the
    original post I agree that my interpretation is probably wrong and yours
    is probably correct.

    >> You either have to figure out how to get at the popup's url via
    >> WWW::Mechanize (probably using some hand-written rules), and/or you'll
    >> need to "mechanize" a javascript capable browser. Since you're on
    >> windows, you may want to try Win32::IE::Mechanize.

    >
    > No, no, no, this is really bad advice. You didn't read the poster's
    > question properly.


    I do wonder why this is bad advice. AFAIK it should "work" either way.

    --
    Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/
     
    Joost Diepenmaat, Mar 22, 2008
    #5
  6. Babul

    Ben Bullock Guest

    On Sat, 22 Mar 2008 02:11:02 +0100, Joost Diepenmaat wrote:

    > Ben Bullock <> writes:


    >>> You either have to figure out how to get at the popup's url via
    >>> WWW::Mechanize (probably using some hand-written rules), and/or you'll
    >>> need to "mechanize" a javascript capable browser. Since you're on
    >>> windows, you may want to try Win32::IE::Mechanize.

    >>
    >> No, no, no, this is really bad advice. You didn't read the poster's
    >> question properly.

    >
    > I do wonder why this is bad advice. AFAIK it should "work" either way.


    I'm sorry to have written like that. I had this vision of the original
    poster spending hours and hours trying to do something which probably
    wouldn't have solved his problem.
     
    Ben Bullock, Mar 22, 2008
    #6
  7. Babul

    Babul Guest

    On Mar 21, 5:53 pm, Ben Bullock <> wrote:
    > On Sat, 22 Mar 2008 01:05:32 +0100, Joost Diepenmaat wrote:
    > >Babul<> writes:

    >
    > >> Hello everyone,

    >
    > > hi!

    >
    > >> I am trying to write a program in perl (windows) that I will use to
    > >> fill out online form.

    >
    > >> 2. I fill in the Login form and submit that form. Then a page open with
    > >> a link "Choose a period". When I click on that link, a small window
    > >> pops up with links for different weeks. I click on my desired week.

    >
    > > ok

    >
    > >> I tried WWW::Mechanize to get the contents of the web page.

    >
    > > That's usually the best module to use for this, yes. but it sounds like
    > > the popup is opened via javascript and WWW::Mechanize doesn't support
    > > javascript.

    >
    > It's possible but highly unlikely. I think you didn't read where the
    > poster said that he keeps getting the source code for the login window
    > when he tries to access the page he wants. The problem is lack of
    > authentication. If he has the URL for the popup window, window 3, which
    > he said that he does:
    >
    > $mech->get("web address from the 3rd step above");
    >
    > it doesn't matter how window 2 opens window 3. Being opened with
    > JavaScript has nothing to do with it.
    >
    > > You either have to figure out how to get at the popup's url via
    > > WWW::Mechanize (probably using some hand-written rules), and/or you'll
    > > need to "mechanize" a javascript capable browser. Since you're on
    > > windows, you may want to try Win32::IE::Mechanize.

    >
    > No, no, no, this is really bad advice. You didn't read the poster's
    > question properly.- Hide quoted text -
    >
    > - Show quoted text -


    Thanks everyone for the guidance and useful thought.

    The reason I am using Perl, I already developed a program in Perl to
    keep track of my time spend on any project. The program is working
    fine and basically it is create a pipe delimited text file with the
    following information:
    Project name|Task name|Date|Amount of time spent

    Now my intention is to add a button in that program so that when I
    will click that button it will grab the information from the pipe
    delimited file and will fill-out the online form.
     
    Babul, Mar 24, 2008
    #7
  8. Babul

    J. Gleixner Guest

    Babul wrote:
    [...]
    > The reason I am using Perl, I already developed a program in Perl to
    > keep track of my time spend on any project. The program is working
    > fine and basically it is create a pipe delimited text file with the
    > following information:
    > Project name|Task name|Date|Amount of time spent
    >
    > Now my intention is to add a button in that program so that when I
    > will click that button it will grab the information from the pipe
    > delimited file and will fill-out the online form.


    Thanks for sharing your intentions. If you have questions, then post
    your code and ask, otherwise no one really needs to know your
    intentions.
     
    J. Gleixner, Mar 24, 2008
    #8
  9. Babul

    ccc31807 Guest

    On Mar 21, 7:57 pm, Babul <> wrote:
    > Hello everyone,
    >
    > I am trying to write a program in perl (windows) that I will use to
    > fill out online form.


    I assume that since you can write scripts on the server that you have
    access to the httpd. This is how I would do it.

    1. Configure your httpd for cgi. Then, write a test script to make
    sure it works. You can configure your web server a couple of different
    ways, so pick the one you like.

    2. On the client side, make your html form the way you normally do.
    <html><body>
    <form action='post' method='/myurl/cgi-bin/form.cgi'>
    <input type='text' name='week' />
    <input type='submit' value='OK' />
    </form>
    </body></html>

    3. On the server side use CGI.
    #!/usr/bin/perl;
    use CGI;
    use strict;
    my $week = param('week');
    print "Content-type: text/html\n\n";
    print "<html><body>My week is $week</body></html>
    exit();

    CC
     
    ccc31807, Mar 24, 2008
    #9
  10. Babul

    Babul Guest

    On Mar 24, 3:16 pm, "J. Gleixner" <>
    wrote:
    > Babulwrote:
    >
    > [...]
    >
    > > The reason I am using Perl, I already developed a program in Perl to
    > > keep track of my time spend on any project. The program is working
    > > fine and basically it is create a pipe delimited text file with the
    > > following information:
    > > Project name|Task name|Date|Amount of time spent

    >
    > > Now my intention is to add a button in that program so that when I
    > > will click that button it will grab the information from the pipe
    > > delimited file and will fill-out the online form.

    >
    > Thanks for sharing your intentions.  If you have questions, then post
    > your code and ask, otherwise no one really needs to know your
    > intentions.


    If you think that you are really helpful, please see my code at the
    beginning of the discussion. Ben Bullock suggested me to use
    greasemonkey, so I explained here why I am using Perl.
     
    Babul, Mar 25, 2008
    #10
  11. On 2008-03-25 00:47, Babul <> wrote:
    > On Mar 24, 3:16 pm, "J. Gleixner" <>
    > wrote:
    >> Babulwrote:
    >> > The reason I am using Perl,

    [...]
    >>
    >> Thanks for sharing your intentions.  If you have questions, then post
    >> your code and ask, otherwise no one really needs to know your
    >> intentions.

    >
    > If you think that you are really helpful, please see my code at the
    > beginning of the discussion. Ben Bullock suggested me to use
    > greasemonkey, so I explained here why I am using Perl.


    Be more careful in quoting, then. You quoted a lot from Ben's reply but
    you didn't quote the part about gresemonkey. Please quote the part you
    are replying to (and only that) - otherwise nobody knows what you are
    talking about.

    The part which you did quote however, did probably identify your
    problem. Did you try his advice?

    hp
     
    Peter J. Holzer, Mar 25, 2008
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Matt MC
    Replies:
    2
    Views:
    540
    Thomas 'PointedEars' Lahn
    May 25, 2005
  2. Bill Gower

    Need advice on online fill in forms

    Bill Gower, Feb 17, 2008, in forum: ASP .Net
    Replies:
    8
    Views:
    349
    Jesse Houwing
    Feb 18, 2008
  3. nani
    Replies:
    0
    Views:
    430
  4. chinni
    Replies:
    0
    Views:
    427
    chinni
    Nov 20, 2009
  5. Matt MC
    Replies:
    2
    Views:
    180
    Thomas 'PointedEars' Lahn
    May 25, 2005
Loading...

Share This Page