read a webpage with username/password

Discussion in 'Perl Misc' started by jiehuang001@hotmail.com, Jun 12, 2006.

  1. Guest

    I am trying to use perl to extract the content of a webpage, such as an
    online newspaper. For example, I might want to have the program to send
    me any material with the word of "computer".

    I know that I can use Get("url") > output.txt to do that. But the
    problem is that to reach to the "url", it needs a username/password to
    login first. Of course, I know the username/password. My question is,
    how to pass the username/password from perl?

    Thank you very much!

    JH
    , Jun 12, 2006
    #1
    1. Advertising

  2. Paul Lalli Guest

    wrote:
    > I am trying to use perl to extract the content of a webpage, such as an
    > online newspaper. For example, I might want to have the program to send
    > me any material with the word of "computer".
    >
    > I know that I can use Get("url") > output.txt to do that


    Assuming you mean the get() method of the LWP::Simple module, first
    note that case matters. Second note that LWP::Simple also provides
    getstore() which puts the output in a file for you, rather than messing
    with shell redirection.

    > . But the
    > problem is that to reach to the "url", it needs a username/password to
    > login first. Of course, I know the username/password. My question is,
    > how to pass the username/password from perl?


    I don't think you can with LWP::Simple. However, the lwpcook
    documentation shows this example, using LWP::UserAgent ...
    http://search.cpan.org/~gaas/libwww-perl-5.805/lwpcook.pod#ACCESS_TO_PROTECTED_DOCUMENTS

    Does that help?

    Paul Lalli
    Paul Lalli, Jun 12, 2006
    #2
    1. Advertising

  3. Guest

    Hi, Paul:

    Thank you very much. But as you will see, the following code does not
    work. Any help would be deeply appreciated.

    #! /Perl/bin/perl

    use LWP;

    my $url = 'http://www.dpgn.com/resultDetail.cfm?resultID=4543';
    my $ua = new LWP::UserAgent;
    my $req = new HTTP::Request (GET => $url);

    $req->authorization_basic ('jieacs', 'cancer');
    my $request = $ua->request ($req);
    die unless $request->code eq '200';
    my $page = $request->content;
    print $page;
    , Jun 12, 2006
    #3
  4. wrote:
    > I know that I can use Get("url") > output.txt to do that. But the
    > problem is that to reach to the "url", it needs a username/password to
    > login first. Of course, I know the username/password. My question is,
    > how to pass the username/password from perl?


    Depends how the server expects the username and password to be presented. It
    could be as post data, or cookies, or http basic auth or ...

    --
    David Dorward <http://blog.dorward.me.uk/> <http://dorward.me.uk/>
    Home is where the ~/.bashrc is
    David Dorward, Jun 12, 2006
    #4
  5. Ben Morrow Guest

    Quoth :
    > $req->authorization_basic (<username/pwd snipped>);


    Go and change your password. Now.
    Don't post passwords in public forums.

    Ben

    --
    I must not fear. Fear is the mind-killer. I will face my fear and
    I will let it pass through me. When the fear is gone there will be
    nothing. Only I will remain.
    Frank Herbert, 'Dune'
    Ben Morrow, Jun 12, 2006
    #5
  6. * <> wrote:
    > I am trying to use perl to extract the content of a webpage, such as an
    > online newspaper. For example, I might want to have the program to send
    > me any material with the word of "computer".
    >
    > I know that I can use Get("url") > output.txt to do that. But the
    > problem is that to reach to the "url", it needs a username/password to
    > login first. Of course, I know the username/password. My question is,
    > how to pass the username/password from perl?
    >
    > Thank you very much!
    >
    > JH
    >

    try WWW:Mechanize
    Quoting From http://search.cpan.org/dist/WWW-Mechanize/lib/WWW/Mechanize.pm
    {QUOTE}
    WWW::Mechanize, or Mech for short, helps you automate interaction with a
    website. It supports performing a sequence of page fetches including
    following links and submitting forms. Each fetched page is parsed and
    its links and forms are extracted. A link or a form can be selected,
    form fields can be filled and the next page can be fetched. Mech also
    stores a history of the URLs you've visited, which can be queried and
    revisited.
    {/QUOTE}

    Regards
    Jahagirdar Vijayvithal S
    --
    "Tell your friend a lie. If he keeps it secret, then tell him the truth." - Portuguese Proverb
    Jahagirdar .V.S 91-80-25099129(O) 91-80-28540394(R)
    IC Design Engineer RGES-WLAN,
    Texas Instruments (India) Ltd.
    Jahagirdar Vijayvithal S, Jun 13, 2006
    #6
  7. Paul Lalli Guest

    wrote:
    > Hi, Paul:
    >
    > Thank you very much.


    You're welcome. For what, may I ask? Please quote context when
    posting a reply.

    > But as you will see, the following code does not
    > work.


    The code works perfectly for what it was designed to do. The problem
    is that you incorrectly stated your problem description.

    > Any help would be deeply appreciated.


    You incorrectly stated your problem description. You said that you had
    a URL, but that a user/password was required to access that URL. That
    turns out to not be the case. Instead, you have a URL whose page
    contains a form. That form has user and password fields. Submission
    of that form will take you to the actual URL you wanted to go to.

    Use WWW::Mechanize.

    Paul Lalli
    Paul Lalli, Jun 13, 2006
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. john brown
    Replies:
    2
    Views:
    918
    Christopher Shatto
    Aug 3, 2004
  2. KDawg44
    Replies:
    2
    Views:
    227
  3. AAaron123
    Replies:
    2
    Views:
    2,107
    AAaron123
    Jan 16, 2009
  4. AAaron123
    Replies:
    1
    Views:
    1,321
    Oriane
    Jan 16, 2009
  5. Shailesh Patel
    Replies:
    0
    Views:
    439
    Shailesh Patel
    Nov 8, 2006
Loading...

Share This Page