[perl-python] get web page programatically

Discussion in 'Perl Misc' started by Xah Lee, Feb 4, 2005.

  1. Xah Lee

    Xah Lee Guest

    # -*- coding: utf-8 -*-
    # Python

    # suppose you want to fetch a webpage.
    from urllib import urlopen
    print
    urlopen('http://xahlee.org/Periodic_dosage_dir/_p2/russell-lecture.html').read()

    # note the line
    # from <library_name> import <function_name1,function_name2...>
    # it reads the library and import the function name
    # to see available functions in a module one can use "dir"
    # import urllib; print dir(urllib)

    # for more about this module import syntax, see
    # http://python.org/doc/tut/node8.html

    #---------------------
    # sometimes in working with html pages, you need to creat links
    # In url, some chars need to be encoded.
    # the "quote" function does it. "unquote" function reverses it. Very
    nice.

    from urllib import quote
    print quote("~joe's home page")
    print 'http://www.google.com/search?q=' + quote("ménage à trois")
    # (rely on the French to teach us interesting words)

    # for more about the urllib module, see
    # http://python.org/doc/lib/module-urllib.html

    ----------------------------
    in perl, it's messy as usual. Long story short the simplest way is to
    use the perl program HEAD or GET in /usr/bin or /usr/local/bin. When
    one of the networking module is installed, perl contaminate your bin
    dirs with these programs. In the unix shell, try
    GET 'http://yahoo.com/'
    should do the job. HEAD is similar for http
    head. (assuming they are installed.)

    if you need more complexty, perl has LWP::Simple and LWP::UserAgent to
    begin with. (there are a host of spaghetti others) Both of these needs
    to be installed extra. Perhaps consult your sys admin. The last time i
    used them was some 2 years ago, so the following code is untested, but
    should be it. I don't recall which one can't do what. Your milage may
    vary.

    use strict;
    # use LWP::Simple;
    use LWP::UserAgent;
    my $ua = new LWP::UserAgent;
    $ua->timeout(120);
    my $url='http://yahoo.com/';
    my $request = new HTTP::Request('GET', $url);
    my $response = $ua->request($request);
    my $content = $response->content();
    print $content;
    __END__

    # note the above perl code. In many perl codes, they sport the Object
    Oriented syntax, often concomitantly with a normal syntax version as
    well.

    ----------------
    this post is from the perl-python a-day mailing list. Please see
    http://xahlee.org/perl-python/python.html

    Xah

    http://xahlee.org/PageTwo_dir/more.html
    Xah Lee, Feb 4, 2005
    #1
    1. Advertising

  2. Xah Lee wrote:

    <snip>

    Just the standard warnings for any novices unfamiliar with Mr. Lee.
    Mr. Lee's posts are regularly riddled with severe errors (I found
    the assertion that LWP::Simple and LWP::UserAgent aren't part of
    the standard base perl install a particularly amusing one in this
    particular post). Please be advised that you should get your
    perl information from accurate sources. http://learn.perl.org
    is an excellent place to start, with pointers to excellent Perl
    books and even some readable for free online (notably Beginning
    Perl).
    --
    Christopher Mattern

    "Which one you figure tracked us?"
    "The ugly one, sir."
    "...Could you be more specific?"
    Chris Mattern, Feb 4, 2005
    #2
    1. Advertising

  3. Chris Mattern wrote:
    > Just the standard warnings for any novices unfamiliar with Mr. Lee.
    > Mr. Lee's posts are regularly riddled with severe errors


    Concur.

    > (I found the assertion that LWP::Simple and LWP::UserAgent aren't
    > part of the standard base perl install a particularly amusing one in
    > this particular post).


    I'm not sure about that, though. I thought that libwww-perl was not a
    core package.

    --
    Gunnar Hjalmarsson
    Email: http://www.gunnar.cc/cgi-bin/contact.pl
    Gunnar Hjalmarsson, Feb 4, 2005
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. M-
    Replies:
    1
    Views:
    515
  2. francois

    get the form id programatically

    francois, Jun 17, 2004, in forum: ASP .Net
    Replies:
    1
    Views:
    371
    bloomfield
    Jun 17, 2004
  3. Xah Lee
    Replies:
    2
    Views:
    585
    Chris Mattern
    Feb 4, 2005
  4. Marcia
    Replies:
    0
    Views:
    161
    Marcia
    Jul 9, 2003
  5. Ezra Epstein
    Replies:
    0
    Views:
    121
    Ezra Epstein
    Sep 13, 2003
Loading...

Share This Page