https with LWP::UserAgent

Discussion in 'Perl Misc' started by one man army, Sep 21, 2005.

  1. one man army

    one man army Guest

    Hi All-

    I am an infrequent user of Perl, but have succeeded in a few endeavors.
    I have a script that reads data from a series of web pages. But now I
    want to modify it to handle an HTTPS page.

    I looked briefly in UserAgent.pm, most of the code is beyond my skill
    level. I found the lines:

    use Carp ();

    if ($ENV{PERL_LWP_USE_HTTP_10}) {
    require LWP::protocol::http10;
    LWP::protocol::implementor('http', 'LWP::protocol::http10');
    eval {
    require LWP::protocol::https10;
    LWP::protocol::implementor('https', 'LWP::protocol::https10');
    };
    }

    And there is an option described, $ua->protocols_allowed(). So I added
    the following line after init'ing my $ua

    $ua->protocols_allowed( [ 'http', 'https'] ); # to set


    Alas, still an error connecting to the page. The very same urls pasted
    in to Firefox result in no error.

    What to do?

    thanks much
     
    one man army, Sep 21, 2005
    #1
    1. Advertising

  2. On Wed, 21 Sep 2005 17:47:09 +0000, one man army wrote:

    > Hi All-
    >
    > I am an infrequent user of Perl, but have succeeded in a few endeavors.
    > I have a script that reads data from a series of web pages. But now I
    > want to modify it to handle an HTTPS page.
    >
    > I looked briefly in UserAgent.pm


    That way lies pain and death. Well, pain anyway. You haven't specified
    your OS, so I'll say how it is for mine, Linux.

    Using LWP with the https protocol is as simple as

    my $ua = LWP::UserAgent->new;
    my $req = HTTP::Request->new(GET => 'https://secure.tld');
    my $res = $ua->request($req);

    The key on *nix is to have some SSL libraries installed to do the hard
    work. LWP itself doesn't have any SSL code.

    If LWP finds a 3rd party SSL library, such as openssl, when it is
    installed then it will be capable of fetching from https URLs. If not,
    then you will see an error like 'unknown protocol' or similar.

    This could well be different on winders et al.

    HTH
    Rich
     
    Richard Gration, Sep 22, 2005
    #2
    1. Advertising

  3. one man army

    one man army Guest

    Aha! I am using Perl 5.8.1 on the FreeBSD derivative, Mac OS X/Darwin.
    How do I install the SSL Library? I have a folder called /Library/Perl,
    then :
    5.8.1
    File
    lwptut.pod
    lwpcook.pod
    Regexp
    Net
    LWP.pm
    LWP
    WWW
    URI.pm
    URI
    Bundle
    HTTP
    HTML
    darwin-thread-multi-2level
    HTML-LinkExtractor-0.13
    HTML-Tagset-3.04
    HTML-Parser-3.45
    URIC-2.02
    libwww-perl-5.69
    Regexp-Common-2.120
    libnet-1.19
    URI-1.35


    In article <>,
    Richard Gration <> wrote:

    > On Wed, 21 Sep 2005 17:47:09 +0000, one man army wrote:
    >
    > > Hi All-
    > >
    > > I am an infrequent user of Perl, but have succeeded in a few endeavors.
    > > I have a script that reads data from a series of web pages. But now I
    > > want to modify it to handle an HTTPS page.
    > >
    > > I looked briefly in UserAgent.pm

    >
    > That way lies pain and death. Well, pain anyway. You haven't specified
    > your OS, so I'll say how it is for mine, Linux.
    >
    > Using LWP with the https protocol is as simple as
    >
    > my $ua = LWP::UserAgent->new;
    > my $req = HTTP::Request->new(GET => 'https://secure.tld');
    > my $res = $ua->request($req);
    >
    > The key on *nix is to have some SSL libraries installed to do the hard
    > work. LWP itself doesn't have any SSL code.
    >
    > If LWP finds a 3rd party SSL library, such as openssl, when it is
    > installed then it will be capable of fetching from https URLs. If not,
    > then you will see an error like 'unknown protocol' or similar.
    >
    > This could well be different on winders et al.
    >
    > HTH
    > Rich
     
    one man army, Sep 22, 2005
    #3
  4. On Thu, 22 Sep 2005 14:48:32 +0000, one man army wrote:
    >
    > In article <>,
    > Richard Gration <> wrote:
    >
    >> On Wed, 21 Sep 2005 17:47:09 +0000, one man army wrote:
    >>
    >> > Hi All-
    >> >
    >> > I am an infrequent user of Perl, but have succeeded in a few endeavors.
    >> > I have a script that reads data from a series of web pages. But now I
    >> > want to modify it to handle an HTTPS page.
    >> >
    >> > I looked briefly in UserAgent.pm

    >>
    >> That way lies pain and death. Well, pain anyway. You haven't specified
    >> your OS, so I'll say how it is for mine, Linux.
    >>
    >> Using LWP with the https protocol is as simple as
    >>
    >> my $ua = LWP::UserAgent->new;
    >> my $req = HTTP::Request->new(GET => 'https://secure.tld');
    >> my $res = $ua->request($req);
    >>
    >> The key on *nix is to have some SSL libraries installed to do the hard
    >> work. LWP itself doesn't have any SSL code.
    >>
    >> If LWP finds a 3rd party SSL library, such as openssl, when it is
    >> installed then it will be capable of fetching from https URLs. If not,
    >> then you will see an error like 'unknown protocol' or similar.
    >>
    >> This could well be different on winders et al.
    >>
    >> HTH
    >> Rich


    > Aha! I am using Perl 5.8.1 on the FreeBSD derivative, Mac OS X/Darwin.
    > How do I install the SSL Library? I have a folder called /Library/Perl,
    > then :
    > 5.8.1
    > File
    > lwptut.pod
    > lwpcook.pod
    > Regexp
    > Net
    > LWP.pm
    > LWP
    > WWW
    > URI.pm
    > URI
    > Bundle
    > HTTP
    > HTML
    > darwin-thread-multi-2level
    > HTML-LinkExtractor-0.13
    > HTML-Tagset-3.04
    > HTML-Parser-3.45
    > URIC-2.02
    > libwww-perl-5.69
    > Regexp-Common-2.120
    > libnet-1.19
    > URI-1.35


    [ Please don't top post ]

    SSL isn't part of perl. It's a thing in and of itself. I know next to
    nothing about Macs so I can't tell you specifically how to do what you
    need to but I would:

    1. Find out if you have SSL libraries. On my system I have
    /usr/lib/libssl.*

    2. If not, then install openssl or equivalent.

    3. Once you have ssl on your machine, the most likely point of failure is
    that libwww-perl doesn't know it's there. Once you are certain you have an
    ssl implementation, reinstall libwww-perl. Make sure that it notices ssl
    during configuration / installation. You might have to give some options
    when configuring to specify the location of your ssl.

    If none of this helps, then I'm afraid I'm at a bit of a loss.

    Rich
     
    Richard Gration, Sep 22, 2005
    #4
  5. one man army

    one man army Guest

    I found Crypy::SSLeay on CPAN. I brought it down, and in the readme it
    says that you must have OpenSSL installed before compiling/building.

    I searched for openSSL (case-insensitive) anywhere on my system, but
    found nothing.

    How to install OpenSSL on MacOS 10.3? thanks

    > Aha! I am using Perl 5.8.1 on the FreeBSD derivative, Mac OS X/Darwin.
    > How do I install the SSL Library? I have a folder called /Library/Perl,
    > then :
    > 5.8.1
    > File
    > lwptut.pod
    > lwpcook.pod
    > Regexp
    > Net
    > LWP.pm
    > LWP
    > WWW
    > URI.pm
    > URI
    > Bundle
    > HTTP
    > HTML
    > darwin-thread-multi-2level
    > HTML-LinkExtractor-0.13
    > HTML-Tagset-3.04
    > HTML-Parser-3.45
    > URIC-2.02
    > libwww-perl-5.69
    > Regexp-Common-2.120
    > libnet-1.19
    > URI-1.35
    >
    >
    > In article <>,
    > Richard Gration <> wrote:
    >
    > > On Wed, 21 Sep 2005 17:47:09 +0000, one man army wrote:
    > >
    > > > Hi All-
    > > >
    > > > I am an infrequent user of Perl, but have succeeded in a few endeavors.
    > > > I have a script that reads data from a series of web pages. But now I
    > > > want to modify it to handle an HTTPS page.
    > > >
    > > > I looked briefly in UserAgent.pm

    > >
    > > That way lies pain and death. Well, pain anyway. You haven't specified
    > > your OS, so I'll say how it is for mine, Linux.
    > >
    > > Using LWP with the https protocol is as simple as
    > >
    > > my $ua = LWP::UserAgent->new;
    > > my $req = HTTP::Request->new(GET => 'https://secure.tld');
    > > my $res = $ua->request($req);
    > >
    > > The key on *nix is to have some SSL libraries installed to do the hard
    > > work. LWP itself doesn't have any SSL code.
    > >
    > > If LWP finds a 3rd party SSL library, such as openssl, when it is
    > > installed then it will be capable of fetching from https URLs. If not,
    > > then you will see an error like 'unknown protocol' or similar.
    > >
    > > This could well be different on winders et al.
    > >
    > > HTH
    > > Rich
     
    one man army, Sep 22, 2005
    #5
  6. On Thu, 22 Sep 2005 16:20:31 +0000, one man army wrote:

    > How to install OpenSSL on MacOS 10.3? thanks


    This thread has now lost even a tenuous connection to perl, I'm afraid ;-)

    You should try asking in a MacOS newsgroup.

    Cheers
    Rich
     
    Richard Gration, Sep 22, 2005
    #6
  7. one man army <> writes:

    > I searched for openSSL (case-insensitive) anywhere on my system, but
    > found nothing.
    >
    > How to install OpenSSL on MacOS 10.3?


    You don't need to - it's included with the OS. Honestly, couldn't you have at
    least *tried* to install the module before assuming it would fail?

    sherm--

    --
    Cocoa programming in Perl: http://camelbones.sourceforge.net
    Hire me! My resume: http://www.dot-app.org
     
    Sherm Pendley, Sep 22, 2005
    #7
  8. one man army <> writes:

    > Aha! I am using Perl 5.8.1 on the FreeBSD derivative, Mac OS X/Darwin.
    > How do I install the SSL Library?


    Same way you install it on any other *nix, by following the instructions found
    in "perldoc perlmodinstall".

    I'm not being snide - the instructions there are far better and more detailed
    than any usnet posting could be. Be sure to follow the Unix instructions - the
    Mac instructions are for MacPerl running on MacOS 9 and earlier.

    sherm--

    --
    Cocoa programming in Perl: http://camelbones.sourceforge.net
    Hire me! My resume: http://www.dot-app.org
     
    Sherm Pendley, Sep 22, 2005
    #8
  9. one man army

    Babacio Guest

    one man army <> writes:

    > I found Crypy::SSLeay on CPAN. I brought it down, and in the readme it
    > says that you must have OpenSSL installed before compiling/building.
    >
    > I searched for openSSL (case-insensitive) anywhere on my system, but
    > found nothing.
    >
    > How to install OpenSSL on MacOS 10.3? thanks


    Have a look on darwinports. There is a port 'openssl' which sure gives
    you all what you need. There is even a package named p5-net-ssleay
    which installs Net::SSLeay, which may be of interest.

    --
    Bé erre hue ixe eu elle, Bruxelles.
     
    Babacio, Sep 22, 2005
    #9
  10. one man army

    one man army Guest

    > > On Wed, 21 Sep 2005 17:47:09 +0000, one man army wrote:
    > >
    > > > Hi All-
    > > >
    > > > I am an infrequent user of Perl, but have succeeded in a few endeavors.
    > > > I have a script that reads data from a series of web pages. But now I
    > > > want to modify it to handle an HTTPS page.
    > > >


    SUMMARY:
    With the addition of Perl Module Crypt-SSLeay-0.51, LWP will use
    OpenSSL to provide HTTPS (perl 5.8 / Mac OS X Darwin).

    OpenSSL is built in to Mac OS X. The Finder's find does not look in
    system libraries by default, so it will not find OpenSSL. /usr/lib has a
    few versions of ssl with various encoded names.

    I built Crypt::SSLeay with only a minor error (the man files don't get
    put into a place the make expects)

    I rebuilt libwww-perl just in case it needs to know SSLeay is present.
    The makefile seems confused about the Mac's .DS_Store files and
    generates a hard error. But the library runs and code that uses it runs.

    <someone should probably add a check for .DS_Store files in the make,
    since there are so many Darwin machines out there now>

    All in all, CPAN is the best feature of the Perl library system IMHO.
    The system of pieces being built is not yet as bullet-proof as it could
    be, since so many smart people use this stuff all the time.

    thanks for <most of> the replies
     
    one man army, Sep 22, 2005
    #10
  11. one man army

    Babacio Guest

    one man army.

    > I rebuilt libwww-perl just in case it needs to know SSLeay is present.
    > The makefile seems confused about the Mac's .DS_Store files and
    > generates a hard error. But the library runs and code that uses it runs.
    >
    > <someone should probably add a check for .DS_Store files in the make,
    > since there are so many Darwin machines out there now>


    Well, one of my favorite things is
    find . -name '.DS_Store' -exec rm -i {} \;
    ....
    --
    Bé erre hue ixe eu elle, Bruxelles.
     
    Babacio, Sep 23, 2005
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. La Jesus
    Replies:
    9
    Views:
    1,356
    Gunnar Hjalmarsson
    Oct 27, 2003
  2. Chandra

    Problem with LWP::USERAGENT

    Chandra, Feb 25, 2004, in forum: Perl
    Replies:
    0
    Views:
    520
    Chandra
    Feb 25, 2004
  3. Vinay Gupta
    Replies:
    2
    Views:
    1,139
    Michael Schlenker
    Jul 27, 2004
  4. jarkun
    Replies:
    1
    Views:
    130
    Charles DeRykus
    Sep 5, 2003
  5. Replies:
    7
    Views:
    391
Loading...

Share This Page