Help: Open an URL an keep reading from it, non-blocking

Discussion in 'Perl Misc' started by John Bokma, Aug 31, 2004.

  1. John Bokma

    John Bokma Guest

    A web based chat uses a method I call infinite loading. Basically the
    browser opens the url, but it never stops, like it is downloading an
    infinite long page. The chat system just keeps adding new lines.

    Is there a way to do this in Perl. I mean, is there a module that let's me
    see if there is data available, and read it, and move on in a loop?

    I can write it, but I don't like reinventing wheels.

    Thanks,
    John

    --
    John MexIT: http://johnbokma.com/mexit/
    personal page: http://johnbokma.com/
    Experienced programmer available: http://castleamber.com/
    Happy Customers: http://castleamber.com/testimonials.html
     
    John Bokma, Aug 31, 2004
    #1
    1. Advertising

  2. John Bokma

    Uwe Disch Guest

    Re: Open an URL an keep reading from it, non-blocking

    "John Bokma" <> schrieb im Newsbeitrag
    news:Xns9556A97A6215castleamber@130.133.1.4...
    > Basically the browser opens the url, but it never stops, like it is
    > downloading an infinite long page. The chat system just keeps adding
    > new lines.


    This would be a good starting point if you are running Apache and
    Perl: http://httpd.apache.org/docs/misc/rewriteguide.html

    See chapter about "Document With Autorefresh".


    Uwe
    --
    Interchange Consulting: http://disch-online.de
    EIB-Temperatursensoren: http://disch-systems.de
     
    Uwe Disch, Sep 1, 2004
    #2
    1. Advertising

  3. John Bokma

    John Bokma Guest

    Re: Open an URL an keep reading from it, non-blocking

    "Uwe Disch" <> wrote in news:-
    online.de:

    >
    > "John Bokma" <> schrieb im Newsbeitrag
    > news:Xns9556A97A6215castleamber@130.133.1.4...
    >> Basically the browser opens the url, but it never stops, like it is
    >> downloading an infinite long page. The chat system just keeps adding
    >> new lines.

    >
    > This would be a good starting point if you are running Apache and
    > Perl: http://httpd.apache.org/docs/misc/rewriteguide.html
    >
    > See chapter about "Document With Autorefresh".


    Uhm, it has nothing to do with Apache, nor rewriting. I want to read a
    infinite document and printing each line when it comes in. Can't use the
    HTTP::Request stuff, since it reads all (which never finishes), and then
    returns.

    --
    John MexIT: http://johnbokma.com/mexit/
    personal page: http://johnbokma.com/
    Experienced programmer available: http://castleamber.com/
    Happy Customers: http://castleamber.com/testimonials.html
     
    John Bokma, Sep 1, 2004
    #3
  4. Also sprach John Bokma:

    > A web based chat uses a method I call infinite loading. Basically the
    > browser opens the url, but it never stops, like it is downloading an
    > infinite long page. The chat system just keeps adding new lines.
    >
    > Is there a way to do this in Perl. I mean, is there a module that let's me
    > see if there is data available, and read it, and move on in a loop?


    IO::Select (or maybe IO::poll) is commonly used for that. The purpose of
    the select(2)/poll(2) system calls is to watch file descriptors for
    certain events. Becoming readable or writable is such an event.

    On a higher level, you could use one of the event modules as provided by
    the CPAN: Event, Event::Lib or even POE.

    > I can write it, but I don't like reinventing wheels.


    It's not so easy to rewrite this. It's more convenient to use one of the
    mentioned notification mechanisms which are provided by your kernel.

    Tassilo
    --
    $_=q#",}])!JAPH!qq(tsuJ[{@"tnirp}3..0}_$;//::niam/s~=)]3[))_$-3(rellac(=_$({
    pam{rekcahbus})(rekcah{lrePbus})(lreP{rehtonabus})!JAPH!qq(rehtona{tsuJbus#;
    $_=reverse,s+(?<=sub).+q#q!'"qq.\t$&."'!#+sexisexiixesixeseg;y~\n~~dddd;eval
     
    Tassilo v. Parseval, Sep 1, 2004
    #4
  5. Re: Open an URL an keep reading from it, non-blocking

    [A complimentary Cc of this posting was sent to
    John Bokma
    <>], who wrote in article <Xns9556DF6614453castleamber@130.133.1.4>:
    > "Uwe Disch" <> wrote in news:-
    > online.de:


    > Uhm, it has nothing to do with Apache, nor rewriting. I want to read a
    > infinite document and printing each line when it comes in. Can't use the
    > HTTP::Request stuff, since it reads all (which never finishes), and then
    > returns.


    My copy of lwp-rget supports --progress option. This clearly shows
    that your analysis of "HTTP::Request stuff" is faulty. Read the docs
    again.

    Hope this helps,
    Ilya
     
    Ilya Zakharevich, Sep 1, 2004
    #5
  6. John Bokma

    Guest

    John Bokma <> writes:

    > Is there a way to do this in Perl. I mean, is there a module that let's me
    > see if there is data available, and read it, and move on in a loop?


    LWP give you a callback that will be called each time data is received
    from the server. Try something like this:

    $ua = LWP::UserAgent->new;
    $ua->get( $url,
    ':content_cb' => sub {
    my($data, $response) = @_;
    print $data;
    })

    --
    Gisle Aas
     
    , Sep 1, 2004
    #6
  7. John Bokma

    John Bokma Guest

    Re: Open an URL an keep reading from it, non-blocking

    Ilya Zakharevich <> wrote in
    news:ch3oa1$ue5$:

    > [A complimentary Cc of this posting was sent to
    > John Bokma
    > <>], who wrote in article
    > <Xns9556DF6614453castleamber@130.133.1.4>:
    >> "Uwe Disch" <> wrote in
    >> news:- online.de:

    >
    >> Uhm, it has nothing to do with Apache, nor rewriting. I want to read
    >> a infinite document and printing each line when it comes in. Can't
    >> use the HTTP::Request stuff, since it reads all (which never
    >> finishes), and then returns.

    >
    > My copy of lwp-rget supports --progress option. This clearly shows
    > that your analysis of "HTTP::Request stuff" is faulty. Read the docs
    > again.


    I did, and I am right, and you are right. Thanks for the pointer, grmbl, I
    *should* have read the lwp-cookbook, heading LARGE DOCUMENTS :-(

    --
    John MexIT: http://johnbokma.com/mexit/
    personal page: http://johnbokma.com/
    Experienced programmer available: http://castleamber.com/
    Happy Customers: http://castleamber.com/testimonials.html
     
    John Bokma, Sep 1, 2004
    #7
  8. John Bokma

    Sandman Guest

    In article <Xns9556A97A6215castleamber@130.133.1.4>,
    John Bokma <> wrote:

    > A web based chat uses a method I call infinite loading. Basically the
    > browser opens the url, but it never stops, like it is downloading an
    > infinite long page. The chat system just keeps adding new lines.
    >
    > Is there a way to do this in Perl. I mean, is there a module that let's me
    > see if there is data available, and read it, and move on in a loop?
    >
    > I can write it, but I don't like reinventing wheels.


    I once wrote a IRC bot using only IO::Socket, that connected to an IRC server,
    logged on and executed command when specific strings were sent from the server.
    Sounds like all you want to do is connect to a web server on port 80, request a
    specific URI and then just read the data the server sends you.

    It should be noted that I was a -complete- novice when I wrote that script (as
    if THAT has changed significantly), so using IO::Socket might have been a
    really bad idea, but it worked.

    It looked something like this (bits and parts of the app, with what I remember
    to be the things that made it work, this won't work since I snipped it heavily):

    ------------------------
    #!/usr/bin/perl
    # no strict or warnings here, suppress them all! Screw the Real Way! :)
    # lots of setup stuff snipped, like $server declarations and such.

    $|++; # Don't buffer output
    use IO::Socket;
    $host = IO::Socket::INET->new(
    Proto => "tcp",
    PeerAddr => "$server",
    PeerPort => "$port",
    ) or { error("Cannot connect to host: $server \n$!") };

    $host->autoflush(1);
    # This, uh.. flushes every line to output...? I don't remmeber
    $SIG{ALRM}='timeout'; # When alarm triggers, run sub 'timeout'.

    # If we didn't die, the connection is made
    print $host "NICK $nickname\n";
    print $host "USER sandman sandman.net $server\n";

    alarm 30; # trigger the alarm in 30 seconds.
    while (<$host>) {
    last if / 376 /; # We recieved what we wanted to recieve at this point
    }
    alarm 0; # reset the alarm if we completed within 30 seconds

    # Then next block of commands, working pretty much the same way

    sub timeout {
    error("No response in 30 seconds, timeout.");
    }
    ------------------------

    So, I had a series of while loops to wait for the correct answer from the irc
    server and so on. There are probably a bunch of oldies that will twist in agony
    over the above code, but what the hell - maybe it helps you in some way? Oh,
    the error sub just printed the error and then quit the program, nothing fancy.

    If you want I could probably code a working example of talking to an IRC
    server, and use strict and warnings.

    --
    Sandman[.net]
     
    Sandman, Sep 1, 2004
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Hendra Gunawan
    Replies:
    1
    Views:
    13,057
    Allan Herriman
    Apr 8, 2004
  2. Andre Kelmanson

    blocking i/o vs. non blocking i/o (performance)

    Andre Kelmanson, Oct 10, 2003, in forum: C Programming
    Replies:
    3
    Views:
    996
    Valentin Tihomirov
    Oct 12, 2003
  3. nukleus
    Replies:
    14
    Views:
    909
    Chris Uppal
    Jan 22, 2007
  4. Christian
    Replies:
    5
    Views:
    757
    Esmond Pitt
    Dec 2, 2007
  5. Serge Savoie
    Replies:
    4
    Views:
    291
    Serge Savoie
    Oct 1, 2008
Loading...

Share This Page