Help: Open an URL an keep reading from it, non-blocking

[John Bokma]

A web based chat uses a method I call infinite loading. Basically the
browser opens the url, but it never stops, like it is downloading an
infinite long page. The chat system just keeps adding new lines.

Is there a way to do this in Perl. I mean, is there a module that let's me
see if there is data available, and read it, and move on in a loop?

I can write it, but I don't like reinventing wheels.

Thanks,
John

 

[Uwe Disch]

Basically the browser opens the url, but it never stops, like it is
downloading an infinite long page. The chat system just keeps adding
new lines.

This would be a good starting point if you are running Apache and
Perl: http://httpd.apache.org/docs/misc/rewriteguide.html

See chapter about "Document With Autorefresh".


Uwe

 

[John Bokma]

This would be a good starting point if you are running Apache and
Perl: http://httpd.apache.org/docs/misc/rewriteguide.html

See chapter about "Document With Autorefresh".

Uhm, it has nothing to do with Apache, nor rewriting. I want to read a
infinite document and printing each line when it comes in. Can't use the
HTTP::Request stuff, since it reads all (which never finishes), and then
returns.

 

[Tassilo v. Parseval]

Also sprach John Bokma:

A web based chat uses a method I call infinite loading. Basically the
browser opens the url, but it never stops, like it is downloading an
infinite long page. The chat system just keeps adding new lines.

Is there a way to do this in Perl. I mean, is there a module that let's me
see if there is data available, and read it, and move on in a loop?

IO::Select (or maybe IO:oll) is commonly used for that. The purpose of
the select(2)/poll(2) system calls is to watch file descriptors for
certain events. Becoming readable or writable is such an event.

On a higher level, you could use one of the event modules as provided by
the CPAN: Event, Event::Lib or even POE.

I can write it, but I don't like reinventing wheels.

It's not so easy to rewrite this. It's more convenient to use one of the
mentioned notification mechanisms which are provided by your kernel.

Tassilo

 

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
John Bokma

online.de:

Uhm, it has nothing to do with Apache, nor rewriting. I want to read a
infinite document and printing each line when it comes in. Can't use the
HTTP::Request stuff, since it reads all (which never finishes), and then
returns.

My copy of lwp-rget supports --progress option. This clearly shows
that your analysis of "HTTP::Request stuff" is faulty. Read the docs
again.

Hope this helps,
Ilya

 

[gisle]

Is there a way to do this in Perl. I mean, is there a module that let's me
see if there is data available, and read it, and move on in a loop?

LWP give you a callback that will be called each time data is received
from the server. Try something like this:

$ua = LWP::UserAgent->new;
$ua->get( $url,
':content_cb' => sub {
my($data, $response) = @_;
print $data;
})

 

[John Bokma]

[A complimentary Cc of this posting was sent to
John Bokma

news:[email protected] online.de:

Uhm, it has nothing to do with Apache, nor rewriting. I want to read
a infinite document and printing each line when it comes in. Can't
use the HTTP::Request stuff, since it reads all (which never
finishes), and then returns.

My copy of lwp-rget supports --progress option. This clearly shows
that your analysis of "HTTP::Request stuff" is faulty. Read the docs
again.

I did, and I am right, and you are right. Thanks for the pointer, grmbl, I
*should* have read the lwp-cookbook, heading LARGE DOCUMENTS :-(

 

[Sandman]

A web based chat uses a method I call infinite loading. Basically the
browser opens the url, but it never stops, like it is downloading an
infinite long page. The chat system just keeps adding new lines.

Is there a way to do this in Perl. I mean, is there a module that let's me
see if there is data available, and read it, and move on in a loop?

I can write it, but I don't like reinventing wheels.

I once wrote a IRC bot using only IO::Socket, that connected to an IRC server,
logged on and executed command when specific strings were sent from the server.
Sounds like all you want to do is connect to a web server on port 80, request a
specific URI and then just read the data the server sends you.

It should be noted that I was a -complete- novice when I wrote that script (as
if THAT has changed significantly), so using IO::Socket might have been a
really bad idea, but it worked.

It looked something like this (bits and parts of the app, with what I remember
to be the things that made it work, this won't work since I snipped it heavily):

------------------------
#!/usr/bin/perl
# no strict or warnings here, suppress them all! Screw the Real Way!
# lots of setup stuff snipped, like $server declarations and such.

$|++; # Don't buffer output
use IO::Socket;
$host = IO::Socket::INET->new(
Proto => "tcp",
PeerAddr => "$server",
PeerPort => "$port",
) or { error("Cannot connect to host: $server \n$!") };

$host->autoflush(1);
# This, uh.. flushes every line to output...? I don't remmeber
$SIG{ALRM}='timeout'; # When alarm triggers, run sub 'timeout'.

# If we didn't die, the connection is made
print $host "NICK $nickname\n";
print $host "USER sandman sandman.net $server\n";

alarm 30; # trigger the alarm in 30 seconds.
while (<$host>) {
last if / 376 /; # We recieved what we wanted to recieve at this point
}
alarm 0; # reset the alarm if we completed within 30 seconds

# Then next block of commands, working pretty much the same way

sub timeout {
error("No response in 30 seconds, timeout.");
}
------------------------

So, I had a series of while loops to wait for the correct answer from the irc
server and so on. There are probably a bunch of oldies that will twist in agony
over the above code, but what the hell - maybe it helps you in some way? Oh,
the error sub just printed the error and then quit the program, nothing fancy.

If you want I could probably code a working example of talking to an IRC
server, and use strict and warnings.