How to read from URL line-wise?

Discussion in 'Perl Misc' started by kj, May 6, 2014.

  1. kj

    kj Guest

    First, profuse apologies for the original posting of this query,
    which had gibberish ("sdfsdfsf") in the Subject: line. I goofed.

    ---

    I'm looking for the "moral equivalent" of the (fictitious) `openremote`
    function below:

    my $handle = openremote( 'http://some.domain.org/huge.tsv' ) or die $!;
    while ( <$handle> ) {
    chomp;
    # etc.
    # do stuff with $_
    }
    close $handle;

    IOW, I'm looking for a way to open a read handle to a remote file
    so that I can read from it *line-by-line*. (Typically this file
    will be larger than I want to read all at once into memory. IOW,
    I want to avoid solutions based on stuffing the value returned into
    LWP::Simple::get into an IO::String.)

    I'm sure this is really basic stuff, but I have not been able to
    find it after a lot of searching.

    TIA!

    kj
    kj, May 6, 2014
    #1
    1. Advertising

  2. On 2014-05-06 22:12, Jürgen Exner <> wrote:
    > kj <> wrote:
    >>I'm looking for the "moral equivalent" of the (fictitious) `openremote`
    >>function below:
    >>
    >> my $handle = openremote( 'http://some.domain.org/huge.tsv' ) or die $!;

    > [...]
    >>IOW, I'm looking for a way to open a read handle to a remote file
    >>so that I can read from it *line-by-line*. (Typically this file
    >>will be larger than I want to read all at once into memory. IOW,
    >>I want to avoid solutions based on stuffing the value returned into
    >>LWP::Simple::get into an IO::String.)
    >>
    >>I'm sure this is really basic stuff, but I have not been able to
    >>find it after a lot of searching.

    >
    > I very much doubt that HTTP supports such a line-by-line retrieval.


    Not line-by-line (files don't support that either on most platforms),
    but byte-ranges are supported by HTTP/1.1. Whether the server supports
    it for the file is another question, but most servers do for files
    stored in the file system (but not dynamically created content).

    But I associate "line-by-line" with sequential access, not random
    access, and you are of course always free to process the response in
    little chunks as you receive it (see "Handlers in LWP::UserAgent for a
    standard way of doing this).

    hp


    --
    _ | Peter J. Holzer | Fluch der elektronischen Textverarbeitung:
    |_|_) | | Man feilt solange an seinen Text um, bis
    | | | | die Satzbestandteile des Satzes nicht mehr
    __/ | http://www.hjp.at/ | zusammenpaßt. -- Ralph Babel
    Peter J. Holzer, May 7, 2014
    #2
    1. Advertising

  3. kj

    kj Guest

    In <> Jürgen Exner <> writes:

    >kj <> wrote:
    >>I'm looking for the "moral equivalent" of the (fictitious) `openremote`
    >>function below:
    >>
    >> my $handle = openremote( 'http://some.domain.org/huge.tsv' ) or die $!;

    >[...]
    >>IOW, I'm looking for a way to open a read handle to a remote file
    >>so that I can read from it *line-by-line*. (Typically this file
    >>will be larger than I want to read all at once into memory. IOW,
    >>I want to avoid solutions based on stuffing the value returned into
    >>LWP::Simple::get into an IO::String.)
    >>
    >>I'm sure this is really basic stuff, but I have not been able to
    >>find it after a lot of searching.


    >I very much doubt that HTTP supports such a line-by-line retrieval. And
    >if line-by-line is not supported by the underlying protocol, then at the
    >very best you can only hope for a local simulation, but at that point
    >the resource has been retrived in full.already.


    I was under the impression that HTTP supported incremental downloads
    (some fixed number of bytes at a time); if so, a client could easily
    implement a line-by-line interface to that stream... But now I
    think I need to do some homework and review HTTP.

    Thanks!

    kj
    kj, May 7, 2014
    #3
  4. kj

    kj Guest

    In <> "Peter J. Holzer" <> writes:

    >...you are of course always free to process the response in
    >little chunks as you receive it (see "Handlers in LWP::UserAgent for a
    >standard way of doing this).


    Thanks for this pointer! This approaches what I'm after. I'd
    hoped to find a package (in some obscure corner of LWP) that already
    implemented this line-oriented interface to the stream, but I guess
    I'll have to write it myself. (Conceptually it's not a hard thing
    to do, but IME *robust* implementations of even simple tasks like
    this one can take a lot more work than one would expect.)

    kj
    kj, May 7, 2014
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?U3Jpbmk=?=

    Site wise url access

    =?Utf-8?B?U3Jpbmk=?=, May 19, 2004, in forum: ASP .Net
    Replies:
    1
    Views:
    366
    Kevin Spencer
    May 19, 2004
  2. Hugo
    Replies:
    10
    Views:
    1,305
    Matt Humphrey
    Oct 18, 2004
  3. Enrique Cruiz

    Row-wise vs. column-wise image processing

    Enrique Cruiz, Jan 25, 2007, in forum: C Programming
    Replies:
    10
    Views:
    718
    christian.bau
    Jan 26, 2007
  4. kaushikshome
    Replies:
    4
    Views:
    762
    kaushikshome
    Sep 10, 2006
  5. Enrique Cruiz
    Replies:
    5
    Views:
    416
    Jim Langston
    Jan 25, 2007
Loading...

Share This Page