How to read from URL line-wise?

Discussion in 'Perl Misc' started by kj, May 6, 2014.

  1. kj

    kj Guest

    First, profuse apologies for the original posting of this query,
    which had gibberish ("sdfsdfsf") in the Subject: line. I goofed.

    ---

    I'm looking for the "moral equivalent" of the (fictitious) `openremote`
    function below:

    my $handle = openremote( 'http://some.domain.org/huge.tsv' ) or die $!;
    while ( <$handle> ) {
    chomp;
    # etc.
    # do stuff with $_
    }
    close $handle;

    IOW, I'm looking for a way to open a read handle to a remote file
    so that I can read from it *line-by-line*. (Typically this file
    will be larger than I want to read all at once into memory. IOW,
    I want to avoid solutions based on stuffing the value returned into
    LWP::Simple::get into an IO::String.)

    I'm sure this is really basic stuff, but I have not been able to
    find it after a lot of searching.

    TIA!

    kj
     
    kj, May 6, 2014
    #1
    1. Advertisements

  2. Not line-by-line (files don't support that either on most platforms),
    but byte-ranges are supported by HTTP/1.1. Whether the server supports
    it for the file is another question, but most servers do for files
    stored in the file system (but not dynamically created content).

    But I associate "line-by-line" with sequential access, not random
    access, and you are of course always free to process the response in
    little chunks as you receive it (see "Handlers in LWP::UserAgent for a
    standard way of doing this).

    hp
     
    Peter J. Holzer, May 7, 2014
    #2
    1. Advertisements

  3. kj

    kj Guest

    I was under the impression that HTTP supported incremental downloads
    (some fixed number of bytes at a time); if so, a client could easily
    implement a line-by-line interface to that stream... But now I
    think I need to do some homework and review HTTP.

    Thanks!

    kj
     
    kj, May 7, 2014
    #3
  4. kj

    kj Guest

    Thanks for this pointer! This approaches what I'm after. I'd
    hoped to find a package (in some obscure corner of LWP) that already
    implemented this line-oriented interface to the stream, but I guess
    I'll have to write it myself. (Conceptually it's not a hard thing
    to do, but IME *robust* implementations of even simple tasks like
    this one can take a lot more work than one would expect.)

    kj
     
    kj, May 7, 2014
    #4
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.