Download Files With a perl script

Discussion in 'Perl Misc' started by heli0s, Aug 4, 2005.

  1. heli0s

    heli0s Guest

    Hi,

    I'm having a bit of a problem with a script I'm writing. I want the script
    to read in a txt file with a list of files to download and then go through
    the list of files to get them from the net.

    it should be something like this:

    use LWP::Simple;

    open (FILE,"all.m3u");
    @file = <FILE>;
    close FILE;

    foreach $line (@file){
    get ( $line );
    }

    As seen in the above code, i've been looking at LWP::Simple, but what i
    have here is no good. Maybe i'm doing something wrong but i havent figured
    it out yet.

    Greets Christiaan
     
    heli0s, Aug 4, 2005
    #1
    1. Advertising

  2. heli0s

    Paul Lalli Guest

    heli0s wrote:
    > Hi,
    >
    > I'm having a bit of a problem with a script I'm writing. I want the script
    > to read in a txt file with a list of files to download and then go through
    > the list of files to get them from the net.
    >
    > it should be something like this:
    >
    > use LWP::Simple;


    Please always put these lines in your script:
    use strict;
    use warnings;

    > open (FILE,"all.m3u");


    Always check the return value of open()
    open my $file, '<', 'all.m3u' or die "Cannot open file: $!";

    > @file = <FILE>;


    Why are you storing all lines of this file in memory? Why not process
    them one at a time?

    > close FILE;
    >
    > foreach $line (@file){


    Each $line contains a "\n" on the end. Do you want that newline? I'm
    betting not.
    perldoc -f chomp

    > get ( $line );


    >From perldoc LWP::Simple, get() returns the data fetched. Here, you're

    calling get() in a void context (ie, not assigning its return value to
    any variable), so you're throwing the data away.

    You're also not checking to see if this is actually succeeding or not.

    my $html = get ($line) or die "Could not fetch from $line: $!";

    (note: I don't know for a fact that $! contains anything useful if
    get() fails)

    > }
    >
    > As seen in the above code, i've been looking at LWP::Simple, but what i
    > have here is no good. Maybe i'm doing something wrong but i havent figured
    > it out yet.


    Please fix up your code, making it strict- and warnings-compliant, and
    check the return values of your calls, and then post the results here
    if you still haven't gotten it figured out.

    Paul Lalli
     
    Paul Lalli, Aug 4, 2005
    #2
    1. Advertising

  3. heli0s

    Sisyphus Guest

    "heli0s" <> wrote in message

    >
    > use LWP::Simple;


    use warnings;

    >
    > open (FILE,"all.m3u");
    > @file = <FILE>;
    > close FILE;
    >
    > foreach $line (@file){


    chomp($line);

    > get ( $line );
    > }
    >


    Hth.

    Cheers,
    Rob
     
    Sisyphus, Aug 4, 2005
    #3
  4. heli0s

    Guest

    heli0s <> wrote:
    > I'm having a bit of a problem with a script I'm writing. I want the script
    > to read in a txt file with a list of files to download and then go through
    > the list of files to get them from the net.
    >
    > it should be something like this:
    >
    > use LWP::Simple;
    >
    > open (FILE,"all.m3u");
    > @file = <FILE>;
    > close FILE;
    >
    > foreach $line (@file){
    > get ( $line );
    > }


    > As seen in the above code, i've been looking at LWP::Simple, but what i
    > have here is no good. Maybe i'm doing something wrong but i havent figured
    > it out yet.


    Do you have a sample of a couple of lines from all.m3u?

    Axel
     
    , Aug 4, 2005
    #4
  5. heli0s

    heli0s Guest

    On Thu, 04 Aug 2005 17:23:45 +0000, axel wrote:

    > heli0s <> wrote:
    >> I'm having a bit of a problem with a script I'm writing. I want the script
    >> to read in a txt file with a list of files to download and then go through
    >> the list of files to get them from the net.
    >>
    >> it should be something like this:
    >>
    >> use LWP::Simple;
    >>
    >> open (FILE,"all.m3u");
    >> @file = <FILE>;
    >> close FILE;
    >>
    >> foreach $line (@file){
    >> get ( $line );
    >> }

    >
    >> As seen in the above code, i've been looking at LWP::Simple, but what i
    >> have here is no good. Maybe i'm doing something wrong but i havent figured
    >> it out yet.

    >
    > Do you have a sample of a couple of lines from all.m3u?
    >
    > Axel


    there something like this:

    http://my.site.org/Albums/Asian Dub Foundation - Community
    %20Music/Asian%20Dub%20Foundation%20-%2001%20-%20real%20great%20britain.mp3

    (with all the spaces. Getting them out is kinda out of the question
    unfortunately)

    Christiaan
     
    heli0s, Aug 4, 2005
    #5
  6. heli0s

    Guest

    Hello,

    See if the following stuff helps you :

    Following is content of all.m3u file.
    $ cat all.m3u
    http://www.manning-source.com/books/hatcher2/hatcher2_chp1.pdf
    http://www.manning-source.com/books/hatcher2/hatcher2_chp3.pdf
    http://www.manning-source.com/books/hatcher2/hatcher2_chp4.pdf
    $

    Following is the script which can be used to download the above files.
    $ cat download.pl
    #!/usr/bin/perl
    use strict;
    use warnings;
    use LWP::Simple;
    my $filelist = 'all.m3u';
    open (FILE, $filelist) or die "Not able to open $filelist : $!";

    foreach my $file (<FILE>) {
    chomp($file); ## remove the last \n
    my @filearray = split /\//, $file;
    my $filename = pop @filearray;
    print "Downloading : $filename" . "\n";
    my $status = getstore($file, $filename);
    if(is_success($status)) { print ("Download of $filename
    complete\n");}
    else { print ("Download of $filename failed\n"); }
    }
    close (FILE);


    If you want to keep the directory structure consitent, you have to
    workout on $filename variable which is constucted from $file, the line
    of all.m3u file.
    you can add some stuff there so that directories are created and the
    files are stored in those directories.

    Regards,

    --sopan shewale



    heli0s wrote:
    > Hi,
    >
    > I'm having a bit of a problem with a script I'm writing. I want the script
    > to read in a txt file with a list of files to download and then go through
    > the list of files to get them from the net.
    >
    > it should be something like this:
    >
    > use LWP::Simple;
    >
    > open (FILE,"all.m3u");
    > @file = <FILE>;
    > close FILE;
    >
    > foreach $line (@file){
    > get ( $line );
    > }
    >
    > As seen in the above code, i've been looking at LWP::Simple, but what i
    > have here is no good. Maybe i'm doing something wrong but i havent figured
    > it out yet.
    >
    > Greets Christiaan
     
    , Aug 5, 2005
    #6
  7. heli0s

    J. Gleixner Guest

    wrote:
    [...]
    > foreach my $file (<FILE>) {
    > chomp($file); ## remove the last \n


    > my @filearray = split /\//, $file;
    > my $filename = pop @filearray;


    No need to store everything, only to pop one item.

    More succinct:

    my $filename = ( split '/', $file )[-1];

    > print "Downloading : $filename" . "\n";
    > my $status = getstore($file, $filename);
    > if(is_success($status)) { print ("Download of $filename
    > complete\n");}
    > else { print ("Download of $filename failed\n"); }


    As with open(), it's a good to show why get() failed.

    else {
    print "Error downloading $filename: ",
    $status->as_string, "\n",
    $status->status_line, "\n";
    }
    > }
    > close (FILE);
     
    J. Gleixner, Aug 5, 2005
    #7
  8. heli0s

    Guest

    heli0s <> wrote:
    >>> I'm having a bit of a problem with a script I'm writing. I want the script
    >>> to read in a txt file with a list of files to download and then go through
    >>> the list of files to get them from the net.


    >>> it should be something like this:


    >>> use LWP::Simple;
    >>>
    >>> open (FILE,"all.m3u");
    >>> @file = <FILE>;
    >>> close FILE;
    >>>
    >>> foreach $line (@file){
    >>> get ( $line );
    >>> }


    >>> As seen in the above code, i've been looking at LWP::Simple, but what i
    >>> have here is no good. Maybe i'm doing something wrong but i havent figured
    >>> it out yet.


    >> Do you have a sample of a couple of lines from all.m3u?


    > there something like this:


    > http://my.site.org/Albums/Asian Dub Foundation - Community
    > %20Music/Asian%20Dub%20Foundation%20-%2001%20-%20real%20great%20britain.mp3


    > (with all the spaces. Getting them out is kinda out of the question
    > unfortunately)


    The spaces should not be causing a problem.
    Nor should not chomping the input from the file as in your original script,
    although it is very bad practice not to.

    Maybe the URLs are not correct or that you are getting an error
    response. A quick way of checking is to use lynx or
    another browser to see if you can pull one up.

    One possible reason for getting an error response is a site
    refusing to return documents based on the User Agent. In
    that case you could use LWP::UserAgent and give an agent
    name such as 'Mozilla' which should fool the site.

    Axel
     
    , Aug 5, 2005
    #8
  9. heli0s

    Guest

    J. Gleixner wrote:
    > wrote:
    > [...]
    > > foreach my $file (<FILE>) {
    > > chomp($file); ## remove the last \n

    >
    > > my @filearray = split /\//, $file;
    > > my $filename = pop @filearray;

    >
    > No need to store everything, only to pop one item.
    >
    > More succinct:
    >
    > my $filename = ( split '/', $file )[-1];
    >
    > > print "Downloading : $filename" . "\n";
    > > my $status = getstore($file, $filename);
    > > if(is_success($status)) { print ("Download of $filename
    > > complete\n");}
    > > else { print ("Download of $filename failed\n"); }

    >
    > As with open(), it's a good to show why get() failed.
    >
    > else {
    > print "Error downloading $filename: ",
    > $status->as_string, "\n",
    > $status->status_line, "\n";
    > }
    > > }
    > > close (FILE);
     
    , Aug 8, 2005
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Wet Basement
    Replies:
    1
    Views:
    2,595
    BobMonk
    Jul 15, 2003
  2. dpackwood
    Replies:
    3
    Views:
    1,873
  3. pj
    Replies:
    3
    Views:
    325
  4. Replies:
    13
    Views:
    299
    Tad McClellan
    Feb 17, 2007
  5. Replies:
    5
    Views:
    184
    Joe Smith
    Feb 26, 2007
Loading...

Share This Page