Download Files With a perl script

H

heli0s

Hi,

I'm having a bit of a problem with a script I'm writing. I want the script
to read in a txt file with a list of files to download and then go through
the list of files to get them from the net.

it should be something like this:

use LWP::Simple;

open (FILE,"all.m3u");
@file = <FILE>;
close FILE;

foreach $line (@file){
get ( $line );
}

As seen in the above code, i've been looking at LWP::Simple, but what i
have here is no good. Maybe i'm doing something wrong but i havent figured
it out yet.

Greets Christiaan
 
P

Paul Lalli

heli0s said:
Hi,

I'm having a bit of a problem with a script I'm writing. I want the script
to read in a txt file with a list of files to download and then go through
the list of files to get them from the net.

it should be something like this:

use LWP::Simple;

Please always put these lines in your script:
use strict;
use warnings;
open (FILE,"all.m3u");

Always check the return value of open()
open my $file, '<', 'all.m3u' or die "Cannot open file: $!";
@file = <FILE>;

Why are you storing all lines of this file in memory? Why not process
them one at a time?
close FILE;

foreach $line (@file){

Each $line contains a "\n" on the end. Do you want that newline? I'm
betting not.
perldoc -f chomp
get ( $line );
From perldoc LWP::Simple, get() returns the data fetched. Here, you're
calling get() in a void context (ie, not assigning its return value to
any variable), so you're throwing the data away.

You're also not checking to see if this is actually succeeding or not.

my $html = get ($line) or die "Could not fetch from $line: $!";

(note: I don't know for a fact that $! contains anything useful if
get() fails)
}

As seen in the above code, i've been looking at LWP::Simple, but what i
have here is no good. Maybe i'm doing something wrong but i havent figured
it out yet.

Please fix up your code, making it strict- and warnings-compliant, and
check the return values of your calls, and then post the results here
if you still haven't gotten it figured out.

Paul Lalli
 
A

axel

heli0s said:
I'm having a bit of a problem with a script I'm writing. I want the script
to read in a txt file with a list of files to download and then go through
the list of files to get them from the net.

it should be something like this:

use LWP::Simple;

open (FILE,"all.m3u");
@file = <FILE>;
close FILE;

foreach $line (@file){
get ( $line );
}
As seen in the above code, i've been looking at LWP::Simple, but what i
have here is no good. Maybe i'm doing something wrong but i havent figured
it out yet.

Do you have a sample of a couple of lines from all.m3u?

Axel
 
S

sopan.shewale

Hello,

See if the following stuff helps you :

Following is content of all.m3u file.
$ cat all.m3u
http://www.manning-source.com/books/hatcher2/hatcher2_chp1.pdf
http://www.manning-source.com/books/hatcher2/hatcher2_chp3.pdf
http://www.manning-source.com/books/hatcher2/hatcher2_chp4.pdf
$

Following is the script which can be used to download the above files.
$ cat download.pl
#!/usr/bin/perl
use strict;
use warnings;
use LWP::Simple;
my $filelist = 'all.m3u';
open (FILE, $filelist) or die "Not able to open $filelist : $!";

foreach my $file (<FILE>) {
chomp($file); ## remove the last \n
my @filearray = split /\//, $file;
my $filename = pop @filearray;
print "Downloading : $filename" . "\n";
my $status = getstore($file, $filename);
if(is_success($status)) { print ("Download of $filename
complete\n");}
else { print ("Download of $filename failed\n"); }
}
close (FILE);


If you want to keep the directory structure consitent, you have to
workout on $filename variable which is constucted from $file, the line
of all.m3u file.
you can add some stuff there so that directories are created and the
files are stored in those directories.

Regards,

--sopan shewale
 
J

J. Gleixner

foreach my $file (<FILE>) {
chomp($file); ## remove the last \n
my @filearray = split /\//, $file;
my $filename = pop @filearray;

No need to store everything, only to pop one item.

More succinct:

my $filename = ( split '/', $file )[-1];
print "Downloading : $filename" . "\n";
my $status = getstore($file, $filename);
if(is_success($status)) { print ("Download of $filename
complete\n");}
else { print ("Download of $filename failed\n"); }

As with open(), it's a good to show why get() failed.

else {
print "Error downloading $filename: ",
$status->as_string, "\n",
$status->status_line, "\n";
}
 
A

axel

there something like this:

(with all the spaces. Getting them out is kinda out of the question
unfortunately)

The spaces should not be causing a problem.
Nor should not chomping the input from the file as in your original script,
although it is very bad practice not to.

Maybe the URLs are not correct or that you are getting an error
response. A quick way of checking is to use lynx or
another browser to see if you can pull one up.

One possible reason for getting an error response is a site
refusing to return documents based on the User Agent. In
that case you could use LWP::UserAgent and give an agent
name such as 'Mozilla' which should fool the site.

Axel
 
S

sopan.shewale

J. Gleixner said:
foreach my $file (<FILE>) {
chomp($file); ## remove the last \n
my @filearray = split /\//, $file;
my $filename = pop @filearray;

No need to store everything, only to pop one item.

More succinct:

my $filename = ( split '/', $file )[-1];
print "Downloading : $filename" . "\n";
my $status = getstore($file, $filename);
if(is_success($status)) { print ("Download of $filename
complete\n");}
else { print ("Download of $filename failed\n"); }

As with open(), it's a good to show why get() failed.

else {
print "Error downloading $filename: ",
$status->as_string, "\n",
$status->status_line, "\n";
}
}
close (FILE);
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top