backup utility for remote hosts

D

dan baker

I recently hacked together a little utility to backup files from
multiple remote hosts to a local PC. The host I use for a lot of
websites does not offer nightly backups, and while they haven't
crashed yet, I wanted to pull copies of databases and other files that
get modified by server-side scripts.

I thought that I'd just post the source in case:
- people want to make constructive comments about how to improve the
code
- somebody needs a similar utility and can start with this source and
run with it.
- because I have used a lot of other people's modules, and thought
maybe this
would help somebody else out.

There are two files. The script, and a config file with the host and
file information.

----------------

#! /usr/bin/perl -w

=head1
# ------------------------------------------------------------------------------

Purpose -
This script backs up specific files from the web to local PC. Its
intended to be run nightly by Task Manager on a PC as an "automated"
utility that is capable of getting multiple files from multiple
servers
sequentially. Intended to get databases, or other files that may
have been changed by server-side tools, and archive to a local PC
just for
backup in cases where the Host does not offer nightly backups, or you
may not trust them to DO them.

Input - 'requires' a config file "config_files.txt" that lists
host,user,password
and files to be backed up

Output -
- pulls files from remote webserver and creates a 'mirror' directory
structure below this script on the local computer.

- creates a log file and emails to desigated address for feedback

History -
written by dan_at_dtbakerprojects.com 10/2004

# ------------------------------------------------------------------------------
=cut
# 3456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-
# ------------------------------------------------------------------------------
# ##############################################################################

# hardcoded parameters

$pCGI_debuglog = './debug.txt';

my $cSMTPserver = 'smtp.yourhost.net' ;
my $Sender = '"Your Name"<[email protected]>';
my $Recipient = $Sender ;
my $Subject = 'web backup utility results' ;
my $Type = 'text/plain';
my $Body ='';

# ##############################################################################
# ------------------------------------------------------------------------------
# external modules

use Net::FTP;
use MIME::Lite ;

# ------------------------------------------------------------------------------
# init local vars

my $tempString = '' ;
my @tempList = () ;
my %tempHash = () ;
my $tempStatus = '';

local $host = '';
local $username = '';
local $password = '';
local @FileList = ();
local $LastRun = 0;

my $ftp = '';
my $DirPath = '';
my $LocalPath = '';
my $FileExt = '';
my $CurrType = 'binary';
my $LastType = 'ascii';

# ########################### Start Main Executable code
#######################

# grab the timestamp from debug file to tell when last run

if (-f $pCGI_debuglog ) {
@tempList = stat($pCGI_debuglog);
$LastRun = $tempList[9];
} else {
$LastRun = 0;
}

$cDEBUG = 1 ; # set to 0 NOT to print debug statements to STDERR
# set to 1 for high-level diagnostics
# set to 'verbose' for detailed diagnostics

$cDEBUG_type = 'overwrite' ;
# 'append' | 'overwrite' debug file

# clear and init logfile
if ( $cDEBUG ) {

if ( $cDEBUG_type eq 'append' ) { # append
open( DEBUG_LOG , ">>$pCGI_debuglog" ) or
die "Failed to open log at $pCGI_debuglog because $!" ;
print DEBUG_LOG "appending to log" ;

} else {
open( DEBUG_LOG , ">$pCGI_debuglog" ) or
die "Failed to open log at $pCGI_debuglog because $!" ;
print DEBUG_LOG "cleared log" ;
}

print DEBUG_LOG " at [".scalar(localtime)."]\n" ;
close DEBUG_LOG ;
}

# redirect stderr to the log
# -----
close STDERR ;
open( STDERR , ">>$pCGI_debuglog" ) or
die "Failed to redirect SDTERR because $!" ;
print STDERR "running: $0\n\n" ;
print scalar(localtime)." running: $0\n\n" ;

# ...and leave open for runtime errors
# note that runtime msgs can be printed to STDERR regardless of
$cDEBUG

# ------------------------------------------------------------------------------

require "config_files.txt" ; # pull in the hosts/files to be backed up

# ------------------------------------------------------------------------------

print "\n\n".scalar(localtime)." ...all done, normal exit.\n";
print STDERR "\n\n".scalar(localtime)." ...all done, normal exit.\n";
close STDERR;

# ##############################################################################
# mail the log to admin

open( DEBUG_LOG , "<$pCGI_debuglog" ) ;
$Body = join('',(<DEBUG_LOG>) ) ;

MIME::Lite->send('smtp', $cSMTPserver , Hello=> $cSMTPserver,
Timeout=>60 );

unless ($Type ) { $Type = 'text/plain' }

# send mail
# -----
my $msg = MIME::Lite->new(
From => $Sender ,
To => $Recipient ,
Subject => $Subject ,
Type => $Type ,
Data => $Body
);

$msg->send() ;

# ##############################################################################

exit;

# ##############################################################################
# ##############################################################################
sub GetFiles {

$ftp = Net::FTP->new($host);

print STDERR "\nLogging into $host \n" ;
print "\nLogging into $host \n";
unless ( $ftp->login( $username , $password ) ) {
print "login to $host failed \n";
print STDERR "login to $host failed \n";
sleep 5;
return(1);
}

foreach $File (@FileList) {

# check directory tree
# -----
$LocalPath = '.';
$DirPath = $host.$File ;
$DirPath =~ s/(.+)\/.*$/$1/ ;
unless ( -d $DirPath ) {
print "gotta create local directories first...\n";

@tempList = split ('/', $DirPath );
foreach $tempString ( @tempList ) {
$LocalPath .= "\/$tempString" ;
unless (-d $LocalPath ) {
print "mkdir $LocalPath \n";
mkdir($LocalPath , 0777);
}
}
}

# check type
# -----
if ( $File =~ m/.*\.(.+)$/ ) {

if ( ( $1 eq 'pdf' ) or
( $1 eq 'jpg' ) or
( $1 eq 'gif' ) )
{
$CurrType = 'binary';
} else {
$CurrType = 'ascii';
}

} else { # no file extension, assume it is binary
$CurrType = 'binary';
}
unless ( $LastType eq $CurrType) {
print "setting type to $CurrType\n";
$ftp->type($CurrType);
$LastType = $CurrType ;
}

# check mod time
# ignoring any time diff between local PC and remote server since
# we will catch a backup the next day even if the times are off a
couple hours
# --------------
if ( ( -f "${host}${File}" ) and
( $ftp->mdtm($File) <= $LastRun )) {

print "$File has not been modified since $LastRun \n";
print STDERR "$File has not been modified since $LastRun \n"
if ($cDEBUG eq 'verbose');
next;
}

# get file
# -----
print "getting $File \n";
print STDERR "getting $File \n" if ($cDEBUG eq 'verbose') ;
$tempStatus = $ftp->get( $File, "${host}${File}" );

unless ( $tempStatus ) {
print "transfer failed, check error log... \n";
print STDERR "getting $File failed\n" ;
}
}

$ftp->quit;
1;
}

# ##############################################################################
1; # ------------------------- end of file
-------------------------------------





.....and the config file example



# config_files.txt
# ----------------
# create a section for each host, and list the files to backup
# be sure to make a call to &GetFiles between each!

# ##############################################################################
$host = 'host1.com' ;
$username = 'user1' ;
$password = 'password1';
@FileList = qw(
/public_html/employees/cgi-bin/databases/DB_OrderInfo
);

&GetFiles;

# ##############################################################################

# ... copy section for another host here...

# ##############################################################################
1; # end of config_files.txt
 
B

Brian McCauley

dan said:
I recently hacked together a little utility to backup files from
multiple remote hosts to a local PC. The host I use for a lot of
websites does not offer nightly backups, and while they haven't
crashed yet, I wanted to pull copies of databases and other files that
get modified by server-side scripts.
I thought that I'd just post the source in case:
- people want to make constructive comments about how to improve the
code

Ooooh... a code critique.

Hope you've got a think skin!

Firstly remember the golden rule:

Write less code. (Unless this would make your code less readable).


The -w switch should be considered deproacted in favour of 'use warnings'.

Your code would be more mainatable if you also say 'use strict'.
=head1
# ------------------------------------------------------------------------------

Purpose -
This script backs up specific files from the web to local PC. Its
intended to be run nightly by Task Manager on a PC as an "automated"
utility that is capable of getting multiple files from multiple
servers
sequentially. Intended to get databases, or other files that may
have been changed by server-side tools, and archive to a local PC
just for
backup in cases where the Host does not offer nightly backups, or you
may not trust them to DO them.

Input - 'requires' a config file "config_files.txt" that lists
host,user,password
and files to be backed up

Output -
- pulls files from remote webserver and creates a 'mirror' directory
structure below this script on the local computer.

- creates a log file and emails to desigated address for feedback

History -
written by dan_at_dtbakerprojects.com 10/2004

# ------------------------------------------------------------------------------
=cut
# 3456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-
# ------------------------------------------------------------------------------
# ##############################################################################

# hardcoded parameters

$pCGI_debuglog = './debug.txt';

my $cSMTPserver = 'smtp.yourhost.net' ;
my $Sender = '"Your Name"<[email protected]>';
my $Recipient = $Sender ;
my $Subject = 'web backup utility results' ;
my $Type = 'text/plain';
my $Body ='';

You missed a my on one of those - using strict would have pointed this
out for you.
# ##############################################################################
# ------------------------------------------------------------------------------
# external modules

use Net::FTP;
use MIME::Lite ;

Comments should add something to the code, not just restate what is
already perfectly well stated in the code. Remember - "Write less code.
(Unless this would make your code less readable)". This applies to
comments too. If a comment adds no readability as in "external modules"
in front of a series of 'use' statements then you should not include it.

# ------------------------------------------------------------------------------
# init local vars

my $tempString = '' ;
my @tempList = () ;
my %tempHash = () ;
my $tempStatus = '';

Theses are bad variable names. This is the wrong place to declare them.
You should always declare all variables as lexically scoped in the
smallest applicable scope unless you have a reason not to. (This
applies in all programming languages not just Perl). The importance of
this advice must not be underestimated.
local $host = '';
local $username = '';
local $password = '';
local @FileList = ();
local $LastRun = 0;

local() here makes no sense. You probably where thinking of 'use vars'.

Explicitly initialising an array to an empty list is totally redundant.

Explicitily initialising a scalar to 0 or '' to represent the concept
'false' or 'undefined' is inappropriate. Perl's scalars are implicitly
initialised to the special value undef and if you want a value to
repestent the concept on undefinedness that is the one to use.
my $ftp = '';
my $DirPath = '';
my $LocalPath = '';
my $FileExt = '';
my $CurrType = 'binary';
my $LastType = 'ascii';


This is the very much the wrong place to declare variables that are
going to be used inside a subroutine.

You should always declare all variables as lexically scoped in the
smallest applicable scope unless you have a reason not to. (This
applies in all programming languages not just Perl). The importance of
this advice must not be underestimated. I'm not kidding. This is
really very important.
# grab the timestamp from debug file to tell when last run

if (-f $pCGI_debuglog ) {
@tempList = stat($pCGI_debuglog);
$LastRun = $tempList[9];

Since @tempList is only being used within the block then that would be
the place to decare it. Except you could simply use a list-slice instead.
} else {
$LastRun = 0;

This is redundant. $LastRun is 0 already. And as I said above using
the number zero to represent the concept of undefinedness is un-Perlish.
}

$cDEBUG = 1 ; # set to 0 NOT to print debug statements to STDERR
# set to 1 for high-level diagnostics
# set to 'verbose' for detailed diagnostics

$cDEBUG_type = 'overwrite' ;
# 'append' | 'overwrite' debug file

You forgot to decalre those variables. Perl will do implicit variable
declaration if you omit 'use strict' but this runs a very high risk that
you'll mistype a variable name and Perl will think you really want
another variable.
# clear and init logfile
if ( $cDEBUG ) {

if ( $cDEBUG_type eq 'append' ) { # append
open( DEBUG_LOG , ">>$pCGI_debuglog" ) or
die "Failed to open log at $pCGI_debuglog because $!" ;
print DEBUG_LOG "appending to log" ;

} else {
open( DEBUG_LOG , ">$pCGI_debuglog" ) or
die "Failed to open log at $pCGI_debuglog because $!" ;
print DEBUG_LOG "cleared log" ;
}

print DEBUG_LOG " at [".scalar(localtime)."]\n" ;
close DEBUG_LOG ;
}

# redirect stderr to the log
# -----
close STDERR ;
open( STDERR , ">>$pCGI_debuglog" ) or
die "Failed to redirect SDTERR because $!" ;

You just closed STDERR - where do you expect that error to go? (Don't
close STDERR).
print STDERR "running: $0\n\n" ;
print scalar(localtime)." running: $0\n\n" ;

# ...and leave open for runtime errors
# note that runtime msgs can be printed to STDERR regardless of
$cDEBUG

# ------------------------------------------------------------------------------

require "config_files.txt" ; # pull in the hosts/files to be backed up

IMHO if the config file is written in Perl is it better to make your
utility into a module and make the confing file into a plain script that
uses the module.

exit;

# ##############################################################################
# ##############################################################################
sub GetFiles {

$ftp = Net::FTP->new($host);

Here you are poluting the file-scoped $ftp variable. This is very bad.
You should declare $ftp here.
print STDERR "\nLogging into $host \n" ;
print "\nLogging into $host \n";
unless ( $ftp->login( $username , $password ) ) {
print "login to $host failed \n";
print STDERR "login to $host failed \n";
sleep 5;
return(1);
}

foreach $File (@FileList) {

Get into the habit of always putting 'my' between the for and the
iterator unless you have a positive reason not to.
# check directory tree
# -----
$LocalPath = '.';
$DirPath = $host.$File ;

You sould always declare all variables as lexically scoped in the
smallest applicable scope unless you have a reason not to. So this
would have been the right place to declare those variables.
$DirPath =~ s/(.+)\/.*$/$1/ ;


unless ( -d $DirPath ) {
print "gotta create local directories first...\n";

@tempList = split ('/', $DirPath );

You sould always declare all variables as lexically scoped in the
smallest applicable scope unless you have a reason not to. So this
would have been the right place to declare that variable.

@tempList is still a bloody awful name. Give it a name that says
something or simply eliminate it. The variable is hardly shorter than
the split expression anyhow so you may as well just use the expression
directly.
foreach $tempString ( @tempList ) {
$LocalPath .= "\/$tempString" ;
unless (-d $LocalPath ) {
print "mkdir $LocalPath \n";
mkdir($LocalPath , 0777);
}
}
}

Consider using modules, like File::path, for frequent tasks.
# check type
# -----
if ( $File =~ m/.*\.(.+)$/ ) {

The leading .* is redundant.

....and the config file example



# config_files.txt
# ----------------
# create a section for each host, and list the files to backup
# be sure to make a call to &GetFiles between each!

# ##############################################################################
$host = 'host1.com' ;
$username = 'user1' ;
$password = 'password1';
@FileList = qw(
/public_html/employees/cgi-bin/databases/DB_OrderInfo
);

&GetFiles;

You should not use the special & prefixed syntax for calling subroutines
unless you understand and want the special semanitics it implies.
 
J

John W. Krahn

Brian said:
The leading .* is redundant.

No it is not, but the $ is.

$ perl -le'
for ( qw/ one.two.three.four five six.seven eight.nine.ten / ) {
print "A: $1" if m/.*\.(.+)$/;
print "B: $1" if m/\.(.+)$/;
print "C: $1" if m/.*\.(.+)/;
}
'
A: four
B: two.three.four
C: four
A: seven
B: seven
C: seven
A: ten
B: nine.ten
C: ten



John
 
B

Ben Morrow

Quoth "John W. Krahn said:
No it is not, but the $ is.

$ perl -le'
for ( qw/ one.two.three.four five six.seven eight.nine.ten / ) {
print "A: $1" if m/.*\.(.+)$/;
print "B: $1" if m/\.(.+)$/;
print "C: $1" if m/.*\.(.+)/;
}
'

I would prefer to write it as

/\.(.+?)$/

to make it clear we are matching something at the end of the string.

Ben
 
J

James Willmore

Brian said:
Ooooh... a code critique.

Hope you've got a think skin!

Firstly remember the golden rule:

Write less code. (Unless this would make your code less readable).

I think Brain said what the majority would say.

I'd also like to point out that you can save yourself a lot of time
formating your code by using Perl Tidy
(http://search.cpan.org/~shancock/Perl-Tidy-20031021/lib/Perl/Tidy.pm).
That appears to be the only reason for the line with the numbers
(column numbers maybe?).

HTH

Jim
 
B

botfood

....I recently found a bug in this code (in addition to the style
critiques). Turns out that I was not calling the type() method
correctly when switching between ascii and binary modes. After some
testing and further research, the *better* solution is to call the
ascii() and binary() methods to send the correct TYPE command to the
FTP server.

so, the section of code that switches modes might better be coded as:

unless ( $LastType eq $CurrType) {
print "\t\t changing type to $CurrType \n";
print STDERR "\t\t changing type to $CurrType \n" if ($cDEBUG eq
'verbose') ;
if ( $CurrType eq 'ascii' ) {
$ftp->ascii();
} else {
$ftp->binary();
}
$LastType = $CurrType ;
}
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,901
Latest member
Noble71S45

Latest Threads

Top