backup utility for remote hosts

Discussion in 'Perl Misc' started by dan baker, Oct 20, 2004.

  1. dan baker

    dan baker Guest

    I recently hacked together a little utility to backup files from
    multiple remote hosts to a local PC. The host I use for a lot of
    websites does not offer nightly backups, and while they haven't
    crashed yet, I wanted to pull copies of databases and other files that
    get modified by server-side scripts.

    I thought that I'd just post the source in case:
    - people want to make constructive comments about how to improve the
    code
    - somebody needs a similar utility and can start with this source and
    run with it.
    - because I have used a lot of other people's modules, and thought
    maybe this
    would help somebody else out.

    There are two files. The script, and a config file with the host and
    file information.

    ----------------

    #! /usr/bin/perl -w

    =head1
    # ------------------------------------------------------------------------------

    Purpose -
    This script backs up specific files from the web to local PC. Its
    intended to be run nightly by Task Manager on a PC as an "automated"
    utility that is capable of getting multiple files from multiple
    servers
    sequentially. Intended to get databases, or other files that may
    have been changed by server-side tools, and archive to a local PC
    just for
    backup in cases where the Host does not offer nightly backups, or you
    may not trust them to DO them.

    Input - 'requires' a config file "config_files.txt" that lists
    host,user,password
    and files to be backed up

    Output -
    - pulls files from remote webserver and creates a 'mirror' directory
    structure below this script on the local computer.

    - creates a log file and emails to desigated address for feedback

    History -
    written by dan_at_dtbakerprojects.com 10/2004

    # ------------------------------------------------------------------------------
    =cut
    # 3456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-
    # ------------------------------------------------------------------------------
    # ##############################################################################

    # hardcoded parameters

    $pCGI_debuglog = './debug.txt';

    my $cSMTPserver = 'smtp.yourhost.net' ;
    my $Sender = '"Your Name"<>';
    my $Recipient = $Sender ;
    my $Subject = 'web backup utility results' ;
    my $Type = 'text/plain';
    my $Body ='';

    # ##############################################################################
    # ------------------------------------------------------------------------------
    # external modules

    use Net::FTP;
    use MIME::Lite ;

    # ------------------------------------------------------------------------------
    # init local vars

    my $tempString = '' ;
    my @tempList = () ;
    my %tempHash = () ;
    my $tempStatus = '';

    local $host = '';
    local $username = '';
    local $password = '';
    local @FileList = ();
    local $LastRun = 0;

    my $ftp = '';
    my $DirPath = '';
    my $LocalPath = '';
    my $FileExt = '';
    my $CurrType = 'binary';
    my $LastType = 'ascii';

    # ########################### Start Main Executable code
    #######################

    # grab the timestamp from debug file to tell when last run

    if (-f $pCGI_debuglog ) {
    @tempList = stat($pCGI_debuglog);
    $LastRun = $tempList[9];
    } else {
    $LastRun = 0;
    }

    $cDEBUG = 1 ; # set to 0 NOT to print debug statements to STDERR
    # set to 1 for high-level diagnostics
    # set to 'verbose' for detailed diagnostics

    $cDEBUG_type = 'overwrite' ;
    # 'append' | 'overwrite' debug file

    # clear and init logfile
    if ( $cDEBUG ) {

    if ( $cDEBUG_type eq 'append' ) { # append
    open( DEBUG_LOG , ">>$pCGI_debuglog" ) or
    die "Failed to open log at $pCGI_debuglog because $!" ;
    print DEBUG_LOG "appending to log" ;

    } else {
    open( DEBUG_LOG , ">$pCGI_debuglog" ) or
    die "Failed to open log at $pCGI_debuglog because $!" ;
    print DEBUG_LOG "cleared log" ;
    }

    print DEBUG_LOG " at [".scalar(localtime)."]\n" ;
    close DEBUG_LOG ;
    }

    # redirect stderr to the log
    # -----
    close STDERR ;
    open( STDERR , ">>$pCGI_debuglog" ) or
    die "Failed to redirect SDTERR because $!" ;
    print STDERR "running: $0\n\n" ;
    print scalar(localtime)." running: $0\n\n" ;

    # ...and leave open for runtime errors
    # note that runtime msgs can be printed to STDERR regardless of
    $cDEBUG

    # ------------------------------------------------------------------------------

    require "config_files.txt" ; # pull in the hosts/files to be backed up

    # ------------------------------------------------------------------------------

    print "\n\n".scalar(localtime)." ...all done, normal exit.\n";
    print STDERR "\n\n".scalar(localtime)." ...all done, normal exit.\n";
    close STDERR;

    # ##############################################################################
    # mail the log to admin

    open( DEBUG_LOG , "<$pCGI_debuglog" ) ;
    $Body = join('',(<DEBUG_LOG>) ) ;

    MIME::Lite->send('smtp', $cSMTPserver , Hello=> $cSMTPserver,
    Timeout=>60 );

    unless ($Type ) { $Type = 'text/plain' }

    # send mail
    # -----
    my $msg = MIME::Lite->new(
    From => $Sender ,
    To => $Recipient ,
    Subject => $Subject ,
    Type => $Type ,
    Data => $Body
    );

    $msg->send() ;

    # ##############################################################################

    exit;

    # ##############################################################################
    # ##############################################################################
    sub GetFiles {

    $ftp = Net::FTP->new($host);

    print STDERR "\nLogging into $host \n" ;
    print "\nLogging into $host \n";
    unless ( $ftp->login( $username , $password ) ) {
    print "login to $host failed \n";
    print STDERR "login to $host failed \n";
    sleep 5;
    return(1);
    }

    foreach $File (@FileList) {

    # check directory tree
    # -----
    $LocalPath = '.';
    $DirPath = $host.$File ;
    $DirPath =~ s/(.+)\/.*$/$1/ ;
    unless ( -d $DirPath ) {
    print "gotta create local directories first...\n";

    @tempList = split ('/', $DirPath );
    foreach $tempString ( @tempList ) {
    $LocalPath .= "\/$tempString" ;
    unless (-d $LocalPath ) {
    print "mkdir $LocalPath \n";
    mkdir($LocalPath , 0777);
    }
    }
    }

    # check type
    # -----
    if ( $File =~ m/.*\.(.+)$/ ) {

    if ( ( $1 eq 'pdf' ) or
    ( $1 eq 'jpg' ) or
    ( $1 eq 'gif' ) )
    {
    $CurrType = 'binary';
    } else {
    $CurrType = 'ascii';
    }

    } else { # no file extension, assume it is binary
    $CurrType = 'binary';
    }
    unless ( $LastType eq $CurrType) {
    print "setting type to $CurrType\n";
    $ftp->type($CurrType);
    $LastType = $CurrType ;
    }

    # check mod time
    # ignoring any time diff between local PC and remote server since
    # we will catch a backup the next day even if the times are off a
    couple hours
    # --------------
    if ( ( -f "${host}${File}" ) and
    ( $ftp->mdtm($File) <= $LastRun )) {

    print "$File has not been modified since $LastRun \n";
    print STDERR "$File has not been modified since $LastRun \n"
    if ($cDEBUG eq 'verbose');
    next;
    }

    # get file
    # -----
    print "getting $File \n";
    print STDERR "getting $File \n" if ($cDEBUG eq 'verbose') ;
    $tempStatus = $ftp->get( $File, "${host}${File}" );

    unless ( $tempStatus ) {
    print "transfer failed, check error log... \n";
    print STDERR "getting $File failed\n" ;
    }
    }

    $ftp->quit;
    1;
    }

    # ##############################################################################
    1; # ------------------------- end of file
    -------------------------------------





    .....and the config file example



    # config_files.txt
    # ----------------
    # create a section for each host, and list the files to backup
    # be sure to make a call to &GetFiles between each!

    # ##############################################################################
    $host = 'host1.com' ;
    $username = 'user1' ;
    $password = 'password1';
    @FileList = qw(
    /public_html/employees/cgi-bin/databases/DB_OrderInfo
    );

    &GetFiles;

    # ##############################################################################

    # ... copy section for another host here...

    # ##############################################################################
    1; # end of config_files.txt
     
    dan baker, Oct 20, 2004
    #1
    1. Advertising

  2. dan baker wrote:
    > I recently hacked together a little utility to backup files from
    > multiple remote hosts to a local PC. The host I use for a lot of
    > websites does not offer nightly backups, and while they haven't
    > crashed yet, I wanted to pull copies of databases and other files that
    > get modified by server-side scripts.


    > I thought that I'd just post the source in case:
    > - people want to make constructive comments about how to improve the
    > code


    Ooooh... a code critique.

    Hope you've got a think skin!

    Firstly remember the golden rule:

    Write less code. (Unless this would make your code less readable).


    > ----------------
    >
    > #! /usr/bin/perl -w


    The -w switch should be considered deproacted in favour of 'use warnings'.

    Your code would be more mainatable if you also say 'use strict'.

    > =head1
    > # ------------------------------------------------------------------------------
    >
    > Purpose -
    > This script backs up specific files from the web to local PC. Its
    > intended to be run nightly by Task Manager on a PC as an "automated"
    > utility that is capable of getting multiple files from multiple
    > servers
    > sequentially. Intended to get databases, or other files that may
    > have been changed by server-side tools, and archive to a local PC
    > just for
    > backup in cases where the Host does not offer nightly backups, or you
    > may not trust them to DO them.
    >
    > Input - 'requires' a config file "config_files.txt" that lists
    > host,user,password
    > and files to be backed up
    >
    > Output -
    > - pulls files from remote webserver and creates a 'mirror' directory
    > structure below this script on the local computer.
    >
    > - creates a log file and emails to desigated address for feedback
    >
    > History -
    > written by dan_at_dtbakerprojects.com 10/2004
    >
    > # ------------------------------------------------------------------------------
    > =cut
    > # 3456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-
    > # ------------------------------------------------------------------------------
    > # ##############################################################################
    >
    > # hardcoded parameters
    >
    > $pCGI_debuglog = './debug.txt';
    >
    > my $cSMTPserver = 'smtp.yourhost.net' ;
    > my $Sender = '"Your Name"<>';
    > my $Recipient = $Sender ;
    > my $Subject = 'web backup utility results' ;
    > my $Type = 'text/plain';
    > my $Body ='';


    You missed a my on one of those - using strict would have pointed this
    out for you.

    > # ##############################################################################
    > # ------------------------------------------------------------------------------
    > # external modules
    >
    > use Net::FTP;
    > use MIME::Lite ;


    Comments should add something to the code, not just restate what is
    already perfectly well stated in the code. Remember - "Write less code.
    (Unless this would make your code less readable)". This applies to
    comments too. If a comment adds no readability as in "external modules"
    in front of a series of 'use' statements then you should not include it.


    > # ------------------------------------------------------------------------------
    > # init local vars
    >
    > my $tempString = '' ;
    > my @tempList = () ;
    > my %tempHash = () ;
    > my $tempStatus = '';


    Theses are bad variable names. This is the wrong place to declare them.
    You should always declare all variables as lexically scoped in the
    smallest applicable scope unless you have a reason not to. (This
    applies in all programming languages not just Perl). The importance of
    this advice must not be underestimated.

    > local $host = '';
    > local $username = '';
    > local $password = '';
    > local @FileList = ();
    > local $LastRun = 0;


    local() here makes no sense. You probably where thinking of 'use vars'.

    Explicitly initialising an array to an empty list is totally redundant.

    Explicitily initialising a scalar to 0 or '' to represent the concept
    'false' or 'undefined' is inappropriate. Perl's scalars are implicitly
    initialised to the special value undef and if you want a value to
    repestent the concept on undefinedness that is the one to use.

    > my $ftp = '';
    > my $DirPath = '';
    > my $LocalPath = '';
    > my $FileExt = '';
    > my $CurrType = 'binary';
    > my $LastType = 'ascii';



    This is the very much the wrong place to declare variables that are
    going to be used inside a subroutine.

    You should always declare all variables as lexically scoped in the
    smallest applicable scope unless you have a reason not to. (This
    applies in all programming languages not just Perl). The importance of
    this advice must not be underestimated. I'm not kidding. This is
    really very important.

    > # grab the timestamp from debug file to tell when last run
    >
    > if (-f $pCGI_debuglog ) {
    > @tempList = stat($pCGI_debuglog);
    > $LastRun = $tempList[9];


    Since @tempList is only being used within the block then that would be
    the place to decare it. Except you could simply use a list-slice instead.

    > } else {
    > $LastRun = 0;


    This is redundant. $LastRun is 0 already. And as I said above using
    the number zero to represent the concept of undefinedness is un-Perlish.

    > }
    >
    > $cDEBUG = 1 ; # set to 0 NOT to print debug statements to STDERR
    > # set to 1 for high-level diagnostics
    > # set to 'verbose' for detailed diagnostics
    >
    > $cDEBUG_type = 'overwrite' ;
    > # 'append' | 'overwrite' debug file


    You forgot to decalre those variables. Perl will do implicit variable
    declaration if you omit 'use strict' but this runs a very high risk that
    you'll mistype a variable name and Perl will think you really want
    another variable.

    > # clear and init logfile
    > if ( $cDEBUG ) {
    >
    > if ( $cDEBUG_type eq 'append' ) { # append
    > open( DEBUG_LOG , ">>$pCGI_debuglog" ) or
    > die "Failed to open log at $pCGI_debuglog because $!" ;
    > print DEBUG_LOG "appending to log" ;
    >
    > } else {
    > open( DEBUG_LOG , ">$pCGI_debuglog" ) or
    > die "Failed to open log at $pCGI_debuglog because $!" ;
    > print DEBUG_LOG "cleared log" ;
    > }
    >
    > print DEBUG_LOG " at [".scalar(localtime)."]\n" ;
    > close DEBUG_LOG ;
    > }
    >
    > # redirect stderr to the log
    > # -----
    > close STDERR ;
    > open( STDERR , ">>$pCGI_debuglog" ) or
    > die "Failed to redirect SDTERR because $!" ;


    You just closed STDERR - where do you expect that error to go? (Don't
    close STDERR).

    > print STDERR "running: $0\n\n" ;
    > print scalar(localtime)." running: $0\n\n" ;
    >
    > # ...and leave open for runtime errors
    > # note that runtime msgs can be printed to STDERR regardless of
    > $cDEBUG
    >
    > # ------------------------------------------------------------------------------
    >
    > require "config_files.txt" ; # pull in the hosts/files to be backed up


    IMHO if the config file is written in Perl is it better to make your
    utility into a module and make the confing file into a plain script that
    uses the module.


    > exit;
    >
    > # ##############################################################################
    > # ##############################################################################
    > sub GetFiles {
    >
    > $ftp = Net::FTP->new($host);


    Here you are poluting the file-scoped $ftp variable. This is very bad.
    You should declare $ftp here.

    >
    > print STDERR "\nLogging into $host \n" ;
    > print "\nLogging into $host \n";
    > unless ( $ftp->login( $username , $password ) ) {
    > print "login to $host failed \n";
    > print STDERR "login to $host failed \n";
    > sleep 5;
    > return(1);
    > }
    >
    > foreach $File (@FileList) {


    Get into the habit of always putting 'my' between the for and the
    iterator unless you have a positive reason not to.

    > # check directory tree
    > # -----
    > $LocalPath = '.';
    > $DirPath = $host.$File ;


    You sould always declare all variables as lexically scoped in the
    smallest applicable scope unless you have a reason not to. So this
    would have been the right place to declare those variables.

    > $DirPath =~ s/(.+)\/.*$/$1/ ;




    > unless ( -d $DirPath ) {
    > print "gotta create local directories first...\n";
    >
    > @tempList = split ('/', $DirPath );


    You sould always declare all variables as lexically scoped in the
    smallest applicable scope unless you have a reason not to. So this
    would have been the right place to declare that variable.

    @tempList is still a bloody awful name. Give it a name that says
    something or simply eliminate it. The variable is hardly shorter than
    the split expression anyhow so you may as well just use the expression
    directly.

    > foreach $tempString ( @tempList ) {
    > $LocalPath .= "\/$tempString" ;
    > unless (-d $LocalPath ) {
    > print "mkdir $LocalPath \n";
    > mkdir($LocalPath , 0777);
    > }
    > }
    > }


    Consider using modules, like File::path, for frequent tasks.

    > # check type
    > # -----
    > if ( $File =~ m/.*\.(.+)$/ ) {


    The leading .* is redundant.


    > ....and the config file example
    >
    >
    >
    > # config_files.txt
    > # ----------------
    > # create a section for each host, and list the files to backup
    > # be sure to make a call to &GetFiles between each!
    >
    > # ##############################################################################
    > $host = 'host1.com' ;
    > $username = 'user1' ;
    > $password = 'password1';
    > @FileList = qw(
    > /public_html/employees/cgi-bin/databases/DB_OrderInfo
    > );
    >
    > &GetFiles;


    You should not use the special & prefixed syntax for calling subroutines
    unless you understand and want the special semanitics it implies.
     
    Brian McCauley, Oct 20, 2004
    #2
    1. Advertising

  3. Brian McCauley wrote:
    >
    > dan baker wrote:
    >
    >> I recently hacked together a little utility to backup files from
    >> multiple remote hosts to a local PC. The host I use for a lot of
    >> websites does not offer nightly backups, and while they haven't
    >> crashed yet, I wanted to pull copies of databases and other files that
    >> get modified by server-side scripts.

    >
    >> I thought that I'd just post the source in case:
    >> - people want to make constructive comments about how to improve the
    >> code

    >
    > Ooooh... a code critique.


    Ooooh... a spelling critique.

    > Hope you've got a think skin!


    Hope it's THICK too! ;-)



    John
    --
    use Perl;
    program
    fulfillment
     
    John W. Krahn, Oct 20, 2004
    #3
  4. Brian McCauley wrote:
    >
    > dan baker wrote:
    >>
    >> # check type
    >> # -----
    >> if ( $File =~ m/.*\.(.+)$/ ) {

    >
    > The leading .* is redundant.


    No it is not, but the $ is.

    $ perl -le'
    for ( qw/ one.two.three.four five six.seven eight.nine.ten / ) {
    print "A: $1" if m/.*\.(.+)$/;
    print "B: $1" if m/\.(.+)$/;
    print "C: $1" if m/.*\.(.+)/;
    }
    '
    A: four
    B: two.three.four
    C: four
    A: seven
    B: seven
    C: seven
    A: ten
    B: nine.ten
    C: ten



    John
    --
    use Perl;
    program
    fulfillment
     
    John W. Krahn, Oct 20, 2004
    #4
  5. dan baker

    Ben Morrow Guest

    Quoth "John W. Krahn" <>:
    > Brian McCauley wrote:
    > >
    > > dan baker wrote:
    > >>
    > >> # check type
    > >> # -----
    > >> if ( $File =~ m/.*\.(.+)$/ ) {

    > >
    > > The leading .* is redundant.

    >
    > No it is not, but the $ is.
    >
    > $ perl -le'
    > for ( qw/ one.two.three.four five six.seven eight.nine.ten / ) {
    > print "A: $1" if m/.*\.(.+)$/;
    > print "B: $1" if m/\.(.+)$/;
    > print "C: $1" if m/.*\.(.+)/;
    > }
    > '


    I would prefer to write it as

    /\.(.+?)$/

    to make it clear we are matching something at the end of the string.

    Ben

    --
    don't get my sympathy hanging out the 15th floor. you've changed the locks 3
    times, he still comes reeling though the door, and soon he'll get to you, teach
    you how to get to purest hell. you do it to yourself and that's what really
    hurts is you do it to yourself just you, you and noone else **
     
    Ben Morrow, Oct 21, 2004
    #5
  6. Brian McCauley wrote:
    >
    >
    > dan baker wrote:
    >
    >> I recently hacked together a little utility to backup files from
    >> multiple remote hosts to a local PC. The host I use for a lot of
    >> websites does not offer nightly backups, and while they haven't
    >> crashed yet, I wanted to pull copies of databases and other files that
    >> get modified by server-side scripts.

    >
    >
    >> I thought that I'd just post the source in case:
    >> - people want to make constructive comments about how to improve the
    >> code

    >
    >
    > Ooooh... a code critique.
    >
    > Hope you've got a think skin!
    >
    > Firstly remember the golden rule:
    >
    > Write less code. (Unless this would make your code less readable).


    <snip>

    >> 3456789-123456789-123456789-123456789-123456789-123456789-123456789-123456789-


    I think Brain said what the majority would say.

    I'd also like to point out that you can save yourself a lot of time
    formating your code by using Perl Tidy
    (http://search.cpan.org/~shancock/Perl-Tidy-20031021/lib/Perl/Tidy.pm).
    That appears to be the only reason for the line with the numbers
    (column numbers maybe?).

    HTH

    Jim
     
    James Willmore, Oct 21, 2004
    #6
  7. dan baker

    botfood Guest

    ....I recently found a bug in this code (in addition to the style
    critiques). Turns out that I was not calling the type() method
    correctly when switching between ascii and binary modes. After some
    testing and further research, the *better* solution is to call the
    ascii() and binary() methods to send the correct TYPE command to the
    FTP server.

    so, the section of code that switches modes might better be coded as:

    unless ( $LastType eq $CurrType) {
    print "\t\t changing type to $CurrType \n";
    print STDERR "\t\t changing type to $CurrType \n" if ($cDEBUG eq
    'verbose') ;
    if ( $CurrType eq 'ascii' ) {
    $ftp->ascii();
    } else {
    $ftp->binary();
    }
    $LastType = $CurrType ;
    }
     
    botfood, Jan 6, 2005
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Bla
    Replies:
    0
    Views:
    2,887
  2. gavino
    Replies:
    13
    Views:
    1,069
  3. KDawg44
    Replies:
    1
    Views:
    293
  4. Bla
    Replies:
    1
    Views:
    240
    Tad McClellan
    Apr 10, 2005
  5. Replies:
    0
    Views:
    665
Loading...

Share This Page