get todays files

[Andrew D]

I have a script that will login to my ftp server and download all the
backup files, but I want it to only download the files that were
created today, e.g. if I ran the script today I want it to only fetch
files created today.

I am really not sure about how to do this, but it is quite important
to me, so all help is highly appreciated!

Andrew

 

[Steve Holden]

I have a script that will login to my ftp server and download all the
backup files, but I want it to only download the files that were
created today, e.g. if I ran the script today I want it to only fetch
files created today.

I am really not sure about how to do this, but it is quite important
to me, so all help is highly appreciated!

IIRC there's an "ftpmirror" script in the Tools directory (if you're on
Windows - Linux/Unix users have to download the source). I adapted it to
several different purposes.

regards
Steve

 

[Andrew D]

IIRC there's an "ftpmirror" script in the Tools directory (if you're on
Windows - Linux/Unix users have to download the source). I adapted it to
several different purposes.

regards
 Steve

Thanks Steve,

I don't want to sound rude here, but I really want some code to add to
my script so that it will only download todays files.

Andrew

 

[Steve Holden]

Thanks Steve,

I don't want to sound rude here, but I really want some code to add to
my script so that it will only download todays files.

That's OK. Maybe somebody's already got that code ready to go. I don't ...

regards
Steve

 

[Andrew D]

That's OK. Maybe somebody's already got that code ready to go. I don't ....

regards
 Steve

Thank you anyway Steve!

Lets hope someone has the code

Andrew

 

[Tim Chase]

I have a script that will login to my ftp server and download

all the backup files, but I want it to only download the files
that were created today, e.g. if I ran the script today I want
it to only fetch files created today.

Use Python's ftp module and send the MDTM command to get back the
timestamp of the filename. Insecurely, this would look something
like

from ftplib import FTP
hostname = "ftp.mozilla.org"
conn = FTP(hostname)
user = "anonymous"
password = "(e-mail address removed)"
conn.login(user, password)
filename = "pub/README"
results = conn.sendcmd("MDTM %s" % filename)
code, stamp = results.split(None, 1)
assert code == "213", "Unexpected result"
print "%s was modified on %s" % (filename, stamp)
today = '20081210'
if stamp[:8] == today:
process(filename)
else:
print "ignoring", filename

The MDTM command is not part of the core RFC-959, but rather the
RFC-3659[1] so you might run across some servers that don't
support it. You can read more about the Python ftplib module at
[2] which would be where you want to read up on pulling back a
listing of the directory of file-names to check. There is a NLST
command (I don't have a server handy that supports this command).
The LIST command returns pretty/readable information that's not
quite so machine-parsing friendly (at least in a cross-FTP-server
sort of way). However, that part, I leave as an exercise for the
reader along with the complications of the "today" bit.

Oh, SteveH, I checked your FTP-cloning source in my Python dir,
and it doesn't look like it does anything regarding file-times in
it, so that may have been a red-herring. Unless you've added
something since the ver. I've got here.

-tkc


[1]
http://en.wikipedia.org/wiki/List_of_FTP_commands

[2]
http://www.python.org/doc/2.5.2/lib/ftp-objects.html

 

[Andrew D]

I have a script that will login to my ftp server and download
all the backup files, but I want it to only download the files
that were created today, e.g. if I ran the script today I want
it to only fetch files created today.

Use Python's ftp module and send the MDTM command to get back the
timestamp of the filename.  Insecurely, this would look something
like

   from ftplib import FTP
   hostname = "ftp.mozilla.org"
   conn = FTP(hostname)
   user = "anonymous"
   password = "(e-mail address removed)"
   conn.login(user, password)
   filename = "pub/README"
   results = conn.sendcmd("MDTM %s" % filename)
   code, stamp = results.split(None, 1)
   assert code == "213", "Unexpected result"
   print "%s was modified on %s" % (filename, stamp)
   today = '20081210'
   if stamp[:8] == today:
     process(filename)
   else:
     print "ignoring", filename

The MDTM command is not part of the core RFC-959, but rather the
RFC-3659[1] so you might run across some servers that don't
support it.  You can read more about the Python ftplib module at
[2] which would be where you want to read up on pulling back a
listing of the directory of file-names to check.  There is a NLST
command (I don't have a server handy that supports this command).
  The LIST command returns pretty/readable information that's not
quite so machine-parsing friendly (at least in a cross-FTP-server
sort of way).  However, that part, I leave as an exercise for the
reader along with the complications of the "today" bit.

Oh, SteveH, I checked your FTP-cloning source in my Python dir,
and it doesn't look like it does anything regarding file-times in
it, so that may have been a red-herring.  Unless you've added
something since the ver. I've got here.

-tkc

[1]http://en.wikipedia.org/wiki/List_of_FTP_commands

[2]http://www.python.org/doc/2.5.2/lib/ftp-objects.html

This looks very good and I have tested successfully, but is there a
way I can set the today to automatically become todays date in that
format?

Thanks though, this is what I was looking for!

Andrew

 

[Tim Chase]

This looks very good and I have tested successfully, but is there a

way I can set the today to automatically become todays date in that
format?

Yes...see the python datetime module[1]...particularly the
strftime() call on date/datetime objects.

-tkc

[1]
http://docs.python.org/library/datetime.html

 

[Andrew Doades]

This looks very good and I have tested successfully, but is there a
way I can set the today to automatically become todays date in that
format?

Yes...see the python datetime module[1]...particularly the strftime()
call on date/datetime objects.

-tkc

[1]
http://docs.python.org/library/datetime.html

Thanks Tim, I got it in there myself.

Thanks for all your help with this and the links proved very useful and
interesting reads.

Andrew

 

[Andrew Doades]

This looks very good and I have tested successfully, but is there a
way I can set the today to automatically become todays date in that
format?

Yes...see the python datetime module[1]...particularly the strftime()
call on date/datetime objects.

-tkc

[1]
http://docs.python.org/library/datetime.html

I know this will sound like I am being very cheeky, but is there a way
you can make this for where the ftp server is actually windows server?

The script runs and it shows the date the file was modified, but still
downloads them, I have attached the script I am now using with your code in.


import os
from ftplib import FTP
import subprocess
from datetime import datetime
from time import gmtime, strftime

curr_date = strftime("%d %B %Y", gmtime())

# config
ftp_uname = ''
ftp_pwd = ''
ftp_server = ''
# end of config

fileroot = 'DBBackup_'
os.chdir('c:/')
files = []

logfile = open('c:/dbbackup/getmdbackups.log','a+')

def makelist(line):
if (line.startswith(fileroot)):
fs = [line]
files.append(fs)

def log(line):
ll = str(datetime.now()) + ' : ' + str(line)
print ll
logfile.write(ll + '\n')

def fileexists(ff, size):
if (os.path.exists(ff)):
stat = os.stat(ff)
if (stat.st_size == size):
return True
return False

try:
# first connect using ftp to get a list of valid backup failes available
log('Connecting to ftp server')
ftp = FTP(ftp_server)
ftp.set_pasv(False)
#ftp.set_debuglevel(2)
resp = ftp.login(ftp_uname,ftp_pwd)
log(resp)
ftp.retrlines('NLST',makelist)
log(str(files))
ftp.quit()

# fetch files in a loop using wget.
for ff in files:
ftp = FTP(ftp_server)
ftp.set_pasv(False)
resp = ftp.login(ftp_uname,ftp_pwd)
log(resp)
size = ftp.size(ff[0])
log('Size of server file = ' + str(size))
#ftp.quit()
try:
if (not fileexists(ff[0],size)):
results = ftp.sendcmd("MDTM %s" % ff[0])
code, stamp = results.split(None, 1)
assert code == "213", "Unexpected result"
print "%s was modified on %s" % (ff[0], stamp)
today = curr_date
if stamp[:8] == today:
log('Transferring: ' + ff[0])
# make parameters to wget the backup file
params = ' ftp://' + ftp_server + '/' + ff[0]
rcode = subprocess.call('c:/wget.exe ' + params)
log('Return code from wget = ' + str(rcode))
if (rcode == 0):
ff[1] = 1

else:
log('File ' + ff[0] + ' already exists locally, not
transferring')
ff[1] = 1

except Exception, e:
log(str(e))

log('Transfer complete')
# delete the server files that have been transferred or are already
here with the right filesize.
for ff in files:
if (ff[1] == 1):
log('delete ' + ff[0])
except Exception,e:
log(str(e))
# clean up temp files
log('Finished.')
logfile.close()

 

[Tim Chase]

I know this will sound like I am being very cheeky, but is there a way

you can make this for where the ftp server is actually windows server?

For Windows Server, I don't have a Windows FTP server to test
with -- I've got the company Linux server, and the previous
testing site I used (I think I used ftp.mozilla.org) which also
likely runs some flavor of Linux. Neither supports the NLST from
my testing.

curr_date = strftime("%d %B %Y", gmtime())

The first thing I noticed was that your strftime formating needs
to match the format of the date that comes back from the FTP
site. In my test, that was "YYYYMMDD". As such, your "%d %B %Y"
would likely need to be "%Y%m%d".

ftp.retrlines('NLST',makelist)

The servers I tried didn't support the NLST command so I can't
exactly follow along here. However assuming that it correctly
populates the list of files here

for ff in files:

correctly, that's immaterial to me.

ftp = FTP(ftp_server)
ftp.set_pasv(False)
resp = ftp.login(ftp_uname,ftp_pwd)

Just curious why you're logging into the server each pass through
the loop -- I'd just connect once at the beginning of the loop,
pull the files, and then disconnect at the end of the loop.

assert code == "213", "Unexpected result"

Does this assert fail at any point?

if stamp[:8] == today:

Given the above date-formatting, this should fail *every* time
unless your FTP server is returning the date in some format other
than "YYYYMMDDhhmmss"



It's hard to pinpoint actual problems as this block of code has
been modified, or doesn't run...there's some bogus indentation in
your post:

log('Transferring: ' + ff[0])
# make parameters to wget the backup file
params = ' ftp://' + ftp_server + '/' + ff[0]
rcode = subprocess.call('c:/wget.exe ' + params)
log('Return code from wget = ' + str(rcode))
if (rcode == 0):
ff[1] = 1

else:
log('File ' + ff[0] + ' already exists locally, not
transferring')

because this "else" is hanging oddly. Additionally, the FTP
object has methods for downloading the content of a file, so I'd
not bother shelling out to wget as an additional dependency.

-tkc

 

[Andrew Doades]

For Windows Server, I don't have a Windows FTP server to test with --
I've got the company Linux server, and the previous testing site I
used (I think I used ftp.mozilla.org) which also likely runs some
flavor of Linux. Neither supports the NLST from my testing.


The first thing I noticed was that your strftime formating needs to
match the format of the date that comes back from the FTP site. In my
test, that was "YYYYMMDD". As such, your "%d %B %Y" would likely need
to be "%Y%m%d".


The servers I tried didn't support the NLST command so I can't exactly
follow along here. However assuming that it correctly populates the
list of files here


correctly, that's immaterial to me.


Just curious why you're logging into the server each pass through the
loop -- I'd just connect once at the beginning of the loop, pull the
files, and then disconnect at the end of the loop.

I support it would be somewhat better to download in a 'bulk' download
rather that a file at a time, this script was not written by me, I am
just the developer who has to make a new or modify the old one.

Does this assert fail at any point?

Nope, nothing shows up in my logs or on screen.

if stamp[:8] == today:

Given the above date-formatting, this should fail *every* time unless
your FTP server is returning the date in some format other than
"YYYYMMDDhhmmss"

This line appears to just get missed in the code, I think it might be
one of the problems when it downloads all the files.

It's hard to pinpoint actual problems as this block of code has been
modified, or doesn't run...there's some bogus indentation in your post:

log('Transferring: ' + ff[0])
# make parameters to wget the backup file
params = ' ftp://' + ftp_server + '/' + ff[0]
rcode = subprocess.call('c:/wget.exe ' + params)
log('Return code from wget = ' + str(rcode))
if (rcode == 0):
ff[1] = 1
else:
log('File ' + ff[0] + ' already exists locally, not
transferring')

because this "else" is hanging oddly. Additionally, the FTP object
has methods for downloading the content of a file, so I'd not bother
shelling out to wget as an additional dependency

I am running kubuntu 8.04 and have edited the code in kate, It seemed to
indent on every line, so I just 'pulled' it back a little.

.

-tkc

Thanks for you comments, I think I will try and start from scratch and
see what I get.

Andrew