Scanning directories for new files?

Discussion in 'Python' started by Matty Sarro, Dec 21, 2010.

  1. Matty Sarro

    Matty Sarro Guest

    Hey everyone.
    I'm in the midst of writing a parser to clean up incoming files,
    remove extra data that isn't needed, normalize some values, etc. The
    base files will be uploaded via FTP.
    How does one go about scanning a directory for new files? For now
    we're looking to run it as a cron job but eventually would like to
    move away from that into making it a service running in the
    background.
    Matty Sarro, Dec 21, 2010
    #1
    1. Advertising

  2. Matty Sarro

    Jon Clements Guest

    On Dec 21, 7:17 pm, Matty Sarro <> wrote:
    > Hey everyone.
    > I'm in the midst of writing a parser to clean up incoming files,
    > remove extra data that isn't needed, normalize some values, etc. The
    > base files will be uploaded via FTP.
    > How does one go about scanning a directory for new files? For now
    > we're looking to run it as a cron job but eventually would like to
    > move away from that into making it a service running in the
    > background.


    Not a direct answer, but I would choose the approach of letting the
    FTP server know when a new file has been added. For instance:
    http://www.pureftpd.org/project/pure-ftpd -

    "Any external shell script can be called after a successful upload.
    Virus scanners and database archiveal can easily be set up."

    Of course, there's loads more servers, that I'm sure will have
    callback events or similar.

    Although, yes, the monitoring the file system is completely possible.

    hth

    Jon.
    Jon Clements, Dec 21, 2010
    #2
    1. Advertising

  3. On Tue, 21 Dec 2010 14:17:40 -0500, Matty Sarro wrote:

    > Hey everyone.
    > I'm in the midst of writing a parser to clean up incoming files, remove
    > extra data that isn't needed, normalize some values, etc. The base files
    > will be uploaded via FTP.
    > How does one go about scanning a directory for new files? For now we're
    > looking to run it as a cron job but eventually would like to move away
    > from that into making it a service running in the background.
    >

    Make sure the files are initially uploaded using a name that the parser
    isn't looking for and rename it when the upload is finished. This way the
    parser won't try to process a partially loaded file.

    If you are uploading to a *nix machine You the rename can move the file
    between directories provided both directories are in the same filing
    system. Under those conditions rename is always an atomic operation with
    no copying involved. This would you to, say, upload the file to "temp/
    myfile" and renamed it to "uploaded/myfile" with your parser only
    scanning the uploaded directory and, presumably, renaming processed files
    to move them to a third directory ready for further processing.

    I've used this technique reliably with files arriving via FTP at quite
    high rates.


    --
    martin@ | Martin Gregorie
    gregorie. | Essex, UK
    org |
    Martin Gregorie, Dec 21, 2010
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Joel Finkel
    Replies:
    0
    Views:
    488
    Joel Finkel
    Sep 12, 2003
  2. Jeffry van de Vuurst
    Replies:
    2
    Views:
    522
    Jeffry van de Vuurst
    Jul 30, 2003
  3. =?Utf-8?B?TGFzc2UgTmlsc3Nvbg==?=

    Multiple bin-directories with virtual directories?

    =?Utf-8?B?TGFzc2UgTmlsc3Nvbg==?=, Nov 9, 2004, in forum: ASP .Net
    Replies:
    0
    Views:
    823
    =?Utf-8?B?TGFzc2UgTmlsc3Nvbg==?=
    Nov 9, 2004
  4. Bryan
    Replies:
    8
    Views:
    168
    Old Pedant
    Aug 20, 2008
  5. Adam Petrie
    Replies:
    8
    Views:
    303
    Adam Petrie
    Oct 11, 2004
Loading...

Share This Page