run perl program when there is new file?

Discussion in 'Perl Misc' started by Jie, Oct 1, 2007.

  1. Jie

    Jie Guest

    Hi,

    I have a folder where new files will be generated when some other
    programs finished working.

    Now, I need to write a perl program to store some text output files
    generated from the other programs into a MySQL database. i think that
    i could just write a perl code to generate a sql command file and
    insert text into database.

    The challenge is that this perl program needs to know what files have
    already been processed, while only process on NEW files.. Maybe in
    the mysql database, i could create a field to record the time of the
    last run, then compare that time with the timestamp of all files to
    find what files are new and therefore needs to be processed. if this
    is the way to go, can you please let me know how to get the timestamp
    of files? Also, to retrieve the time recorded in mySQL, the idea in my
    mind now is to use something like below. There might be a better way,
    i believe...
    the_times = `mysql -u me - p the_password; select last_time from
    dummp_table`

    Thanks!!

    Jie
     
    Jie, Oct 1, 2007
    #1
    1. Advertising

  2. On Oct 1, 1:26 pm, Jie <> wrote:
    > Hi,
    >
    > I have a folder where new files will be generated when some other
    > programs finished working.
    >
    > Now, I need to write a perl program to store some text output files
    > generated from the other programs into a MySQL database. i think that
    > i could just write a perl code to generate a sql command file and
    > insert text into database.
    >
    > The challenge is that this perl program needs to know what files have
    > already been processed, while only process on NEW files.. Maybe in
    > the mysql database, i could create a field to record the time of the
    > last run, then compare that time with the timestamp of all files to
    > find what files are new and therefore needs to be processed. if this
    > is the way to go, can you please let me know how to get the timestamp
    > of files? Also, to retrieve the time recorded in mySQL, the idea in my
    > mind now is to use something like below. There might be a better way,
    > i believe...
    > the_times = `mysql -u me - p the_password; select last_time from
    > dummp_table`
    >
    > Thanks!!
    >
    > Jie


    Can you move the files once they have been processed by Perl ?

    For the perl script, maybe you can add an entry to the crontab to
    check every minutes if there is new files in the specified directory
    and if there is new files, put the content in the MySQL database and
    move the files somewhere else.

    Or you can create a table of files and if the files listed in the
    directory are not in the table, put it in the file list table and add
    their content to the table where it should be.

    If you can't play with the crontab, maybe you can add the SQL stuff at
    the end of your other program.. or if you don't have the source you
    can create a wrapper for this application that will run the
    application and once it's done, add the output to the database

    I'm pretty sure there is MANY other ways to do the job too.

    --Ben
     
    Benoit Lefebvre, Oct 1, 2007
    #2
    1. Advertising

  3. Jie

    Ben Morrow Guest

    Quoth Jie <>:
    >
    > I have a folder where new files will be generated when some other
    > programs finished working.
    >
    > Now, I need to write a perl program to store some text output files
    > generated from the other programs into a MySQL database. i think that
    > i could just write a perl code to generate a sql command file and
    > insert text into database.


    Better would be to use

    perldoc DBI
    perldoc DBD::mysql

    If you don't have DBI installed, start with

    perldoc CPAN

    basically, you need to run

    perl -MCPAN -e'install DBI'

    and answer all the prompts. Defaults are usually good if you don't
    understand the questions :).

    > The challenge is that this perl program needs to know what files have
    > already been processed, while only process on NEW files.. Maybe in
    > the mysql database, i could create a field to record the time of the
    > last run, then compare that time with the timestamp of all files to
    > find what files are new and therefore needs to be processed. if this
    > is the way to go, can you please let me know how to get the timestamp
    > of files?


    perldoc -f stat

    and also I find

    perldoc File::stat

    very helpful.


    > Also, to retrieve the time recorded in mySQL, the idea in my
    > mind now is to use something like below. There might be a better way,
    > i believe...


    DBI again. The answer to almost any database-related question in Perl
    involves DBI :).

    Ben
     
    Ben Morrow, Oct 1, 2007
    #3
  4. Jie

    Dr.Ruud Guest

    Jie schreef:

    > The challenge is that this perl program needs to know what files have
    > already been processed, while only process on NEW files.


    Mimic how maildir-type delivery works (new. tmp, cur).

    --
    Affijn, Ruud

    "Gewoon is een tijger."
     
    Dr.Ruud, Oct 3, 2007
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. David Stockwell
    Replies:
    2
    Views:
    551
    Grant Edwards
    Jun 8, 2004
  2. Shawn
    Replies:
    2
    Views:
    1,490
    Shawn
    Dec 6, 2006
  3. pradeep nair
    Replies:
    2
    Views:
    552
  4. Replies:
    9
    Views:
    984
    Paddy O'Loughlin
    Feb 27, 2009
  5. Replies:
    2
    Views:
    471
    Thomas 'PointedEars' Lahn
    Mar 11, 2008
Loading...

Share This Page