queues? inotify? anything else?

Discussion in 'Ruby' started by Matt Harrison, Jun 15, 2011.

  1. I don't even really know how to describe this problem. I know what I
    want the end result to be, but I've run into a dead end trying to figure
    out possible solutions.

    I've got a script which looks for files of a given type in a directory,
    processes them, then moves the result elsewhere and removes the sources.

    Currently, if I add another file to the directory while the script is
    running, it will not be noticed (I'm using a simple directory read).

    I would like to be able to add file while the script is busy and have it
    append the new filename to its list. I've considered inotify type
    monitoring but I don't know much about it and if it could be made to fit
    this single-threaded script.

    I've also thought about re-reading the directory after each file
    operation is complete, and adding the previously unknown filenames to
    the list.

    I'm just not sure which way I should begin looking, if there's a good
    solution for this already out there, or...what.

    Thanks for some input
    Matt Harrison, Jun 15, 2011
    1. Advertisements

  2. Matt Harrison

    Eric Wong Guest

    Inotify descriptors have an internal queue[1] so events can be
    backlogged while your script is doing something else.

    The sleepy_penguin[2] RubyGem provides inotify bindings that are
    subclasses of Ruby IO objects, so you can IO.select on them in
    an event loop (sleepy_penguin also provides an epoll wrapper).

    sleepy_penguin is Linux-only and a fairly "raw" interface to the
    underlying system calls. Cool.io (libev wrapper)[3] and EventMachine[4]
    also support inotify and they may be more portable to systems without
    inotify (using kqueue or stat() loops).
    I had to do this once in a daemon since it ran on an NFS directory
    since inotify doesn't work on NFS :<

    [1] The inotify(7) manpage is a great resource for learning about
    how it works.
    [2] http://bogomips.org/sleepy_penguin (disclaimer: I'm the author)
    [3] http://coolio.github.com/
    [4] http://rubyeventmachine.com/
    Eric Wong, Jun 16, 2011
    1. Advertisements

  3. Thanks Eric,

    I'll read up on inotify when I get some time. I'm not quite
    understanding how it would work with a single-threaded process. I mean,
    if it's busy processing data, I don't see how it could receive events.

    I'm sure it'll become more clear as I read up.

    Matt Harrison, Jun 16, 2011
  4. Matt Harrison

    Thiago Massa Guest

    [Note: parts of this message were removed to make it a legal post.]


    I know what is the problem, PROBABLY.

    I've used inotify with node to do the same, the reason why it isn't working
    it's because the "inotify script" is probably waiting to a certain event
    happen(like a file moved to the directory) and that isn't happening.

    When a file is moved to that directory, a chain of events is sent by inotify
    like "directory X is open, X file have been moved here, directory X is

    I don't remember correctly, but there's events that:
    opens a file to read
    opens a file to write
    and so on...

    So that script is probably waiting for a event that NEVER happens.

    Instead of "moving" the file to the directory, try to create a new one, or
    PASTE the file and see if it works...

    Inotify is simple, if you have the script waste a little of your time
    checking what is going on and change the script into your needs.

    I hope my answer were helpful to you!
    Thiago Massa, Jun 16, 2011
  5. In this case you would need to keep track of all files seen already so
    you do not add files multiple times.
    Since your script is single threaded anyway and if you do not want to
    change too much you could simply wrap your whole script in a loop
    which is terminated if the reading of the directory does not find any
    matching files any more, e.g.

    files = nil

    until (files = find_files).empty?
    files.each do |f|
    process f
    move v

    Kind regards

    Robert Klemme, Jun 16, 2011
  6. Thanks Robert,

    It's such a simple solution but it's always the easy ones that I tend to
    Matt Harrison, Jun 16, 2011
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.