queues? inotify? anything else?

Discussion in 'Ruby' started by Matt Harrison, Jun 15, 2011.

  1. I don't even really know how to describe this problem. I know what I
    want the end result to be, but I've run into a dead end trying to figure
    out possible solutions.

    I've got a script which looks for files of a given type in a directory,
    processes them, then moves the result elsewhere and removes the sources.

    Currently, if I add another file to the directory while the script is
    running, it will not be noticed (I'm using a simple directory read).

    I would like to be able to add file while the script is busy and have it
    append the new filename to its list. I've considered inotify type
    monitoring but I don't know much about it and if it could be made to fit
    this single-threaded script.

    I've also thought about re-reading the directory after each file
    operation is complete, and adding the previously unknown filenames to
    the list.

    I'm just not sure which way I should begin looking, if there's a good
    solution for this already out there, or...what.

    Thanks for some input
    Matt Harrison, Jun 15, 2011
    #1
    1. Advertising

  2. Matt Harrison

    Eric Wong Guest

    Matt Harrison <> wrote:
    > I would like to be able to add file while the script is busy and
    > have it append the new filename to its list. I've considered inotify
    > type monitoring but I don't know much about it and if it could be
    > made to fit this single-threaded script.


    Inotify descriptors have an internal queue[1] so events can be
    backlogged while your script is doing something else.

    The sleepy_penguin[2] RubyGem provides inotify bindings that are
    subclasses of Ruby IO objects, so you can IO.select on them in
    an event loop (sleepy_penguin also provides an epoll wrapper).

    sleepy_penguin is Linux-only and a fairly "raw" interface to the
    underlying system calls. Cool.io (libev wrapper)[3] and EventMachine[4]
    also support inotify and they may be more portable to systems without
    inotify (using kqueue or stat() loops).

    > I've also thought about re-reading the directory after each file
    > operation is complete, and adding the previously unknown filenames
    > to the list.


    I had to do this once in a daemon since it ran on an NFS directory
    since inotify doesn't work on NFS :<


    [1] The inotify(7) manpage is a great resource for learning about
    how it works.
    [2] http://bogomips.org/sleepy_penguin (disclaimer: I'm the author)
    [3] http://coolio.github.com/
    [4] http://rubyeventmachine.com/
    --
    Eric Wong
    Eric Wong, Jun 16, 2011
    #2
    1. Advertising

  3. On 16/06/2011 00:10, Eric Wong wrote:
    > Matt Harrison<> wrote:
    >> I would like to be able to add file while the script is busy and
    >> have it append the new filename to its list. I've considered inotify
    >> type monitoring but I don't know much about it and if it could be
    >> made to fit this single-threaded script.

    >
    > Inotify descriptors have an internal queue[1] so events can be
    > backlogged while your script is doing something else.
    >
    > The sleepy_penguin[2] RubyGem provides inotify bindings that are
    > subclasses of Ruby IO objects, so you can IO.select on them in
    > an event loop (sleepy_penguin also provides an epoll wrapper).
    >
    > sleepy_penguin is Linux-only and a fairly "raw" interface to the
    > underlying system calls. Cool.io (libev wrapper)[3] and EventMachine[4]
    > also support inotify and they may be more portable to systems without
    > inotify (using kqueue or stat() loops).
    >
    >> I've also thought about re-reading the directory after each file
    >> operation is complete, and adding the previously unknown filenames
    >> to the list.

    >
    > I had to do this once in a daemon since it ran on an NFS directory
    > since inotify doesn't work on NFS :<
    >
    >
    > [1] The inotify(7) manpage is a great resource for learning about
    > how it works.
    > [2] http://bogomips.org/sleepy_penguin (disclaimer: I'm the author)
    > [3] http://coolio.github.com/
    > [4] http://rubyeventmachine.com/


    Thanks Eric,

    I'll read up on inotify when I get some time. I'm not quite
    understanding how it would work with a single-threaded process. I mean,
    if it's busy processing data, I don't see how it could receive events.

    I'm sure it'll become more clear as I read up.

    Thanks
    Matt Harrison, Jun 16, 2011
    #3
  4. Matt Harrison

    Thiago Massa Guest

    [Note: parts of this message were removed to make it a legal post.]

    Hi!!!

    I know what is the problem, PROBABLY.

    I've used inotify with node to do the same, the reason why it isn't working
    it's because the "inotify script" is probably waiting to a certain event
    happen(like a file moved to the directory) and that isn't happening.

    When a file is moved to that directory, a chain of events is sent by inotify
    like "directory X is open, X file have been moved here, directory X is
    closed".

    I don't remember correctly, but there's events that:
    opens a file to read
    opens a file to write
    and so on...

    So that script is probably waiting for a event that NEVER happens.

    Instead of "moving" the file to the directory, try to create a new one, or
    PASTE the file and see if it works...

    Inotify is simple, if you have the script waste a little of your time
    checking what is going on and change the script into your needs.

    I hope my answer were helpful to you!

    2011/6/15 Matt Harrison <>

    > I don't even really know how to describe this problem. I know what I want
    > the end result to be, but I've run into a dead end trying to figure out
    > possible solutions.
    >
    > I've got a script which looks for files of a given type in a directory,
    > processes them, then moves the result elsewhere and removes the sources.
    >
    > Currently, if I add another file to the directory while the script is
    > running, it will not be noticed (I'm using a simple directory read).
    >
    > I would like to be able to add file while the script is busy and have it
    > append the new filename to its list. I've considered inotify type monitoring
    > but I don't know much about it and if it could be made to fit this
    > single-threaded script.
    >
    > I've also thought about re-reading the directory after each file operation
    > is complete, and adding the previously unknown filenames to the list.
    >
    > I'm just not sure which way I should begin looking, if there's a good
    > solution for this already out there, or...what.
    >
    > Thanks for some input
    >
    >
    >



    --
    Thiago Fernandes Massa
    11 83979414
    Thiago Massa, Jun 16, 2011
    #4
  5. On Wed, Jun 15, 2011 at 11:58 PM, Matt Harrison
    <> wrote:
    > I don't even really know how to describe this problem. I know what I want
    > the end result to be, but I've run into a dead end trying to figure out
    > possible solutions.
    >
    > I've got a script which looks for files of a given type in a directory,
    > processes them, then moves the result elsewhere and removes the sources.
    >
    > Currently, if I add another file to the directory while the script is
    > running, it will not be noticed (I'm using a simple directory read).
    >
    > I would like to be able to add file while the script is busy and have it
    > append the new filename to its list. I've considered inotify type monitoring
    > but I don't know much about it and if it could be made to fit this
    > single-threaded script.
    >
    > I've also thought about re-reading the directory after each file operation
    > is complete, and adding the previously unknown filenames to the list.


    In this case you would need to keep track of all files seen already so
    you do not add files multiple times.

    > I'm just not sure which way I should begin looking, if there's a good
    > solution for this already out there, or...what.
    >
    > Thanks for some input


    Since your script is single threaded anyway and if you do not want to
    change too much you could simply wrap your whole script in a loop
    which is terminated if the reading of the directory does not find any
    matching files any more, e.g.

    files = nil

    until (files = find_files).empty?
    files.each do |f|
    process f
    move v
    end
    end

    Kind regards

    robert

    --
    remember.guy do |as, often| as.you_can - without end
    http://blog.rubybestpractices.com/
    Robert Klemme, Jun 16, 2011
    #5
  6. On 16/06/2011 08:07, Robert Klemme wrote:
    > On Wed, Jun 15, 2011 at 11:58 PM, Matt Harrison
    > <> wrote:
    >> I don't even really know how to describe this problem. I know what I want
    >> the end result to be, but I've run into a dead end trying to figure out
    >> possible solutions.
    >>
    >> I've got a script which looks for files of a given type in a directory,
    >> processes them, then moves the result elsewhere and removes the sources.
    >>
    >> Currently, if I add another file to the directory while the script is
    >> running, it will not be noticed (I'm using a simple directory read).
    >>
    >> I would like to be able to add file while the script is busy and have it
    >> append the new filename to its list. I've considered inotify type monitoring
    >> but I don't know much about it and if it could be made to fit this
    >> single-threaded script.
    >>
    >> I've also thought about re-reading the directory after each file operation
    >> is complete, and adding the previously unknown filenames to the list.

    >
    > In this case you would need to keep track of all files seen already so
    > you do not add files multiple times.
    >
    >> I'm just not sure which way I should begin looking, if there's a good
    >> solution for this already out there, or...what.
    >>
    >> Thanks for some input

    >
    > Since your script is single threaded anyway and if you do not want to
    > change too much you could simply wrap your whole script in a loop
    > which is terminated if the reading of the directory does not find any
    > matching files any more, e.g.
    >
    > files = nil
    >
    > until (files = find_files).empty?
    > files.each do |f|
    > process f
    > move v
    > end
    > end
    >
    > Kind regards
    >
    > robert
    >


    Thanks Robert,

    It's such a simple solution but it's always the easy ones that I tend to
    overlook.
    Matt Harrison, Jun 16, 2011
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. tom

    Inotify on netfilter socket

    tom, Nov 17, 2006, in forum: Python
    Replies:
    0
    Views:
    269
  2. Jean-Paul Calderone

    Re: Inotify on netfilter socket

    Jean-Paul Calderone, Nov 17, 2006, in forum: Python
    Replies:
    0
    Views:
    469
    Jean-Paul Calderone
    Nov 17, 2006
  3. Christian Maier
    Replies:
    1
    Views:
    518
    Keith Thompson
    Mar 10, 2006
  4. Roc Zhou
    Replies:
    0
    Views:
    619
    Roc Zhou
    Oct 24, 2007
  5. Martin DeMello

    inotify over nfs

    Martin DeMello, Nov 22, 2006, in forum: Ruby
    Replies:
    4
    Views:
    772
    Martin DeMello
    Nov 22, 2006
Loading...

Share This Page