Finding the first file in a dir, fast, for message queue.

Discussion in 'Ruby' started by Alex Maccaw, Jan 8, 2008.

  1. Alex Maccaw

    Alex Maccaw Guest

    I have a file based msg queue, and msgs are stored five folders down.

    When a request for the next msg is received, I need to grab the first
    file I can find, as fast as I can.

    I've tried Dir.glob, and only selecting the first file. This however is
    an awful way of doing it as it loads every file into memory, before
    selecting the first.

    A bit better is Find.find, which finds files incrementally. However,
    this still takes about 0.005 seconds (I presume since it's also
    'finding' directories').

    Is there a faster way to do this?
    --
    Posted via http://www.ruby-forum.com/.
    Alex Maccaw, Jan 8, 2008
    #1
    1. Advertising

  2. 2008/1/8, Alex Maccaw <>:
    > I have a file based msg queue, and msgs are stored five folders down.
    >
    > When a request for the next msg is received, I need to grab the first
    > file I can find, as fast as I can.
    >
    > I've tried Dir.glob, and only selecting the first file. This however is
    > an awful way of doing it as it loads every file into memory, before
    > selecting the first.
    >
    > A bit better is Find.find, which finds files incrementally. However,
    > this still takes about 0.005 seconds (I presume since it's also
    > 'finding' directories').
    >
    > Is there a faster way to do this?


    You find 5ms when accessing the file system long? I'd say that's
    pretty fast considering what you do (recursive search). I doubt you
    will get much improvement as long as you always access the file system
    for your search. If you know the change frequency of files then you
    could store file system contents in memory and only update every n
    seconds / minutes or whatever or have a background thread that
    continuously updates your in memory representation.

    Kind regards

    robert


    --
    use.inject do |as, often| as.you_can - without end
    Robert Klemme, Jan 8, 2008
    #2
    1. Advertising

  3. Alex Maccaw

    Alex Maccaw Guest

    Re: Finding the first file in a dir, fast, for message queue

    > You find 5ms when accessing the file system long? I'd say that's
    > pretty fast considering what you do (recursive search). I doubt you
    > will get much improvement as long as you always access the file system
    > for your search. If you know the change frequency of files then you
    > could store file system contents in memory and only update every n
    > seconds / minutes or whatever or have a background thread that
    > continuously updates your in memory representation.



    Well, it's long compared to generating folder names, a guid, and writing
    files. It means that I can publish about 1000 msgs per second on to my
    queue, but only pull of 243 msgs per second.
    --
    Posted via http://www.ruby-forum.com/.
    Alex Maccaw, Jan 8, 2008
    #3
  4. Re: Finding the first file in a dir, fast, for message queue

    2008/1/8, Alex Maccaw <>:
    > > You find 5ms when accessing the file system long? I'd say that's
    > > pretty fast considering what you do (recursive search). I doubt you
    > > will get much improvement as long as you always access the file system
    > > for your search. If you know the change frequency of files then you
    > > could store file system contents in memory and only update every n
    > > seconds / minutes or whatever or have a background thread that
    > > continuously updates your in memory representation.

    >
    >
    > Well, it's long compared to generating folder names, a guid, and writing
    > files. It means that I can publish about 1000 msgs per second on to my
    > queue, but only pull of 243 msgs per second.


    Here's another option: find all files and put them in a queue. Only
    redo search when the queue is empty. This might pay off over all.

    robert


    --
    use.inject do |as, often| as.you_can - without end
    Robert Klemme, Jan 8, 2008
    #4
  5. Alex Maccaw

    Alex Maccaw Guest

    Re: Finding the first file in a dir, fast, for message queue

    Robert Klemme wrote:
    > 2008/1/8, Alex Maccaw <>:
    >> files. It means that I can publish about 1000 msgs per second on to my
    >> queue, but only pull of 243 msgs per second.

    >
    > Here's another option: find all files and put them in a queue. Only
    > redo search when the queue is empty. This might pay off over all.
    >
    > robert


    Yes, that's what I've done. Message polling is now the same speed as
    publishing (so the overall speed is about 10000 msg per second). If you
    interested, here's the queue:

    http://code.google.com/p/sparrow
    --
    Posted via http://www.ruby-forum.com/.
    Alex Maccaw, Jan 8, 2008
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. vivienne wykes
    Replies:
    1
    Views:
    457
    Andrew Thompson
    Aug 3, 2004
  2. Russell Warren

    Is Queue.Queue.queue.clear() thread-safe?

    Russell Warren, Jun 22, 2006, in forum: Python
    Replies:
    4
    Views:
    678
    Russell Warren
    Jun 27, 2006
  3. Ikkezelf

    Write message in windows message queue (msmq)

    Ikkezelf, May 2, 2006, in forum: C Programming
    Replies:
    5
    Views:
    711
    Richard Heathfield
    May 2, 2006
  4. Kris
    Replies:
    0
    Views:
    477
  5. Kga Agk
    Replies:
    2
    Views:
    145
    Kga Agk
    Jun 29, 2009
Loading...

Share This Page