Apache::AutoIndex - Perl replacment for mod_autoindex

Discussion in 'Perl Misc' started by Petyr David, Apr 24, 2008.

  1. Petyr David

    Petyr David Guest

    anyone have an experience using this?

    The real question:

    does using this speed the creation of a directory index in Apache
    significantly? We have directories with thousands of small files.

    Thanks!
     
    Petyr David, Apr 24, 2008
    #1
    1. Advertising

  2. Petyr David

    J. Gleixner Guest

    Petyr David wrote:
    > anyone have an experience using this?
    >
    > The real question:
    >
    > does using this speed the creation of a directory index in Apache
    > significantly? We have directories with thousands of small files.


    It's more likely that it'll be slower because mod_autoindex is
    written in C and compiled into the Apache daemon, you're not
    going to get much faster than that.

    Possibly you could list 500 at a time, or something, which
    would be faster, however having thousands of files in a directory
    isn't typically a good design.
     
    J. Gleixner, Apr 24, 2008
    #2
    1. Advertising

  3. Petyr David

    Petyr David Guest

    On Apr 24, 5:15 pm, "J. Gleixner" <>
    wrote:
    > Petyr David wrote:
    > > anyone have an experience using this?

    >
    > > The real question:

    >
    > > does using this speed the creation of a  directory index  in Apache
    > > significantly? We have directories with thousands of small files.

    >
    > It's more likely that it'll be slower because mod_autoindex is
    > written in C and compiled into the Apache daemon, you're not
    > going to get much faster than that.
    >
    > Possibly you could list 500 at a time, or something, which
    > would be faster, however having thousands of files in a directory
    > isn't typically a good design.


    agreed, but the nature of the data forces us to store the files in
    this fashion so there's some sense of meaningfulness: every file is
    named and then has a sequential number appended to it - there 64K
    potential numbers. we might have to start something like breaking it
    into 1000 files/directory.

    either way - thanks for your opinion
     
    Petyr David, Apr 25, 2008
    #3
  4. Petyr David

    smallpond Guest

    On Apr 24, 8:48 pm, Petyr David <> wrote:
    > On Apr 24, 5:15 pm, "J. Gleixner" <>
    > wrote:
    >
    > > Petyr David wrote:
    > > > anyone have an experience using this?

    >
    > > > The real question:

    >
    > > > does using this speed the creation of a directory index in Apache
    > > > significantly? We have directories with thousands of small files.

    >
    > > It's more likely that it'll be slower because mod_autoindex is
    > > written in C and compiled into the Apache daemon, you're not
    > > going to get much faster than that.

    >
    > > Possibly you could list 500 at a time, or something, which
    > > would be faster, however having thousands of files in a directory
    > > isn't typically a good design.

    >
    > agreed, but the nature of the data forces us to store the files in
    > this fashion so there's some sense of meaningfulness: every file is
    > named and then has a sequential number appended to it - there 64K
    > potential numbers. we might have to start something like breaking it
    > into 1000 files/directory.
    >
    > either way - thanks for your opinion



    Why not use a real database instead of making one out of a
    filesystem? Directory searches are slow, sequential string
    compares. Database lookups use fast hash techniques.
     
    smallpond, Apr 25, 2008
    #4
  5. Petyr David

    J. Gleixner Guest

    Petyr David wrote:
    > On Apr 24, 5:15 pm, "J. Gleixner" <>
    > wrote:
    >> Petyr David wrote:
    >>> anyone have an experience using this?
    >>> The real question:
    >>> does using this speed the creation of a directory index in Apache
    >>> significantly? We have directories with thousands of small files.

    >> It's more likely that it'll be slower because mod_autoindex is
    >> written in C and compiled into the Apache daemon, you're not
    >> going to get much faster than that.
    >>
    >> Possibly you could list 500 at a time, or something, which
    >> would be faster, however having thousands of files in a directory
    >> isn't typically a good design.

    >
    > agreed, but the nature of the data forces us to store the files in
    > this fashion so there's some sense of meaningfulness: every file is
    > named and then has a sequential number appended to it - there 64K
    > potential numbers. we might have to start something like breaking it
    > into 1000 files/directory.
    >
    > either way - thanks for your opinion


    You could also generate one or more static HTML files, once a day or
    whenever new files are added, which would display the filename
    and contain a link to the file. Heck, you could probably display
    5-10 per line, too. That way you'd avoid having to stat
    the thousands of files every time the directory is accessed.
     
    J. Gleixner, Apr 25, 2008
    #5
  6. Petyr David

    Guest

    smallpond <> wrote:
    > On Apr 24, 8:48 pm, Petyr David <> wrote:
    > > On Apr 24, 5:15 pm, "J. Gleixner" <>
    > > wrote:
    > >
    > > > Petyr David wrote:
    > > > > anyone have an experience using this?

    > >
    > > > > The real question:

    > >
    > > > > does using this speed the creation of a directory index in Apache
    > > > > significantly? We have directories with thousands of small files.

    > >
    > > > It's more likely that it'll be slower because mod_autoindex is
    > > > written in C and compiled into the Apache daemon, you're not
    > > > going to get much faster than that.

    > >
    > > > Possibly you could list 500 at a time, or something, which
    > > > would be faster, however having thousands of files in a directory
    > > > isn't typically a good design.

    > >
    > > agreed, but the nature of the data forces us to store the files in
    > > this fashion so there's some sense of meaningfulness: every file is
    > > named and then has a sequential number appended to it - there 64K
    > > potential numbers. we might have to start something like breaking it
    > > into 1000 files/directory.
    > >
    > > either way - thanks for your opinion

    >
    > Why not use a real database instead of making one out of a
    > filesystem?


    Why not use a file system rather than making one out of a database?
    Admitted, appending a sequential number to the file name is rather strange,
    but that still doesn't mean that he really wants a database rather than a
    file system.

    > Directory searches are slow, sequential string
    > compares.


    Not if you already know the exact name of the file. At least, not on any
    reasonable file system. They use trees or hashes or something to find the
    file quickly.

    > Database lookups use fast hash techniques.


    Same as reasonable file systems, given an exact name. Of course, if you
    are doing globs, or just extracting all entries, then neither file systems
    nor databases will use fast hash techniques.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    The costs of publication of this article were defrayed in part by the
    payment of page charges. This article must therefore be hereby marked
    advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
    this fact.
     
    , Apr 25, 2008
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    0
    Views:
    389
  2. Oliver Kowalke

    drop-in replacment for function pointers

    Oliver Kowalke, Feb 5, 2005, in forum: C++
    Replies:
    3
    Views:
    323
    Victor Bazarov
    Feb 5, 2005
  3. The Poor
    Replies:
    2
    Views:
    258
    Gregory Toomey
    Sep 27, 2003
  4. Kostas Hatzikokolakis

    Use perl in apache without #!/path/to/perl

    Kostas Hatzikokolakis, Oct 18, 2003, in forum: Perl Misc
    Replies:
    2
    Views:
    181
    Ron Savage
    Oct 19, 2003
  5. joe
    Replies:
    8
    Views:
    216
    Matthias Weckman
    Nov 14, 2003
Loading...

Share This Page