Avoiding fragmentation

Discussion in 'Java' started by Roedy Green, Mar 18, 2011.

  1. Roedy Green

    Roedy Green Guest

    I just notice how when I create files with Java, they often are badly
    fragmented, with ten or more fragments.

    This could be avoided with a simple strategy. When you open the file
    for write you specify the estimated size of the file. The OS then
    looks for a continuous hunk of disk space to satisfy. If you guess too
    big, you return the tail to the free space pool. If you guess too
    small, it adds another extent or two.
    --
    Roedy Green Canadian Mind Products
    http://mindprod.com
    If you think it’s expensive to hire a professional to do the job, wait until you hire an amateur.
    ~ Red Adair
     
    Roedy Green, Mar 18, 2011
    #1
    1. Advertising

  2. In message <>, Roedy Green wrote:

    > I just notice how when I create files with Java, they often are badly
    > fragmented, with ten or more fragments.


    On Windows, is it?
     
    Lawrence D'Oliveiro, Mar 18, 2011
    #2
    1. Advertising

  3. On 2011-03-18 15:33:45 -0400, Roedy Green said:

    > I just notice how when I create files with Java, they often are badly
    > fragmented, with ten or more fragments.
    >
    > This could be avoided with a simple strategy. When you open the file
    > for write you specify the estimated size of the file. The OS then
    > looks for a continuous hunk of disk space to satisfy. If you guess too
    > big, you return the tail to the free space pool. If you guess too
    > small, it adds another extent or two.


    You can do that, or something much like it, by building up larger
    chunks of the file in memory before writing them to disk (using NIO,
    RandomAccessFile, your own buffering, or BufferedOutputStream). There's
    no guarantee that the filesystem will fragment less given larger
    writes, but a large block written at once at least propagates some size
    information to the OS, whereas a small write to a stream indicates
    nearly nothing about the volume of data following it.

    This isn't really a Java issue, though: the lowest common denominator
    for file creation is POSIX-like, which assumes infintely-appendable
    files with no fixed size (or size hints). Blame Bell, basically. ;)

    -o
     
    Owen Jacobson, Mar 19, 2011
    #3
  4. On 03/18/2011 07:43 PM, Lawrence D'Oliveiro wrote:
    > In message<>, Roedy Green wrote:
    >
    >> I just notice how when I create files with Java, they often are badly
    >> fragmented, with ten or more fragments.

    >
    > On Windows, is it?


    Actually, this may come as a shock to you, but Linux filesystems can and
    do have fragmentation problems. See
    <http://blog.mozilla.com/tglek/2010/07/22/file-fragmentation/> for some
    statistics (one of my files had "only" 400 fragments). But, hey, why let
    truth get in the way of Windows-bashing?

    --
    Beware of bugs in the above code; I have only proved it correct, not
    tried it. -- Donald E. Knuth
     
    Joshua Cranmer, Mar 19, 2011
    #4
  5. In message <im19mj$73k$-september.org>, Joshua Cranmer wrote:

    > On 03/18/2011 07:43 PM, Lawrence D'Oliveiro wrote:
    >
    >> In message<>, Roedy Green
    >> wrote:
    >>
    >>> I just notice how when I create files with Java, they often are badly
    >>> fragmented, with ten or more fragments.

    >>
    >> On Windows, is it?

    >
    > Actually, this may come as a shock to you, but Linux filesystems can and
    > do have fragmentation problems.


    And yet I have never known any of my customers’ Linux servers—or my own
    Linux machines, for that matter—to slow down over time.
     
    Lawrence D'Oliveiro, Mar 19, 2011
    #5
  6. On 03/19/2011 01:56 AM, Lawrence D'Oliveiro wrote:
    > In message<im19mj$73k$-september.org>, Joshua Cranmer wrote:
    >
    >> On 03/18/2011 07:43 PM, Lawrence D'Oliveiro wrote:
    >>
    >>> In message<>, Roedy Green
    >>> wrote:
    >>>
    >>>> I just notice how when I create files with Java, they often are badly
    >>>> fragmented, with ten or more fragments.
    >>>
    >>> On Windows, is it?

    >>
    >> Actually, this may come as a shock to you, but Linux filesystems can and
    >> do have fragmentation problems.

    >
    > And yet I have never known any of my customers’ Linux servers—or my own
    > Linux machines, for that matter—to slow down over time.


    And I have sped up cold startup by an order of magnitude just by
    defragging my harddrive. Note that Linux's filesystems are geared more
    towards server use, not towards desktop use.

    --
    Beware of bugs in the above code; I have only proved it correct, not
    tried it. -- Donald E. Knuth
     
    Joshua Cranmer, Mar 19, 2011
    #6
  7. Roedy Green

    Lew Guest

    On 03/19/2011 09:21 AM, Joshua Cranmer wrote:
    > On 03/19/2011 01:56 AM, Lawrence D'Oliveiro wrote:
    >> In message<im19mj$73k$-september.org>, Joshua Cranmer wrote:
    >>
    >>> On 03/18/2011 07:43 PM, Lawrence D'Oliveiro wrote:
    >>>
    >>>> In message<>, Roedy Green
    >>>> wrote:
    >>>>
    >>>>> I just notice how when I create files with Java, they often are badly
    >>>>> fragmented, with ten or more fragments.
    >>>>
    >>>> On Windows, is it?


    I am going to undo your presumption in eliding vital parts of Joshua's post,
    to revert the effect of your intellectual dishonesty, "Lawrence".

    >>> Actually, this may come as a shock to you, but Linux filesystems can and
    >>> do have fragmentation problems.
    >>> See <http://blog.mozilla.com/tglek/2010/07/22/file-fragmentation/>
    >>> for some statistics (one of my files had "only" 400 fragments).
    >>> But, hey, why let truth get in the way of Windows-bashing?


    >> And yet I have never known any of my customers’ Linux servers—or my own
    >> Linux machines, for that matter—to slow down over time.


    Not surprising. Also not surprising that you use your own very, very limited
    personal experience to try to countervail the documented, technical evidence
    that Joshua presented, then *omit* the reference lest the blinding light of
    truth reveal your pathetic ignorance, "Lawrence".

    To paraphrase Joshua, "But, hey, 'Lawrence', why let truth get in the way of
    your ignorance?"

    > And I have sped up cold startup by an order of magnitude just by defragging my
    > harddrive. Note that Linux's filesystems are geared more towards server use,
    > not towards desktop use.


    So Joshua not only has facts, dear boy, but personal experience that supports
    the truth and not your narrow, blindered and inaccurate view of the world,
    "Lawrence". So, "Larry"-boy, how 'bout you accept truth and allow your world
    view to expand? It's not in your own best interest to fight facts, "Lar".

    --
    Lew
    Honi soit qui mal y pense.
     
    Lew, Mar 19, 2011
    #7
  8. In message <im2alc$lv9$-september.org>, Joshua Cranmer wrote:

    > And I have sped up cold startup by an order of magnitude just by
    > defragging my harddrive.


    On a Linux system?

    > Note that Linux's filesystems are geared more towards server use, not
    > towards desktop use.


    What does that mean?
     
    Lawrence D'Oliveiro, Mar 20, 2011
    #8
  9. Roedy Green

    Roedy Green Guest

    On Fri, 18 Mar 2011 22:46:54 -0400, Owen Jacobson
    <> wrote, quoted or indirectly quoted someone
    who said :

    >
    >This isn't really a Java issue, though: the lowest common denominator


    If the various Java stream open methods developed an optional size
    parameter, either that fact could be passed to the OS for
    optimisation, or Java could explicitly allocate that much space, as
    per random i/o, and then discard the excess on close.

    I remember discovering that writing 0 byte record in DOS was the way
    to chop the file. At first I thought it was a bug, then a joke.
    --
    Roedy Green Canadian Mind Products
    http://mindprod.com
    If you think it’s expensive to hire a professional to do the job, wait until you hire an amateur.
    ~ Red Adair
     
    Roedy Green, Mar 20, 2011
    #9
  10. Roedy Green

    Roedy Green Guest

    On Fri, 18 Mar 2011 12:33:45 -0700, Roedy Green
    <> wrote, quoted or indirectly quoted
    someone who said :

    >This could be avoided with a simple strategy. When you open the file
    >for write you specify the estimated size of the file. The OS then
    >looks for a continuous hunk of disk space to satisfy. If you guess too
    >big, you return the tail to the free space pool. If you guess too
    >small, it adds another extent or two.


    The current approach is to use after-the-fact defragging. I have
    written extensive reviews on Windows defraggers.

    See http://mindprod.com/jgloss/defragger.html

    This does you no good at all for temp files or short-lived files.

    --
    Roedy Green Canadian Mind Products
    http://mindprod.com
    If you think it’s expensive to hire a professional to do the job, wait until you hire an amateur.
    ~ Red Adair
     
    Roedy Green, Mar 20, 2011
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ben_
    Replies:
    1
    Views:
    4,933
  2. DNass
    Replies:
    6
    Views:
    2,770
    Darryl L. Pierce
    Nov 28, 2004
  3. Ike
    Replies:
    4
    Views:
    770
    Thomas Weidenfeller
    Jul 29, 2005
  4. Tron Thomas

    Preventing memory fragmentation

    Tron Thomas, Dec 22, 2003, in forum: C++
    Replies:
    18
    Views:
    897
    Tom Plunket
    Jan 3, 2004
  5. spasmous
    Replies:
    5
    Views:
    2,418
Loading...

Share This Page