Avoiding excessive writes

Discussion in 'Python' started by Gabriel Genellina, Mar 19, 2009.

  1. En Thu, 19 Mar 2009 00:52:54 -0200, <> escribió:

    > I'm aware that C, and thus Python I suppose, will cache file writes
    > for efficiency and to avoid excessive disk activity,


    Python 2.X, yes, because it uses C file streams (FILE) to implement Python
    file objects. Python 3.x avoids that layer.

    > Does the cache get written simply when an X-byte buffer fills up, and
    > can I change X?


    Yes, use the bufsize argument to open.

    > Does it write after waiting so long?


    I don't think so. You may call flush() yourself after some time.

    --
    Gabriel Genellina
    Gabriel Genellina, Mar 19, 2009
    #1
    1. Advertising

  2. Gabriel Genellina

    Guest

    I'm aware that C, and thus Python I suppose, will cache file writes
    for efficiency and to avoid excessive disk activity, causing
    occasional problems when forgetting to flush. How exactly this works
    is still somewhat of a mystery to me, but a concern because of a
    program I'm developing:

    The program performs tests on some hardware attached to a serial port
    and cycles every few seconds; for the reliability tests I'll do this
    anywhere from 10k to 100k times, logging each event. Sometimes the
    file gets targeted to a thumbdrive, and 100k writes likely puts
    significant wear on the flash, so what I've done in the past is just
    make my own cache of ~500 data points then go and write it all at
    once, but is there some way to set up the file operations so it
    effectively does this for me?

    Does the cache get written simply when an X-byte buffer fills up, and
    can I change X? Does it write after waiting so long?
    , Mar 19, 2009
    #2
    1. Advertising

  3. Gabriel Genellina

    John Machin Guest

    On Mar 19, 1:20 pm, "Gabriel Genellina" <>
    wrote:
    > En Thu, 19 Mar 2009 00:52:54 -0200, <> escribió:
    >
    > > I'm aware that C, and thus Python I suppose, will cache file writes
    > > for efficiency and to avoid excessive disk activity,

    >
    > Python 2.X, yes, because it uses C file streams (FILE) to implement Python  
    > file objects. Python 3.x avoids that layer.
    >
    > > Does the cache get written simply when an X-byte buffer fills up, and
    > > can I change X?

    >
    > Yes, use the bufsize argument to open.
    >
    > > Does it write after waiting so long?

    >
    > I don't think so. You may call flush() yourself after some time.
    >


    After it leaves Python, and the C stdio if that's used, isn't it then
    at the whim of the operating system exactly when and how much gets
    written to disk? My understanding is that this layer sometimes makes
    an effort to detect that the client is writing size X blocks
    sequentially and may decide to to buffer them up and do an N*X
    physical write later (like where N * X might represent a whole number
    of tracks) ... and/or you've got the drive itself which may do some
    buffering ...
    John Machin, Mar 19, 2009
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Matt Jensen

    ASP PerfMon Excessive Requests Queued

    Matt Jensen, Dec 14, 2005, in forum: ASP .Net
    Replies:
    1
    Views:
    2,348
    Chris Huddle
    Dec 14, 2005
  2. Don Bala

    Tomcat and excessive threads

    Don Bala, Jul 21, 2003, in forum: Java
    Replies:
    0
    Views:
    419
    Don Bala
    Jul 21, 2003
  3. Mat K.
    Replies:
    17
    Views:
    531
    EightNineThree
    Oct 17, 2003
  4. Replies:
    2
    Views:
    300
    Keith Thompson
    Feb 21, 2007
  5. Chris
    Replies:
    9
    Views:
    142
    Chris
    Nov 20, 2006
Loading...

Share This Page