File IO question.

Discussion in 'Python' started by J Poirier, Jul 15, 2004.

  1. J Poirier

    J Poirier Guest

    Hi All,

    I'm hoping that someone might have some pointers or
    examples on how to proceed with a solution to the
    following problem:

    A test application, which produces a trace file, is
    being run for very long periods of time. Say 72 hours
    or more.

    The application is often running on older PCs that
    have relatively small hard drives in comparison to
    to how big the trace file can become.

    The trace files tend to accumulate as they're not
    always deleted once archived to a network database,
    and often times a machine will crash in the middle of
    a
    very long test due to the hard drive filling up.

    There are dozens of PCs being used for the tests so
    buying bigger hard drives isn't really feasible. And
    I'm guessing the hard drives would fill up eventually
    anyway regardless of the size of the hard drive, the
    crashes just wouldn't occur quite as often.

    The nice thing is that the trace files compress quite
    well.

    I messed around with the mmap and file object stuff
    as well as the win32 extensions thinking that I could
    extract and compress the data that was being written
    to
    the trace file, by the application, in chuncks.

    Although I was able to get it to work on a contrived
    setup, it didn't work when used with the real
    application.

    Any hints on how to get something similar to the above
    to work or recommendations on alternate solutions
    would
    be *greatly* appreciated.

    Thanks,
    Joe





    __________________________________
    Do you Yahoo!?
    Yahoo! Mail - 50x more storage than other providers!
    http://promotions.yahoo.com/new_mail
    J Poirier, Jul 15, 2004
    #1
    1. Advertising

  2. J Poirier

    Larry Bates Guest

    You answered your own question:

    > I'm guessing the hard drives would fill up eventually
    > anyway regardless of the size of the hard drive, the
    > crashes just wouldn't occur quite as often.


    If this statement is true, compressing the files won't
    help either as that is just the same as a larger hard
    drive.

    But, you could limit the size and number of trace files
    and periodically throw away the oldest ones to keep the
    trace files at a nearly constant size. I just can't tell
    from your description exactly what is getting written to
    the trace file that makes it so large (debugging info?).
    Maybe you could only log errors/warnings to the trace
    file so it grows more slowly?

    Larry Bates
    Syscon, Inc.

    "J Poirier" <> wrote in message
    news:...
    > Hi All,
    >
    > I'm hoping that someone might have some pointers or
    > examples on how to proceed with a solution to the
    > following problem:
    >
    > A test application, which produces a trace file, is
    > being run for very long periods of time. Say 72 hours
    > or more.
    >
    > The application is often running on older PCs that
    > have relatively small hard drives in comparison to
    > to how big the trace file can become.
    >
    > The trace files tend to accumulate as they're not
    > always deleted once archived to a network database,
    > and often times a machine will crash in the middle of
    > a
    > very long test due to the hard drive filling up.
    >
    > There are dozens of PCs being used for the tests so
    > buying bigger hard drives isn't really feasible. And
    > I'm guessing the hard drives would fill up eventually
    > anyway regardless of the size of the hard drive, the
    > crashes just wouldn't occur quite as often.
    >
    > The nice thing is that the trace files compress quite
    > well.
    >
    > I messed around with the mmap and file object stuff
    > as well as the win32 extensions thinking that I could
    > extract and compress the data that was being written
    > to
    > the trace file, by the application, in chuncks.
    >
    > Although I was able to get it to work on a contrived
    > setup, it didn't work when used with the real
    > application.
    >
    > Any hints on how to get something similar to the above
    > to work or recommendations on alternate solutions
    > would
    > be *greatly* appreciated.
    >
    > Thanks,
    > Joe
    >
    >
    >
    >
    >
    > __________________________________
    > Do you Yahoo!?
    > Yahoo! Mail - 50x more storage than other providers!
    > http://promotions.yahoo.com/new_mail
    Larry Bates, Jul 15, 2004
    #2
    1. Advertising

  3. J Poirier wrote:
    > A test application, which produces a trace file, is
    > being run for very long periods of time. Say 72 hours
    > or more.


    If the test application is a python app, you coud always replace
    the appropriate
    open('name', 'w')
    with
    bz2.BZ2File('name.bz2', 'w')
    Scott David Daniels, Jul 17, 2004
    #3
  4. J Poirier

    Elbert Lev Guest

    J Poirier <> wrote in message news:<>...

    Here is my piece of advice:

    Assuming that all computers have access to the network at all the
    time:

    1. For your trace files use one of logger classes.
    2. Restrict the size of the file to some reasonable limit.
    3. Name the log files with a combination of computer name and a
    timestamp
    4. As soon as the limit is reached logger will notify the client that
    it is about to change the file. As a reaction to this callback you can
    send the file to some server (see bellow) and delete the local copy.
    5. At application startup send all files to the server and delete the
    local copy.

    "Some server":

    Depending how much you want to "invest", you have several options:

    1. use shared network directory and just copy the file. The drawback:
    all computers have access to this shared directory. This can be a
    problem in some cases.

    2. use xmlrpc server/client combination. This means that you run
    SimpleXMLRPCServer or something like on a dedicated machine and embed
    the xmlrpcclient into your application. You can even do
    filtering/compression/decompression on the fly. XMLRPC is
    transactional, so to simplify the protocol, put the file name as one
    parameter and the file content as another parameter.

    > Hi All,
    >
    > I'm hoping that someone might have some pointers or
    > examples on how to proceed with a solution to the
    > following problem:
    >
    > A test application, which produces a trace file, is
    > being run for very long periods of time. Say 72 hours
    > or more.
    >
    > The application is often running on older PCs that
    > have relatively small hard drives in comparison to
    > to how big the trace file can become.
    >
    > The trace files tend to accumulate as they're not
    > always deleted once archived to a network database,
    > and often times a machine will crash in the middle of
    > a
    > very long test due to the hard drive filling up.
    >
    > There are dozens of PCs being used for the tests so
    > buying bigger hard drives isn't really feasible. And
    > I'm guessing the hard drives would fill up eventually
    > anyway regardless of the size of the hard drive, the
    > crashes just wouldn't occur quite as often.
    >
    > The nice thing is that the trace files compress quite
    > well.
    >
    > I messed around with the mmap and file object stuff
    > as well as the win32 extensions thinking that I could
    > extract and compress the data that was being written
    > to
    > the trace file, by the application, in chuncks.
    >
    > Although I was able to get it to work on a contrived
    > setup, it didn't work when used with the real
    > application.
    >
    > Any hints on how to get something similar to the above
    > to work or recommendations on alternate solutions
    > would
    > be *greatly* appreciated.
    >
    > Thanks,
    > Joe
    >
    >
    >
    >
    >
    > __________________________________
    > Do you Yahoo!?
    > Yahoo! Mail - 50x more storage than other providers!
    > http://promotions.yahoo.com/new_mail
    Elbert Lev, Jul 18, 2004
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Luis Esteban Valencia Muñoz
    Replies:
    3
    Views:
    1,408
    Scott Allen
    Jun 4, 2005
  2. key9
    Replies:
    7
    Views:
    374
    Pete Becker
    Sep 13, 2006
  3. Dave
    Replies:
    1
    Views:
    992
    Juan T. Llibre
    Jun 8, 2007
  4. =?ISO-8859-2?Q?Miros=B3aw?= Makowiecki

    Reading of file by next of map file and by next of file descriptor.

    =?ISO-8859-2?Q?Miros=B3aw?= Makowiecki, Jul 10, 2007, in forum: C++
    Replies:
    1
    Views:
    787
    Alf P. Steinbach
    Jul 10, 2007
  5. scad
    Replies:
    4
    Views:
    952
    James Kanze
    May 28, 2009
Loading...

Share This Page