get the size of a dynamically changing file fast ?

Discussion in 'Python' started by Stef Mientki, Jan 17, 2008.

  1. Stef Mientki

    Stef Mientki Guest

    hello,

    I've a program (not written in Python) that generates a few thousands
    bytes per second,
    these files are dumped in 2 buffers (files), at in interval time of 50 msec,
    the files can be read by another program, to do further processing.

    A program written in VB or delphi can handle the data in the 2 buffers
    perfectly.
    Sometimes Python is also able to process the data correctly,
    but often it can't :-(

    I keep one of the files open en test the size of the open datafile each
    50 msec.
    I have tried
    os.stat ( ....) [ ST_SIZE]
    os.path.getsize ( ... )
    but they both have the same behaviour, sometimes it works, and the data
    is collected each 50 .. 100 msec,
    sometimes 1 .. 1.5 seconds is needed to detect a change in filesize.

    I'm using python 2.4 on winXP.

    Is there a solution for this problem ?

    thanks,
    Stef Mientki
    Stef Mientki, Jan 17, 2008
    #1
    1. Advertising

  2. On Jan 17, 3:56 pm, Stef Mientki <> wrote:
    > hello,
    >
    > I've a program (not written in Python) that generates a few thousands
    > bytes per second,
    > these files are dumped in 2 buffers (files), at in interval time of 50 msec,
    > the files can be read by another program, to do further processing.
    >
    > A program written in VB or delphi can handle the data in the 2 buffers
    > perfectly.
    > Sometimes Python is also able to process the data correctly,
    > but often it can't :-(
    >
    > I keep one of the files open en test the size of the open datafile each
    > 50 msec.
    > I have tried
    > os.stat ( ....) [ ST_SIZE]
    > os.path.getsize ( ... )
    > but they both have the same behaviour, sometimes it works, and the data
    > is collected each 50 .. 100 msec,
    > sometimes 1 .. 1.5 seconds is needed to detect a change in filesize.
    >
    > I'm using python 2.4 on winXP.
    >
    > Is there a solution for this problem ?
    >
    > thanks,
    > Stef Mientki


    Tim Golden has a method to watch for changes in a directory on his
    website:

    http://tgolden.sc.sabren.com/python/win32_how_do_i/watch_directory_for_changes.html

    This old post also mentions something similar:

    http://mail.python.org/pipermail/python-list/2007-October/463065.html

    And here's a cookbook recipe that claims to do it as well using
    decorators:

    http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/426620

    Hopefully that will get you going.

    Mike
    Mike Driscoll, Jan 18, 2008
    #2
    1. Advertising

  3. Stef Mientki

    Stef Mientki Guest

    Mike Driscoll wrote:
    > On Jan 17, 3:56 pm, Stef Mientki <> wrote:
    >
    >> hello,
    >>
    >> I've a program (not written in Python) that generates a few thousands
    >> bytes per second,
    >> these files are dumped in 2 buffers (files), at in interval time of 50 msec,
    >> the files can be read by another program, to do further processing.
    >>
    >> A program written in VB or delphi can handle the data in the 2 buffers
    >> perfectly.
    >> Sometimes Python is also able to process the data correctly,
    >> but often it can't :-(
    >>
    >> I keep one of the files open en test the size of the open datafile each
    >> 50 msec.
    >> I have tried
    >> os.stat ( ....) [ ST_SIZE]
    >> os.path.getsize ( ... )
    >> but they both have the same behaviour, sometimes it works, and the data
    >> is collected each 50 .. 100 msec,
    >> sometimes 1 .. 1.5 seconds is needed to detect a change in filesize.
    >>
    >> I'm using python 2.4 on winXP.
    >>
    >> Is there a solution for this problem ?
    >>
    >> thanks,
    >> Stef Mientki
    >>

    >
    > Tim Golden has a method to watch for changes in a directory on his
    > website:
    >
    > http://tgolden.sc.sabren.com/python/win32_how_do_i/watch_directory_for_changes.html
    >
    > This old post also mentions something similar:
    >
    > http://mail.python.org/pipermail/python-list/2007-October/463065.html
    >
    > And here's a cookbook recipe that claims to do it as well using
    > decorators:
    >
    > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/426620
    >
    > Hopefully that will get you going.
    >
    > Mike
    >

    thanks Mike,
    sorry for the late reaction.
    I've it working perfect now.
    After all, os.stat works perfectly well,
    the problem was in the program that generated the file with increasing
    size,
    by truncating it after each block write, it apperently garantees that
    the file is flushed to disk and all problems are solved.

    cheers,
    Stef Mientki
    Stef Mientki, Jan 22, 2008
    #3
  4. On Jan 22, 3:35 pm, Stef Mientki <> wrote:
    > Mike Driscoll wrote:
    > > On Jan 17, 3:56 pm, Stef Mientki <> wrote:

    >
    > >> hello,

    >
    > >> I've a program (not written in Python) that generates a few thousands
    > >> bytes per second,
    > >> these files are dumped in 2 buffers (files), at in interval time of 50 msec,
    > >> the files can be read by another program, to do further processing.

    >
    > >> A program written in VB or delphi can handle the data in the 2 buffers
    > >> perfectly.
    > >> Sometimes Python is also able to process the data correctly,
    > >> but often it can't :-(

    >
    > >> I keep one of the files open en test the size of the open datafile each
    > >> 50 msec.
    > >> I have tried
    > >> os.stat ( ....) [ ST_SIZE]
    > >> os.path.getsize ( ... )
    > >> but they both have the same behaviour, sometimes it works, and the data
    > >> is collected each 50 .. 100 msec,
    > >> sometimes 1 .. 1.5 seconds is needed to detect a change in filesize.

    >
    > >> I'm using python 2.4 on winXP.

    >
    > >> Is there a solution for this problem ?

    >
    > >> thanks,
    > >> Stef Mientki

    >
    > > Tim Golden has a method to watch for changes in a directory on his
    > > website:

    >
    > >http://tgolden.sc.sabren.com/python/win32_how_do_i/watch_directory_fo...

    >
    > > This old post also mentions something similar:

    >
    > >http://mail.python.org/pipermail/python-list/2007-October/463065.html

    >
    > > And here's a cookbook recipe that claims to do it as well using
    > > decorators:

    >
    > >http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/426620

    >
    > > Hopefully that will get you going.

    >
    > > Mike

    >
    > thanks Mike,
    > sorry for the late reaction.
    > I've it working perfect now.
    > After all, os.stat works perfectly well,
    > the problem was in the program that generated the file with increasing
    > size,
    > by truncating it after each block write, it apperently garantees that
    > the file is flushed to disk and all problems are solved.
    >
    > cheers,
    > Stef Mientki


    I almost asked if you were making sure you had flushed the data to the
    file...oh well.

    Mike
    Mike Driscoll, Jan 22, 2008
    #4
  5. Stef Mientki

    Stef Mientki Guest

    Mike Driscoll wrote:
    > On Jan 22, 3:35 pm, Stef Mientki <> wrote:
    >
    >> Mike Driscoll wrote:
    >>
    >>> On Jan 17, 3:56 pm, Stef Mientki <> wrote:
    >>>
    >>>> hello,
    >>>>
    >>>> I've a program (not written in Python) that generates a few thousands
    >>>> bytes per second,
    >>>> these files are dumped in 2 buffers (files), at in interval time of 50 msec,
    >>>> the files can be read by another program, to do further processing.
    >>>>
    >>>> A program written in VB or delphi can handle the data in the 2 buffers
    >>>> perfectly.
    >>>> Sometimes Python is also able to process the data correctly,
    >>>> but often it can't :-(
    >>>>
    >>>> I keep one of the files open en test the size of the open datafile each
    >>>> 50 msec.
    >>>> I have tried
    >>>> os.stat ( ....) [ ST_SIZE]
    >>>> os.path.getsize ( ... )
    >>>> but they both have the same behaviour, sometimes it works, and the data
    >>>> is collected each 50 .. 100 msec,
    >>>> sometimes 1 .. 1.5 seconds is needed to detect a change in filesize.
    >>>>
    >>>> I'm using python 2.4 on winXP.
    >>>>
    >>>> Is there a solution for this problem ?
    >>>>
    >>>> thanks,
    >>>> Stef Mientki
    >>>>
    >>> Tim Golden has a method to watch for changes in a directory on his
    >>> website:
    >>>
    >>> http://tgolden.sc.sabren.com/python/win32_how_do_i/watch_directory_fo...
    >>>
    >>> This old post also mentions something similar:
    >>>
    >>> http://mail.python.org/pipermail/python-list/2007-October/463065.html
    >>>
    >>> And here's a cookbook recipe that claims to do it as well using
    >>> decorators:
    >>>
    >>> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/426620
    >>>
    >>> Hopefully that will get you going.
    >>>
    >>> Mike
    >>>

    >> thanks Mike,
    >> sorry for the late reaction.
    >> I've it working perfect now.
    >> After all, os.stat works perfectly well,
    >> the problem was in the program that generated the file with increasing
    >> size,
    >> by truncating it after each block write, it apperently garantees that
    >> the file is flushed to disk and all problems are solved.
    >>
    >> cheers,
    >> Stef Mientki
    >>

    >
    > I almost asked if you were making sure you had flushed the data to the
    > file...oh well.
    >

    Yes, that's a small disadavantage of using a "high-level" language,
    where there's no flush available, and you assume it'll done
    automatically ;-)

    cheers,
    Stef
    Stef Mientki, Jan 22, 2008
    #5
  6. Stef Mientki

    Jason Guest

    On Jan 22, 3:22 pm, Stef Mientki <> wrote:
    > Mike Driscoll wrote:
    > > On Jan 22, 3:35 pm, Stef Mientki <> wrote:

    >
    > >> Mike Driscoll wrote:

    >
    > >>> On Jan 17, 3:56 pm, Stef Mientki <> wrote:

    >
    > >>>> hello,

    >
    > >>>> I've a program (not written in Python) that generates a few thousands
    > >>>> bytes per second,
    > >>>> these files are dumped in 2 buffers (files), at in interval time of 50 msec,
    > >>>> the files can be read by another program, to do further processing.

    >
    > >>>> A program written in VB or delphi can handle the data in the 2 buffers
    > >>>> perfectly.
    > >>>> Sometimes Python is also able to process the data correctly,
    > >>>> but often it can't :-(

    >
    > >>>> I keep one of the files open en test the size of the open datafile each
    > >>>> 50 msec.
    > >>>> I have tried
    > >>>> os.stat ( ....) [ ST_SIZE]
    > >>>> os.path.getsize ( ... )
    > >>>> but they both have the same behaviour, sometimes it works, and the data
    > >>>> is collected each 50 .. 100 msec,
    > >>>> sometimes 1 .. 1.5 seconds is needed to detect a change in filesize.

    >
    > >>>> I'm using python 2.4 on winXP.

    >
    > >>>> Is there a solution for this problem ?

    >
    > >>>> thanks,
    > >>>> Stef Mientki

    >
    > >>> Tim Golden has a method to watch for changes in a directory on his
    > >>> website:

    >
    > >>>http://tgolden.sc.sabren.com/python/win32_how_do_i/watch_directory_fo...

    >
    > >>> This old post also mentions something similar:

    >
    > >>>http://mail.python.org/pipermail/python-list/2007-October/463065.html

    >
    > >>> And here's a cookbook recipe that claims to do it as well using
    > >>> decorators:

    >
    > >>>http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/426620

    >
    > >>> Hopefully that will get you going.

    >
    > >>> Mike

    >
    > >> thanks Mike,
    > >> sorry for the late reaction.
    > >> I've it working perfect now.
    > >> After all, os.stat works perfectly well,
    > >> the problem was in the program that generated the file with increasing
    > >> size,
    > >> by truncating it after each block write, it apperently garantees that
    > >> the file is flushed to disk and all problems are solved.

    >
    > >> cheers,
    > >> Stef Mientki

    >
    > > I almost asked if you were making sure you had flushed the data to the
    > > file...oh well.

    >
    > Yes, that's a small disadavantage of using a "high-level" language,
    > where there's no flush available, and you assume it'll done
    > automatically ;-)
    >
    > cheers,
    > Stef


    Uhm, there is a flush method for Python's files. From "http://
    docs.python.org/lib/bltin-file-objects.html":
    flush()
    Flush the internal buffer, like stdio's fflush(). This may
    be a no-op on some file-like objects.

    As for an example:

    >>> import os
    >>> f = open('vikings.txt', 'wb')
    >>> os.stat('vikings.txt').st_size

    0L
    >>> f.write('Spam, spam, spam, spam! ' * 1000) # Bloody vikings...
    >>> os.stat('vikings.txt').st_size

    24576L
    >>> f.flush()
    >>> os.stat('vikings.txt').st_size

    25000L
    >>>


    Is there something that I'm missing here?

    --Jason
    Jason, Jan 23, 2008
    #6
  7. Stef Mientki

    Stef Mientki Guest


    >>>> thanks Mike,
    >>>> sorry for the late reaction.
    >>>> I've it working perfect now.
    >>>> After all, os.stat works perfectly well,
    >>>> the problem was in the program that generated the file with increasing
    >>>> size,
    >>>> by truncating it after each block write, it apperently garantees that
    >>>> the file is flushed to disk and all problems are solved.
    >>>>
    >>>> cheers,
    >>>> Stef Mientki
    >>>>
    >>> I almost asked if you were making sure you had flushed the data to the
    >>> file...oh well.
    >>>

    >> Yes, that's a small disadavantage of using a "high-level" language,
    >> where there's no flush available, and you assume it'll done
    >> automatically ;-)
    >>
    >> cheers,
    >> Stef
    >>

    >
    > Uhm, there is a flush method for Python's files. From "http://
    > docs.python.org/lib/bltin-file-objects.html":
    > flush()
    > Flush the internal buffer, like stdio's fflush(). This may
    > be a no-op on some file-like objects.
    >
    > As for an example:
    >
    >
    >>>> import os
    >>>> f = open('vikings.txt', 'wb')
    >>>> os.stat('vikings.txt').st_size
    >>>>

    > 0L
    >
    >>>> f.write('Spam, spam, spam, spam! ' * 1000) # Bloody vikings...
    >>>> os.stat('vikings.txt').st_size
    >>>>

    > 24576L
    >
    >>>> f.flush()
    >>>> os.stat('vikings.txt').st_size
    >>>>

    > 25000L
    >
    >
    > Is there something that I'm missing here?
    >
    >

    hi Jason,
    I was talking about a "high-level" language, in which the sending
    program was written,
    (Delphi, not about Python ;-)
    cheers,
    Stef
    Stef Mientki, Jan 23, 2008
    #7
  8. En Wed, 23 Jan 2008 16:16:58 -0200, Stef Mientki <>
    escribió:

    >>> Yes, that's a small disadavantage of using a "high-level" language,
    >>> where there's no flush available, and you assume it'll done
    >>> automatically ;-)

    >>
    >> Uhm, there is a flush method for Python's files. From "http://
    >>

    > I was talking about a "high-level" language, in which the sending
    > program was written,
    > (Delphi, not about Python ;-)


    In Delphi, flush(filevar) does work. Or are you using a TFileStream or
    similar?

    --
    Gabriel Genellina
    Gabriel Genellina, Jan 23, 2008
    #8
  9. Stef Mientki

    Stef Mientki Guest

    Gabriel Genellina wrote:
    > En Wed, 23 Jan 2008 16:16:58 -0200, Stef Mientki
    > <> escribió:
    >
    >>>> Yes, that's a small disadavantage of using a "high-level" language,
    >>>> where there's no flush available, and you assume it'll done
    >>>> automatically ;-)
    >>>
    >>> Uhm, there is a flush method for Python's files. From "http://
    >>>

    >> I was talking about a "high-level" language, in which the sending
    >> program was written,
    >> (Delphi, not about Python ;-)

    >
    > In Delphi, flush(filevar) does work.

    flush is only valid for textfiles, not for binary files (at least in D7)

    anyway thanks, cheers,
    Stef
    Stef Mientki, Jan 24, 2008
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Pierre Vigneras
    Replies:
    1
    Views:
    767
    Andrew Hobbs
    Jan 24, 2004
  2. Replies:
    0
    Views:
    646
  3. Michele Simionato

    Python is darn fast (was: How fast is Python)

    Michele Simionato, Aug 23, 2003, in forum: Python
    Replies:
    13
    Views:
    551
  4. Juha Nieminen
    Replies:
    22
    Views:
    994
    Kai-Uwe Bux
    Oct 12, 2007
  5. Ralf
    Replies:
    4
    Views:
    419
Loading...

Share This Page