Re: Fast forward-backward (write-read)

Discussion in 'Python' started by Tim Chase, Oct 23, 2012.

  1. Tim Chase

    Tim Chase Guest

    On 10/23/12 09:31, Virgil Stokes wrote:
    > I am working with some rather large data files (>100GB) that contain time series
    > data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I perform
    > various types of processing on these data (e.g. moving median, moving average,
    > and Kalman-filter, Kalman-smoother) in a sequential manner and only a small
    > number of these data need be stored in RAM when being processed. When performing
    > Kalman-filtering (forward in time pass, k = 0,1,...,N) I need to save to an
    > external file several variables (e.g. 11*32 bytes) for each (t_k, y(t_k)). These
    > are inputs to the Kalman-smoother (backward in time pass, k = N,N-1,...,0).
    > Thus, I will need to input these variables saved to an external file from the
    > forward pass, in reverse order --- from last written to first written.
    >
    > Finally, to my question --- What is a fast way to write these variables to an
    > external file and then read them in backwards?


    Am I missing something, or would the fairly-standard "tac" utility
    do the reversal you want? It should[*] be optimized to handle
    on-disk files in a smart manner.

    Otherwise, if you can pad the record-lengths so they're all the
    same, and you know the total number of records, you can seek to
    Total-(RecSize*OneBasedOffset) and write the record,optionally
    padding if you need/can. At least on *nix-like OSes, you can seek
    into a sparse-file with no problems (untested on Win32).

    -tkc



    [*]
    Just guessing here. Would be disappointed if it *wasn't*.
    Tim Chase, Oct 23, 2012
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Virgil Stokes

    Fast forward-backward (write-read)

    Virgil Stokes, Oct 23, 2012, in forum: Python
    Replies:
    7
    Views:
    148
    Virgil Stokes
    Oct 25, 2012
  2. Dennis Lee Bieber

    Re: Fast forward-backward (write-read)

    Dennis Lee Bieber, Oct 23, 2012, in forum: Python
    Replies:
    0
    Views:
    134
    Dennis Lee Bieber
    Oct 23, 2012
  3. Virgil Stokes

    Re: Fast forward-backward (write-read)

    Virgil Stokes, Oct 23, 2012, in forum: Python
    Replies:
    4
    Views:
    189
    Tim Golden
    Oct 24, 2012
  4. Virgil Stokes

    Re: Fast forward-backward (write-read)

    Virgil Stokes, Oct 23, 2012, in forum: Python
    Replies:
    0
    Views:
    111
    Virgil Stokes
    Oct 23, 2012
  5. Tim Chase
    Replies:
    0
    Views:
    186
    Tim Chase
    Oct 23, 2012
Loading...

Share This Page