Re: Fast forward-backward (write-read)

Discussion in 'Python' started by Virgil Stokes, Oct 23, 2012.

  1. On 23-Oct-2012 18:09, Tim Chase wrote:
    > On 10/23/12 09:31, Virgil Stokes wrote:
    >> I am working with some rather large data files (>100GB) that contain time series
    >> data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I perform
    >> various types of processing on these data (e.g. moving median, moving average,
    >> and Kalman-filter, Kalman-smoother) in a sequential manner and only a small
    >> number of these data need be stored in RAM when being processed. When performing
    >> Kalman-filtering (forward in time pass, k = 0,1,...,N) I need to save to an
    >> external file several variables (e.g. 11*32 bytes) for each (t_k, y(t_k)). These
    >> are inputs to the Kalman-smoother (backward in time pass, k = N,N-1,...,0).
    >> Thus, I will need to input these variables saved to an external file from the
    >> forward pass, in reverse order --- from last written to first written.
    >>
    >> Finally, to my question --- What is a fast way to write these variables to an
    >> external file and then read them in backwards?

    > Am I missing something, or would the fairly-standard "tac" utility
    > do the reversal you want? It should[*] be optimized to handle
    > on-disk files in a smart manner.

    Not sure about "tac" --- could you provide more details on this and/or a simple
    example of how it could be used for fast reversed "reading" of a data file?
    >
    > Otherwise, if you can pad the record-lengths so they're all the
    > same, and you know the total number of records, you can seek to
    > Total-(RecSize*OneBasedOffset) and write the record,optionally
    > padding if you need/can. At least on *nix-like OSes, you can seek
    > into a sparse-file with no problems (untested on Win32).

    The records lengths will all be the same and yes seek could be used; but, I was
    hoping for a faster method.

    Thanks Tim! :)
    >
    > -tkc
    >
    >
    >
    > [*]
    > Just guessing here. Would be disappointed if it *wasn't*.
    Virgil Stokes, Oct 23, 2012
    #1
    1. Advertising

  2. Virgil Stokes wrote:

    > Not sure about "tac" --- could you provide more details on this
    > and/or a simple example of how it could be used for fast reversed
    > "reading" of a data file ?


    tac is available as a command under linux ....

    $ whatis tac
    tac (1) - concatenate and print files in reverse

    $ whereis tac
    tac: /usr/bin/tac /usr/bin/X11/tac /usr/share/man/man1/tac.1.gz

    $ man tac

    SYNOPSIS
    tac [OPTION]... [FILE]...

    DESCRIPTION

    Write each FILE to standard output, last line first.

    With no FILE, or when FILE is -, read standard input.


    I only know that the tac command exists
    but have never used it myself ....


    --
    Stanley C. Kitching
    Human Being
    Phoenix, Arizona
    Cousin Stanley, Oct 23, 2012
    #2
    1. Advertising

  3. On Tue, 23 Oct 2012 20:03:39 +0000 (UTC), Cousin Stanley
    <> declaimed the following in
    gmane.comp.python.general:

    >
    > $ whatis tac
    > tac (1) - concatenate and print files in reverse
    >


    Oh... a backwards "cat"
    --
    Wulfraed Dennis Lee Bieber AF6VN
    HTTP://wlfraed.home.netcom.com/
    Dennis Lee Bieber, Oct 24, 2012
    #3
  4. On 23-Oct-2012 22:03, Cousin Stanley wrote:
    > Virgil Stokes wrote:
    >
    >> Not sure about "tac" --- could you provide more details on this
    >> and/or a simple example of how it could be used for fast reversed
    >> "reading" of a data file ?

    > tac is available as a command under linux ....
    >
    > $ whatis tac
    > tac (1) - concatenate and print files in reverse
    >
    > $ whereis tac
    > tac: /usr/bin/tac /usr/bin/X11/tac /usr/share/man/man1/tac.1.gz
    >
    > $ man tac
    >
    > SYNOPSIS
    > tac [OPTION]... [FILE]...
    >
    > DESCRIPTION
    >
    > Write each FILE to standard output, last line first.
    >
    > With no FILE, or when FILE is -, read standard input.
    >
    >
    > I only know that the tac command exists
    > but have never used it myself ....
    >
    >

    Unfortunately, I may be forced to process the data on a Windows platform; but,
    thanks Cousin for the Linux tip.
    Virgil Stokes, Oct 24, 2012
    #4
  5. Virgil Stokes

    Tim Golden Guest

    On 24/10/2012 08:07, Virgil Stokes wrote:
    > On 23-Oct-2012 22:03, Cousin Stanley wrote:
    >> Virgil Stokes wrote:
    >>
    >>> Not sure about "tac" --- could you provide more details on this
    >>> and/or a simple example of how it could be used for fast reversed
    >>> "reading" of a data file ?

    >> tac is available as a command under linux ....
    >>
    >> $ whatis tac
    >> tac (1) - concatenate and print files in reverse
    >>
    >> $ whereis tac
    >> tac: /usr/bin/tac /usr/bin/X11/tac /usr/share/man/man1/tac.1.gz
    >>
    >> $ man tac
    >>
    >> SYNOPSIS
    >> tac [OPTION]... [FILE]...
    >>
    >> DESCRIPTION
    >>
    >> Write each FILE to standard output, last line first.
    >>
    >> With no FILE, or when FILE is -, read standard input.
    >>
    >>
    >> I only know that the tac command exists
    >> but have never used it myself ....
    >>
    >>

    > Unfortunately, I may be forced to process the data on a Windows
    > platform; but, thanks Cousin for the Linux tip.


    Well, addressing that specific point, tac is available for Windows:

    http://unxutils.sourceforge.net/

    No idea how efficient it is...

    TJG
    Tim Golden, Oct 24, 2012
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Virgil Stokes

    Fast forward-backward (write-read)

    Virgil Stokes, Oct 23, 2012, in forum: Python
    Replies:
    7
    Views:
    147
    Virgil Stokes
    Oct 25, 2012
  2. Tim Chase
    Replies:
    0
    Views:
    175
    Tim Chase
    Oct 23, 2012
  3. Dennis Lee Bieber

    Re: Fast forward-backward (write-read)

    Dennis Lee Bieber, Oct 23, 2012, in forum: Python
    Replies:
    0
    Views:
    134
    Dennis Lee Bieber
    Oct 23, 2012
  4. Virgil Stokes

    Re: Fast forward-backward (write-read)

    Virgil Stokes, Oct 23, 2012, in forum: Python
    Replies:
    0
    Views:
    111
    Virgil Stokes
    Oct 23, 2012
  5. Tim Chase
    Replies:
    0
    Views:
    186
    Tim Chase
    Oct 23, 2012
Loading...

Share This Page