Re: Fast forward-backward (write-read)

Discussion in 'Python' started by emile, Oct 24, 2012.

  1. emile

    emile Guest

    On 10/23/2012 04:19 PM, David Hutto wrote:
    > Whether this is fast enough, or not, I don't know:


    well, the OP's original post started with
    "I am working with some rather large data files (>100GB)..."

    > filename = "data_file.txt"
    > f = open(filename, 'r')
    > forward = [line.rstrip('\n') for line in f.readlines()]


    f.readlines() will be big(!) and have overhead... and forward results in
    something again as big.

    > backward = [line.rstrip('\n') for line in reversed(forward)]


    and defining backward looks to me to require space to build backward and
    hold reversed(forward)

    So, let's see, at that point in time (building backward) you've got
    probably somewhere close to 400-500Gb in memory.

    My guess -- probably not so fast. Thrashing is sure to be a factor on
    all but machines I'll never have a chance to work on.


    > f.close()
    > print forward, "\n\n", "********************\n\n", backward, "\n"



    It's good to retain context.

    Emile
    emile, Oct 24, 2012
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Virgil Stokes

    Fast forward-backward (write-read)

    Virgil Stokes, Oct 23, 2012, in forum: Python
    Replies:
    7
    Views:
    140
    Virgil Stokes
    Oct 25, 2012
  2. Tim Chase
    Replies:
    0
    Views:
    168
    Tim Chase
    Oct 23, 2012
  3. Dennis Lee Bieber

    Re: Fast forward-backward (write-read)

    Dennis Lee Bieber, Oct 23, 2012, in forum: Python
    Replies:
    0
    Views:
    128
    Dennis Lee Bieber
    Oct 23, 2012
  4. Virgil Stokes

    Re: Fast forward-backward (write-read)

    Virgil Stokes, Oct 23, 2012, in forum: Python
    Replies:
    4
    Views:
    180
    Tim Golden
    Oct 24, 2012
  5. Virgil Stokes

    Re: Fast forward-backward (write-read)

    Virgil Stokes, Oct 23, 2012, in forum: Python
    Replies:
    0
    Views:
    104
    Virgil Stokes
    Oct 23, 2012
Loading...

Share This Page