Re: Python does not take up available physical memory

Discussion in 'Python' started by Chris Angelico, Oct 20, 2012.

  1. On Sat, Oct 20, 2012 at 9:22 AM, Pradipto Banerjee
    <> wrote:
    > Dennis,
    >
    > 1. Yes, .readlines() work where .read() fails. Thanks for the suggestion - this has really given a big boost to the size of the data I can read.


    If at all possible, consider reading the file iteratively and
    retaining only the current line. You say you need to "compare one line
    versus another" - are you able, for instance, to sort the file
    (externally; I think every OS these days will have a sort utility that
    can handle files >RAM) and then compare the current line against a
    small amount of retained state? This sort (pun intended) of trick can
    not only get you past the limit, it will often improve running time
    enormously too.

    ChrisA
    Chris Angelico, Oct 20, 2012
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Emile van Sebille
    Replies:
    0
    Views:
    235
    Emile van Sebille
    Oct 20, 2012
  2. Ian Kelly
    Replies:
    0
    Views:
    150
    Ian Kelly
    Oct 19, 2012
  3. Pradipto Banerjee
    Replies:
    1
    Views:
    132
    Steven D'Aprano
    Oct 19, 2012
  4. Pradipto Banerjee
    Replies:
    6
    Views:
    228
    Pradipto Banerjee
    Oct 21, 2012
  5. MRAB
    Replies:
    0
    Views:
    118
Loading...

Share This Page