S
Sean Davis
This should be a relatively simple problem, but I haven't quite got
the idea of how to go about it. I have a VERY large file that I would
like to load a line at a time, do some manipulations on it, and then
make it available to as a file-like object for use as input to a
database module (psycopg2) that wants a file-like object (with read
and readlines methods). I could write the manipulated file out to
disk and then read it back in, but that seems wasteful. So, it seems
like I need a buffer, a way to fill the buffer and a way to have read
and readlines use the buffer. What I can't do is to load the ENTIRE
file into a stringio object, as the file is much too large. Any
suggestions?
Thanks,
Sean
the idea of how to go about it. I have a VERY large file that I would
like to load a line at a time, do some manipulations on it, and then
make it available to as a file-like object for use as input to a
database module (psycopg2) that wants a file-like object (with read
and readlines methods). I could write the manipulated file out to
disk and then read it back in, but that seems wasteful. So, it seems
like I need a buffer, a way to fill the buffer and a way to have read
and readlines use the buffer. What I can't do is to load the ENTIRE
file into a stringio object, as the file is much too large. Any
suggestions?
Thanks,
Sean