beginner: Copy text file to memory

[John]

Hello,

I am trying to read the lines in a text file using:
BufferedReader inputStream = new BufferedReader(new
FileReader(inputFileName)) and readLine().

I want to read the file several times, but reading from a file is
inefficient.

Is there a way to copy the text file to memory first and then create a
stream that reads the lines of text file from the memory.

Thanks in advance



email:
shahint at shaw ca

 

[Andy Flowers]

Create, as an example, an ArrayList and read each line into it.

i.e.

ArrayList lines = new ArrayList();
BufferedReader inputStream = new BufferedReader(new
FileReader(inputFileName));

String line;
while( (line = inputStream.readLine()) != null )
{
lines.add( line );
}


now the file is buffered in lines.

You could wrap all of this up i a new class, say a CachedTextFileReader
class and read the actual file the first time a client calls a method that
requires the file to have been cached, a lazy read system. Methods that
would be useful in such a class would include readLine() both without a
parameter and witha line number parameter, capacity/count, reset, seek etc.

 

[Marco Schmidt]

John:

[...]

I want to read the file several times, but reading from a file is
inefficient.

No, that's too generalized. If you use Linux, Windows NT (or one of
its successors), or any other modern operating system, file system
caching goes a long way. So check if you really have a bottleneck
before you start optimizing.

Chances are that if the file is small, it requires almost no time to
read it, once, ten or a hundred times. If the file is big, it may not
fit into memory.

Is there a way to copy the text file to memory first and then create a
stream that reads the lines of text file from the memory.

You could read the file into a StringBuffer, create a String from it
and create a StringReader from that String. If you keep the String,
you can create readers from it as often as you need.

However, maybe there is a better approach for what you are doing than
reading a file several times. Maybe a database. Care to elaborate on
the problem?

Regards,
Marco

 

[Jon A. Cruz]

Hello,

I am trying to read the lines in a text file using:
BufferedReader inputStream = new BufferedReader(new
FileReader(inputFileName)) and readLine().

I want to read the file several times, but reading from a file is
inefficient.

Who says so?

Have you tested it?

By having that BufferedReader in there your program can be fairly fast.

One quick tool I did not too long ago read and parsed one thousand XML
files in 3 seconds, including the XML parse time.

Is there a way to copy the text file to memory first and then create a
stream that reads the lines of text file from the memory.

Yes.. but there are other ways.


First of all, you should probably not use a FileReader. Instead,
construct an InputStreamReader on a BufferedInputStream on a
FileInputStream. You avoid all sorts of subtle bugs by explicitly
stating the encoding to be used for conversions.



Anyway, back to your memory question. Guess what a
BufferedReader/BufferedInputStream does? Uses memory to do the reading.




So... we come to a couple of questions:

1) Why do you think reading the file is inefficient? What do your
performance measurements show?

2) Why are you wanting to read it several times? (Treating it as bytes
or characters might depend on your answer)

 

[Floppy]

File f = new File( fileName );
FileInputStream fis = new FileInputStream( f );
int len = (int) f.length();
byte buf[] = new byte[len];
int offset = 0;
int rc = 0;
while( offset < len ) {
rc = fis.read( buf, offset, len - offset );
offset += rc;

You read the entire file into a byte[] and then construct
a ByteArrayInputStream with your array as the backing store.
Coax the InputStream into the required class and go to town.
Important note: I narrow the file length from long to int
so you're looking at a file size limitation of 2 GB.

Thanks for the code.
Would you be able to explain the first part of the code for me (until the
end of the while loop) in detail?

Again thank you very much

 

[Marco Schmidt]

Floppy:

Would you be able to explain the first part of the code for me (until the
end of the while loop) in detail?

What about Jon's and my doubts about the necessity of this
micro-optimization? What file sizes are we talking about? What are you
trying to accomplish?

Regards,
Marco

 

[Sudsy]

Floppy wrote:

Thanks for the code.
Would you be able to explain the first part of the code for me (until the
end of the while loop) in detail?

Again thank you very much

We need to construct a File object in order to know the size.
We then use that size as the dimension of the byte array.
Since we can't be guaranteed that a read with a particular
size will always return the desired number of bytes, we have
to continue to read until we've received all the data.
That's why we use an offset variable and the return from the
read to increment the offset into the array. The size - offset
is merely the number of bytes we haven't yet read.
The code supplied is only intended to be a starting point.
Note that there is no invocation of exists() on the File
object or a test for a return of -1 from the read which would
indicate that you're trying to read beyond end-of-file.
Neither of these case should occur but it never hurts to
check for the unlikely or impossible.