Q
qwertmonkey
From: (e-mail address removed)
~
Well, yes. I am not sure if I am right (or the extent to which ...),
but it seems I am dealing here with a physical barrier, that may not be
friendly to "logical" ways. Unless the original file is split into and placed
in 8 different physical disks, the data transfer does not find bottlenecks
through the memory subsystem, ...
~
While reading the characters by means of a (NIO 2 Files.new)BufferedReader
(which uses non-blocking I/O, right?) and parsing the sentences myself, I can
read the whole file on a laptop based on an AMD Turion 64X2 Dual Core and 4Gb
in less than 10 seconds
~
BufferedReader BfR = Files.newBufferedReader(IFlPth,
Charset.forName("UTF-8"));
char[] cBfr = new char[iBL];
// __
int iRdByts = BfR.read(cBfr, iBfr00, iBfr02);
while(iRdByts > -1){
for(int i = iBfr00; (i < iRdByts); ++i){
if(cBfr == cLF){ ++lLns; }
}
iRdByts = BfR.read(cBfr, iBfr00, iBfr02);
}// (iRdByts > -1)
// __
BfR.close();
~
I still have to code the logic on top of it, but that is much, much better
than it was taking before
~
Please, let me know if you have a better idea of how to read very large text
files in a faster way (while using a regular/sub-optimal box) ~
lbrtchx
--- BBBS/Li6 v4.10 Dada-1
* Origin: Prism bbs (1:261/38)
--- Synchronet 3.16a-Win32 NewsLink 1.98
Time Warp of the Future BBS - telnet://time.synchro.net:24
~
Well, yes. I am not sure if I am right (or the extent to which ...),
but it seems I am dealing here with a physical barrier, that may not be
friendly to "logical" ways. Unless the original file is split into and placed
in 8 different physical disks, the data transfer does not find bottlenecks
through the memory subsystem, ...
~
While reading the characters by means of a (NIO 2 Files.new)BufferedReader
(which uses non-blocking I/O, right?) and parsing the sentences myself, I can
read the whole file on a laptop based on an AMD Turion 64X2 Dual Core and 4Gb
in less than 10 seconds
~
BufferedReader BfR = Files.newBufferedReader(IFlPth,
Charset.forName("UTF-8"));
char[] cBfr = new char[iBL];
// __
int iRdByts = BfR.read(cBfr, iBfr00, iBfr02);
while(iRdByts > -1){
for(int i = iBfr00; (i < iRdByts); ++i){
if(cBfr == cLF){ ++lLns; }
}
iRdByts = BfR.read(cBfr, iBfr00, iBfr02);
}// (iRdByts > -1)
// __
BfR.close();
~
I still have to code the logic on top of it, but that is much, much better
than it was taking before
~
Please, let me know if you have a better idea of how to read very large text
files in a faster way (while using a regular/sub-optimal box) ~
lbrtchx
--- BBBS/Li6 v4.10 Dada-1
* Origin: Prism bbs (1:261/38)
--- Synchronet 3.16a-Win32 NewsLink 1.98
Time Warp of the Future BBS - telnet://time.synchro.net:24