Can i improve this function?i need your help...

[gbattine]

Hi guys,
i'm a pratical question for you.
I've developed a jas application that reads a txt file and convert it
into an array of byte,fro putting it into a blob field of a db mysql.
With big file i go in heap size memory!
I want to improve my function that makes it,but i'm a bit
inexpert...can you help me?
I have a file so made...
Each line has the same number of value,but i know this number only when
i read the file

string1 string2 string3................
stringx double double......(ever double)
stringy double double......(ever double)

and other lines equals to them from the second.
Only the first line has only strings.
I have read this file using a string array to read the first line and
an arraylist of "Riga" object to read the others lines.....i've used a
vector to store partial data from lines reading and at the end i've
created an array of byte in which i copy the vector.
Can you help me improving my code?
The file has about 50000 rows......so i go in heap memory size...
i know my code isn't too optimized.....can you help me?Please,i'm a
newbie,help me with code if possible...
When i encode the objects i've created into an array of byte i use a
whitespace to separe different value and a ; to separe different lines.
I've done it because in a second moment i've to read the array of byte.
Thanks...this is my code

public class MyBean {
private UploadedFile myFile;
private ArrayList rows = new ArrayList();

private List lines = new ArrayList();

public MyBean(){

};
public List getLines(){
return lines;
}
public void setLines(List lines){
this.lines=lines;
}
public boolean insRighe(Riga nuovo){
return rows.add(nuovo);
}
public UploadedFile getMyFile() {
return myFile;
}
public void setMyFile(UploadedFile myFile) {
this.myFile = myFile;
}

public ArrayList getRows() {
return rows;
}
public void setRows(ArrayList rows) {
this.rows = rows;
}

public String carica() throws IOException {
Riga r;
Double val[];
Head h;
int col=0;
int row=0;
byte middlerow=' ';
byte endrow=';';
byte[] data=null;
Vector temp=new Vector();
int numberOfNumericColumns=0;
String geneid=null;
String g=null;
String[]intest=null;
BufferedReader br = new BufferedReader(new
InputStreamReader(myFile.getInputStream()));

String line = null;
while ((line = br.readLine()) != null) {
line = line.replace (',', '.');
StringTokenizer st = new StringTokenizer(line);
numberOfNumericColumns = (st.countTokens()-1);
col=(numberOfNumericColumns+1);

//se siamo nella prima riga(contatore segna 0)
if(row==0){
intest=new String[col];
int j=0;
while(st.hasMoreTokens()){
intest[j]=(st.nextToken().trim());
j++;
}
h=new Head(intest);//crei l'oggetto head
String []qa=h.getHvalues();
String asd="";
for(int i=0;i<=qa.length-1;i++){
asd=asd.concat(qa+" ");
}
System.out.println("head "+asd);//stampo contenuto
dell' head
row=1;
}//fine if

else
{
Double[] values=new Double[numberOfNumericColumns];
int z=0;
geneid=st.nextToken();
while (st.hasMoreTokens()) {
String app=st.nextToken();
values[z]=Double.valueOf(app);
z++;
}
r=new Riga(geneid,values); //crei l'oggetto riga
System.out.println("riga");
System.out.println(r.getgeneid());
values=r.getvalues();
for(int e=0;e<values.length;e++){
System.out.println(values[e]);
}
insRighe(r); //aggiungi
}
row++;
}
int i = 0;
while (i < intest.length) {

byte[] bytesnew = intest.getBytes();
// temp.addAll(bytesnew);
// memorizza in byte un elemento del vettore alla volta

for (byte b : bytesnew)
temp.add(new Byte(b)); // provare Byte
// temp.addElement(intest.getBytes());
temp.addElement(Byte.valueOf(middlerow));
i++;
}
temp.addElement(Byte.valueOf(endrow));
System.out.println("Intestazione convertita in byte");

for (int l = 0; l < rows.size(); l++) {
r = (Riga) rows.get(l);
g = r.getgeneid();
// temp.addElement(g.getBytes());

byte[] byte2 = g.getBytes();
for (byte c : byte2)
temp.add(new Byte(c));

temp.addElement(Byte.valueOf(middlerow));
val = r.getvalues();

byte[] tempByte1;
for (int e = 0; e <= val.length - 1; e++) {
// Returns a string representation of the double argument.
tempByte1 = Double.toString(val[e]).getBytes();

for (int j = 0; j < tempByte1.length; j++) {
temp.addElement(Byte.valueOf(tempByte1[j]));

}
temp.addElement(Byte.valueOf(middlerow));
}
temp.addElement(Byte.valueOf(endrow));
}
data = new byte[temp.size()];

for (int t = 0; t < temp.size(); t++) {
data[t] = (((Byte) temp.elementAt(t)).byteValue());

}

return data;
}

 

[gbattine]

this is my Riga object

public class Riga{
private String geneid=null;
private Double[] values=null;

public Riga(String idGene,Double[] x ) {
this.geneid=idGene;
this.values=x;
}
public String getgeneid(){
return this.geneid;
}
public void setgeneid(String idGene){
this.geneid=idGene;
}
public Double[] getvalues(){
return this.values;
}
public void setvalues(Double[] x){
this.values=x;
}}

 

[Philipp Leitner]

Hi guys,
i'm a pratical question for you.
I've developed a jas application that reads a txt file and convert it
into an array of byte,fro putting it into a blob field of a db mysql.

Hi,

why do you want to read in a txt file as /binary/ object, and store it
to a BLOB? Would'nt it make more sense to keep interpreting the text
data as character array, and store them to a CLOB or Text field in the
DB?

Anyway, if you really want to read a file as bit stream you could go
for something like:

<code>
File f = new File("pathToFile");
byte[] mybytes = new byte[(int)f.length()];
InputStream in = new FileInputStream(f);
in.read(mybytes);
</code>

That is extremely bad optimized as well, but it should at least work
(the binary data is in the byte[] mybytes afterwards). I guess with a
quick Google search you can improve the solution (look for 'buffered
reading of binary data' ... ).

/philipp

 

[dsjoblom]

<code>
File f = new File("pathToFile");
byte[] mybytes = new byte[(int)f.length()];
InputStream in = new FileInputStream(f);
in.read(mybytes);
</code>

That is extremely bad optimized as well, but it should at least work
(the binary data is in the byte[] mybytes afterwards). I guess with a
quick Google search you can improve the solution (look for 'buffered
reading of binary data' ... ).

Actually, this is one of the quicker ways of reading a file into memory
in one shot. There is no reason to wrap the FileInputStream in a
BufferedInputStream for that, it will only delegate the work back to
FileInputStream (but this *only* applies to this particular scenario.
If you are reading in small increments, BufferedInputStream is a must.)
It is still slightly faster (if the file is large) to map the file into
memory with NIO and convert it to a byte array, but I can't recommend
this for general use, since there is no way to unmap the file after
reading it (and converting an mmapped file into a byte array just seems
wrong). And for large data sets, reading everything into memory is not
a good idea anyhow.

Regards,
Daniel Sjöblom

 

[gbattine]

thanks very much for your helps...
i have a question for you,excuse me my inexperience.
I'v often the error java heap size, i think because my array of byte
beacame very big....
in my function

temp is the vector that stores each line
and data is the array of byte.

During

data = new byte[temp.size()];

for (int t = 0; t < temp.size(); t++) {
data[t] = (((Byte) temp.elementAt(t)).byteValue());

}
}

where i pass the vector in the array of bytes it happens that the array
of byte became too large.
How can solve this problem without modify the rest of code?
Storing the vector in more array of byte or writing it to a file can
solve my

java heap size problem?

Please help me, i'm not very happy to modify all my precedent code, i
hope there is some way to solve the problem.....
i'm waiting for you help
Thanks

 

[gbattine]

Thanks very much,
i'm not very happy to modify my precedent code,so i hope to find a
solution to my problem.
In my function

temp is the vector in which i store the file line to line and data is
the array of byte i want to send in blob field.

data = new byte[temp.size()];

for (int t = 0; t < temp.size(); t++) {
data[t] = (((Byte) temp.elementAt(t)).byteValue());

}

with this operation i copy the vector in the array of byte and with big
file i have java heap size error.

Can i solve my problem changing only these code lines?
Can i solve the problem splitting the vector in more array of byte o
writing them into files?
Can you help me,please?
I'm waiting for your helps..

 

[Philipp Leitner]

temp is the vector that stores each line
and data is the array of byte.

During

data = new byte[temp.size()];

for (int t = 0; t < temp.size(); t++) {
data[t] = (((Byte) temp.elementAt(t)).byteValue());

}
}

where i pass the vector in the array of bytes it happens that the array
of byte became too large.
How can solve this problem without modify the rest of code?
Storing the vector in more array of byte or writing it to a file can
solve my

java heap size problem?

Please help me, i'm not very happy to modify all my precedent code, i
hope there is some way to solve the problem.....
i'm waiting for you help
Thanks

Well, I'd say if you /should/ modfiy the rest of the code, too ... it
just does not look good to me to store binary data in a Vector of Bytes
.... see, a simple 'byte' in a byte[] should take exactly /1 byte of
memory/ , but AFAIK a Byte in a Vector is a Java object reference and
should take at least 8 bytes of memory, and also (compared to
constructing of a simple 'byte') ages to be instanciated. It is no
wonder to me that you run out of heap space quickly...

If you desparately need a way to work around the problem without
changing your code you can increase the heap size of the JVM ... but
that may just temporarly solve your problem at hand, and not the actual
root problem ... see for instance
http://forum.java.sun.com/thread.jspa?threadID=490356&tstart=0 on how
to change the heap size.

/philipp

 

[gbattine]

well,
thanks to your help i've decided to change my code, but now i'm getting
in panic....
can you give some helps or useful links?
I need to create object Riga to store the lines other than the first?
Please help me and thanks very much...

P.S. When i start to develop this function i've used a vector type
because it can dynamically grows up where byte type no...how can i do
now?
I'm bit inexpert about files and byte...

 

[Philipp Leitner]

well,
thanks to your help i've decided to change my code, but now i'm getting
in panic....
can you give some helps or useful links?
I need to create object Riga to store the lines other than the first?
Please help me and thanks very much...

P.S. When i start to develop this function i've used a vector type
because it can dynamically grows up where byte type no...how can i do
now?
I'm bit inexpert about files and byte...

Sorry, I lost you somewhere ... what has the 'Riga' object that you
posted earlier to do with all that? I cannot see any connection to the
byte reading issue we are talking about here.

Generally - why do you care about 'lines' when you want to read binary
data? If you treat the text file as binary data then it is just a chump
of bytes - to line ends, no characters, no whatever, just 0s and 1s ...

/philipp