Can i improve this function?i need your help...

Discussion in 'Java' started by gbattine, Jul 15, 2006.

  1. gbattine

    gbattine Guest

    Hi guys,
    i'm a pratical question for you.
    I've developed a jas application that reads a txt file and convert it
    into an array of byte,fro putting it into a blob field of a db mysql.
    With big file i go in heap size memory!
    I want to improve my function that makes it,but i'm a bit
    inexpert...can you help me?
    I have a file so made...
    Each line has the same number of value,but i know this number only when
    i read the file

    string1 string2 string3................
    stringx double double......(ever double)
    stringy double double......(ever double)

    and other lines equals to them from the second.
    Only the first line has only strings.
    I have read this file using a string array to read the first line and
    an arraylist of "Riga" object to read the others lines.....i've used a
    vector to store partial data from lines reading and at the end i've
    created an array of byte in which i copy the vector.
    Can you help me improving my code?
    The file has about 50000 rows......so i go in heap memory size...
    i know my code isn't too optimized.....can you help me?Please,i'm a
    newbie,help me with code if possible...
    When i encode the objects i've created into an array of byte i use a
    whitespace to separe different value and a ; to separe different lines.
    I've done it because in a second moment i've to read the array of byte.
    Thanks...this is my code

    public class MyBean {
    private UploadedFile myFile;
    private ArrayList rows = new ArrayList();

    private List lines = new ArrayList();

    public MyBean(){

    };
    public List getLines(){
    return lines;
    }
    public void setLines(List lines){
    this.lines=lines;
    }
    public boolean insRighe(Riga nuovo){
    return rows.add(nuovo);
    }
    public UploadedFile getMyFile() {
    return myFile;
    }
    public void setMyFile(UploadedFile myFile) {
    this.myFile = myFile;
    }

    public ArrayList getRows() {
    return rows;
    }
    public void setRows(ArrayList rows) {
    this.rows = rows;
    }

    public String carica() throws IOException {
    Riga r;
    Double val[];
    Head h;
    int col=0;
    int row=0;
    byte middlerow=' ';
    byte endrow=';';
    byte[] data=null;
    Vector temp=new Vector();
    int numberOfNumericColumns=0;
    String geneid=null;
    String g=null;
    String[]intest=null;
    BufferedReader br = new BufferedReader(new
    InputStreamReader(myFile.getInputStream()));

    String line = null;
    while ((line = br.readLine()) != null) {
    line = line.replace (',', '.');
    StringTokenizer st = new StringTokenizer(line);
    numberOfNumericColumns = (st.countTokens()-1);
    col=(numberOfNumericColumns+1);

    //se siamo nella prima riga(contatore segna 0)
    if(row==0){
    intest=new String[col];
    int j=0;
    while(st.hasMoreTokens()){
    intest[j]=(st.nextToken().trim());
    j++;
    }
    h=new Head(intest);//crei l'oggetto head
    String []qa=h.getHvalues();
    String asd="";
    for(int i=0;i<=qa.length-1;i++){
    asd=asd.concat(qa+" ");
    }
    System.out.println("head "+asd);//stampo contenuto
    dell' head
    row=1;
    }//fine if

    else
    {
    Double[] values=new Double[numberOfNumericColumns];
    int z=0;
    geneid=st.nextToken();
    while (st.hasMoreTokens()) {
    String app=st.nextToken();
    values[z]=Double.valueOf(app);
    z++;
    }
    r=new Riga(geneid,values); //crei l'oggetto riga
    System.out.println("riga");
    System.out.println(r.getgeneid());
    values=r.getvalues();
    for(int e=0;e<values.length;e++){
    System.out.println(values[e]);
    }
    insRighe(r); //aggiungi
    }
    row++;
    }
    int i = 0;
    while (i < intest.length) {

    byte[] bytesnew = intest.getBytes();
    // temp.addAll(bytesnew);
    // memorizza in byte un elemento del vettore alla volta

    for (byte b : bytesnew)
    temp.add(new Byte(b)); // provare Byte
    // temp.addElement(intest.getBytes());
    temp.addElement(Byte.valueOf(middlerow));
    i++;
    }
    temp.addElement(Byte.valueOf(endrow));
    System.out.println("Intestazione convertita in byte");

    for (int l = 0; l < rows.size(); l++) {
    r = (Riga) rows.get(l);
    g = r.getgeneid();
    // temp.addElement(g.getBytes());

    byte[] byte2 = g.getBytes();
    for (byte c : byte2)
    temp.add(new Byte(c));

    temp.addElement(Byte.valueOf(middlerow));
    val = r.getvalues();

    byte[] tempByte1;
    for (int e = 0; e <= val.length - 1; e++) {
    // Returns a string representation of the double argument.
    tempByte1 = Double.toString(val[e]).getBytes();

    for (int j = 0; j < tempByte1.length; j++) {
    temp.addElement(Byte.valueOf(tempByte1[j]));

    }
    temp.addElement(Byte.valueOf(middlerow));
    }
    temp.addElement(Byte.valueOf(endrow));
    }
    data = new byte[temp.size()];

    for (int t = 0; t < temp.size(); t++) {
    data[t] = (((Byte) temp.elementAt(t)).byteValue());

    }

    return data;
    }
    gbattine, Jul 15, 2006
    #1
    1. Advertising

  2. gbattine

    gbattine Guest

    this is my Riga object

    public class Riga{
    private String geneid=null;
    private Double[] values=null;

    public Riga(String idGene,Double[] x ) {
    this.geneid=idGene;
    this.values=x;
    }
    public String getgeneid(){
    return this.geneid;
    }
    public void setgeneid(String idGene){
    this.geneid=idGene;
    }
    public Double[] getvalues(){
    return this.values;
    }
    public void setvalues(Double[] x){
    this.values=x;
    }}
    gbattine, Jul 15, 2006
    #2
    1. Advertising

  3. gbattine wrote:
    > Hi guys,
    > i'm a pratical question for you.
    > I've developed a jas application that reads a txt file and convert it
    > into an array of byte,fro putting it into a blob field of a db mysql.


    Hi,

    why do you want to read in a txt file as /binary/ object, and store it
    to a BLOB? Would'nt it make more sense to keep interpreting the text
    data as character array, and store them to a CLOB or Text field in the
    DB?

    Anyway, if you really want to read a file as bit stream you could go
    for something like:

    <code>
    File f = new File("pathToFile");
    byte[] mybytes = new byte[(int)f.length()];
    InputStream in = new FileInputStream(f);
    in.read(mybytes);
    </code>

    That is extremely bad optimized as well, but it should at least work
    (the binary data is in the byte[] mybytes afterwards). I guess with a
    quick Google search you can improve the solution (look for 'buffered
    reading of binary data' ... ).

    /philipp
    Philipp Leitner, Jul 15, 2006
    #3
  4. gbattine

    Guest

    Philipp Leitner wrote:

    > <code>
    > File f = new File("pathToFile");
    > byte[] mybytes = new byte[(int)f.length()];
    > InputStream in = new FileInputStream(f);
    > in.read(mybytes);
    > </code>
    >
    > That is extremely bad optimized as well, but it should at least work
    > (the binary data is in the byte[] mybytes afterwards). I guess with a
    > quick Google search you can improve the solution (look for 'buffered
    > reading of binary data' ... ).


    Actually, this is one of the quicker ways of reading a file into memory
    in one shot. There is no reason to wrap the FileInputStream in a
    BufferedInputStream for that, it will only delegate the work back to
    FileInputStream (but this *only* applies to this particular scenario.
    If you are reading in small increments, BufferedInputStream is a must.)
    It is still slightly faster (if the file is large) to map the file into
    memory with NIO and convert it to a byte array, but I can't recommend
    this for general use, since there is no way to unmap the file after
    reading it (and converting an mmapped file into a byte array just seems
    wrong). And for large data sets, reading everything into memory is not
    a good idea anyhow.

    Regards,
    Daniel Sjöblom
    , Jul 15, 2006
    #4
  5. gbattine

    gbattine Guest

    thanks very much for your helps...
    i have a question for you,excuse me my inexperience.
    I'v often the error java heap size, i think because my array of byte
    beacame very big....
    in my function

    temp is the vector that stores each line
    and data is the array of byte.

    During

    data = new byte[temp.size()];

    for (int t = 0; t < temp.size(); t++) {
    data[t] = (((Byte) temp.elementAt(t)).byteValue());

    }
    }

    where i pass the vector in the array of bytes it happens that the array
    of byte became too large.
    How can solve this problem without modify the rest of code?
    Storing the vector in more array of byte or writing it to a file can
    solve my

    java heap size problem?

    Please help me, i'm not very happy to modify all my precedent code, i
    hope there is some way to solve the problem.....
    i'm waiting for you help
    Thanks
    gbattine, Jul 16, 2006
    #5
  6. gbattine

    gbattine Guest

    Thanks very much,
    i'm not very happy to modify my precedent code,so i hope to find a
    solution to my problem.
    In my function

    temp is the vector in which i store the file line to line and data is
    the array of byte i want to send in blob field.

    data = new byte[temp.size()];

    for (int t = 0; t < temp.size(); t++) {
    data[t] = (((Byte) temp.elementAt(t)).byteValue());

    }

    with this operation i copy the vector in the array of byte and with big
    file i have java heap size error.

    Can i solve my problem changing only these code lines?
    Can i solve the problem splitting the vector in more array of byte o
    writing them into files?
    Can you help me,please?
    I'm waiting for your helps..
    gbattine, Jul 16, 2006
    #6
  7. gbattine wrote:

    > temp is the vector that stores each line
    > and data is the array of byte.
    >
    > During
    >
    > data = new byte[temp.size()];
    >
    > for (int t = 0; t < temp.size(); t++) {
    > data[t] = (((Byte) temp.elementAt(t)).byteValue());
    >
    > }
    > }
    >
    > where i pass the vector in the array of bytes it happens that the array
    > of byte became too large.
    > How can solve this problem without modify the rest of code?
    > Storing the vector in more array of byte or writing it to a file can
    > solve my
    >
    > java heap size problem?
    >
    > Please help me, i'm not very happy to modify all my precedent code, i
    > hope there is some way to solve the problem.....
    > i'm waiting for you help
    > Thanks


    Well, I'd say if you /should/ modfiy the rest of the code, too ... it
    just does not look good to me to store binary data in a Vector of Bytes
    .... see, a simple 'byte' in a byte[] should take exactly /1 byte of
    memory/ , but AFAIK a Byte in a Vector is a Java object reference and
    should take at least 8 bytes of memory, and also (compared to
    constructing of a simple 'byte') ages to be instanciated. It is no
    wonder to me that you run out of heap space quickly...

    If you desparately need a way to work around the problem without
    changing your code you can increase the heap size of the JVM ... but
    that may just temporarly solve your problem at hand, and not the actual
    root problem ... see for instance
    http://forum.java.sun.com/thread.jspa?threadID=490356&tstart=0 on how
    to change the heap size.

    /philipp
    Philipp Leitner, Jul 16, 2006
    #7
  8. gbattine

    gbattine Guest

    well,
    thanks to your help i've decided to change my code, but now i'm getting
    in panic....
    can you give some helps or useful links?
    I need to create object Riga to store the lines other than the first?
    Please help me and thanks very much...

    P.S. When i start to develop this function i've used a vector type
    because it can dynamically grows up where byte type no...how can i do
    now?
    I'm bit inexpert about files and byte...
    gbattine, Jul 16, 2006
    #8
  9. gbattine wrote:
    > well,
    > thanks to your help i've decided to change my code, but now i'm getting
    > in panic....
    > can you give some helps or useful links?
    > I need to create object Riga to store the lines other than the first?
    > Please help me and thanks very much...
    >
    > P.S. When i start to develop this function i've used a vector type
    > because it can dynamically grows up where byte type no...how can i do
    > now?
    > I'm bit inexpert about files and byte...


    Sorry, I lost you somewhere ... what has the 'Riga' object that you
    posted earlier to do with all that? I cannot see any connection to the
    byte reading issue we are talking about here.

    Generally - why do you care about 'lines' when you want to read binary
    data? If you treat the text file as binary data then it is just a chump
    of bytes - to line ends, no characters, no whatever, just 0s and 1s ...

    /philipp
    Philipp Leitner, Jul 16, 2006
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Nick
    Replies:
    20
    Views:
    749
  2. Matthew Wilson

    Need advice on how to improve this function

    Matthew Wilson, Aug 20, 2006, in forum: Python
    Replies:
    3
    Views:
    280
    Gabriel Genellina
    Aug 22, 2006
  3. Jeff Rush
    Replies:
    0
    Views:
    355
    Jeff Rush
    Mar 9, 2007
  4. 88059355
    Replies:
    0
    Views:
    252
    88059355
    Jan 17, 2008
  5. Replies:
    0
    Views:
    296
Loading...

Share This Page