Time - BuffererdReader takes read in data - store in hash Map TAKING FOREVER

Discussion in 'Java' started by PythonAnimal@gmail.com, Feb 3, 2006.

  1. Guest

    I am having prolems, I am using bufferedReader to read in data from a
    file store it in a hashtabl then store it in a hashmap. My problem is
    even with threading it is running very slow, 45 seconds to load 15,000
    lines. Is this accurate or is my code off?
    Here is a snippet. Any help would be appericated

    while(line != null)
    {

    while(headingToken.hasMoreTokens())
    { counter++;
    heading = headingToken.nextToken();
    try{
    data = lineToken.nextToken();
    //System.out.print(data + " " );
    }catch(Exception e) { data = ""; }
    if(heading.equals(""))
    heading += "heading";
    heading += counter + "NoHeading";
    _headingColumnsOutputFile += heading + ",";

    genevaDataRows.put(heading,data);
    }

    /************* in my code putting and removing

    String bifurcationPrimarykey =
    genevaDataRows.get(_primaryKey).toString() +
    genevaDataRows.get("Description").toString() + genevaDataRows.get("Tax
    Lot");
    genevaDataRows.put("Primary Key",bifurcationPrimarykey);
    this._genevaTradesMap.put (bifurcationPrimarykey,genevaDataRows);
    /****************
    line = bufRea.readLine();
    if(line != null)
    lineToken = new HBMJTokenizer2(line, ",", "\"");
    headingToken = new HBMJTokenizer2(_dataHeading, ",", "\"");
    genevaDataRows = new Hashtable(); // resets hashtable so another row
    may be inputted
    counter = 0;
    nd2 = new Date();
    System.out.println("End line != null " + this._accountingPeriod + " "
    + nd2.getMinutes() + " " + nd2.getSeconds());
    }
    Seems it should not take this long!!
    thanks gfor any advice
     
    , Feb 3, 2006
    #1
    1. Advertising

  2. Kevin Guest

    1) did you give the buffer reader a proper (large) buffer?
    2) adding Strings is very slow. Repalce with StringBuffer will decrease
    the time by 100 times in one of my code.
     
    Kevin, Feb 3, 2006
    #2
    1. Advertising

  3. Roedy Green Guest

    On 2 Feb 2006 16:55:16 -0800, wrote, quoted or
    indirectly quoted someone who said :

    >My problem is
    >even with threading it is running very slow, 45 seconds to load 15,000
    >lines. Is this accurate or is my code off?


    For super speed read the entire file into ram as an array of bytes,
    then convert to string, then parse it yourself. Here is the code to do
    it. This way, I process 200 files a second.

    The other way to do it, is to crank your BufferedReader buffer up to
    say 64K.



    package com.mindprod.hunkio;
    import java.io.File;
    import java.io.FileInputStream;
    import java.io.FileOutputStream;
    import java.io.FileWriter;
    import java.io.IOException;
    import java.io_OutputStreamWriter;
    import java.io.UnsupportedEncodingException;

    /**
    * Read and write entire files as Strings. Presumes the default
    encoding. No
    * main method.
    *
    * @author Roedy Green <br>
    * version 1.5 - 2006-01-25 - readFile/writeFile with
    non-default
    * encoding, raw read/write entire file. <br>
    * version 1.4 - 2006-01-14 - readEntire file now works with
    multi-byte
    * encodings. <br>
    * version 1.3 - 2005-06-18 add more consistent bat files.<br>
    * @since 2002 July 12
    */
    public class HunkIO {

    /**
    * Create a temporary file, Slightly smarter version of
    File.createTempFile
    *
    * @param prefix
    * beginning letters of filename
    * @param suffix
    * ending letters of filename.
    * @param near
    * directory where to put file, or file to place this temp
    file near
    * in the same directory. null means put the temp file in
    the current
    * directory.
    * @return A temporary file. It will not automatically delete on
    program
    * completion, however.
    * @exception IOException
    */
    public static File createTempFile ( String prefix, String suffix,
    File near )
    throws IOException
    {
    if ( near != null )
    {
    if ( near.isDirectory() )
    {
    return File.createTempFile( prefix, suffix, near );
    }
    else if ( near.isFile() )
    {
    String parent = near.getParent();
    if ( parent != null )
    {
    File dir = new File( parent );
    if ( dir.isDirectory() )
    {
    return File.createTempFile( prefix, suffix,
    dir );
    }
    }
    }
    }
    // anything else, just create in the current directory.
    return File.createTempFile( prefix, suffix );
    }

    /**
    * read file of bytes without conversion
    *
    * @param fromFile
    * file to read
    * @return byte array representing whole file
    * @throws IOException
    */
    public static byte[] rawReadEntireFile ( File fromFile ) throws
    IOException
    {
    int size = (int)fromFile.length();

    FileInputStream fis = new FileInputStream( fromFile );
    // R E A D
    byte[] rawContents = new byte[ size ];
    int bytesRead = fis.read( rawContents );
    if ( bytesRead != size )
    {
    throw new IOException( "error: problems reading file " +
    fromFile );
    }
    // C L O S E
    fis.close();
    return rawContents;
    }

    /**
    * read file of bytes without conversion
    *
    * @param fromFile
    * file to read
    * @return byte array representing whole file
    * @throws IOException
    */
    public static byte[] rawReadEntireFile ( String fromFile )
    throws IOException
    {
    return rawReadEntireFile( new File( fromFile ) );
    }

    /**
    * Get all text in a file.
    *
    * @param fromFile
    * file where to get the text.
    * @return all the text in the File.
    * @exception IOException
    */
    public static String readEntireFile ( File fromFile ) throws
    IOException
    {
    // decode with default encoding and return entire file as one
    big string
    return new String( rawReadEntireFile( fromFile ) );
    } // end readEntireFile

    /**
    * Get all text in a file.
    *
    * @param fromFile
    * file where to get the text.
    * @param encoding
    * name of the encoding to use to translate the bytes in
    the file
    * into chars e.g. "windows-1252" "UTF-8" "UTF-16"
    * @return all the text in the File.
    * @exception IOException
    * @exception UnsupportedEncodingException
    * If the named encoding is not supported
    */
    public static String readEntireFile ( File fromFile, String
    encoding )
    throws IOException, UnsupportedEncodingException
    {
    // decode with encoding and return entire file as one big
    string
    return new String( rawReadEntireFile( fromFile ), encoding );
    } // end readEntireFile

    /**
    * Get all text in a file.
    *
    * @param fromFile
    * Name of file where to get the text.
    * @return all the text in the File.
    * @exception IOException
    */
    public static String readEntireFile ( String fromFile ) throws
    IOException
    {
    return readEntireFile( new File( fromFile ) );
    } // end readEntireFile

    /**
    * Get all text in a file.
    *
    * @param fromFile
    * Name of file where to get the text.
    * @param encoding
    * name of the encoding to use to translate the bytes in
    the file
    * into chars e.g. "windows-1252" "UTF-8" "UTF-16"
    * @return all the text in the File.
    * @exception IOException
    * @exception UnsupportedEncodingException
    * If the named encoding is not supported
    */
    public static String readEntireFile ( String fromFile, String
    encoding )
    throws IOException, UnsupportedEncodingException
    {
    return readEntireFile( new File( fromFile ), encoding );
    } // end readEntireFile

    /**
    * Write all the text in a file
    *
    * @param toFile
    * file where to write the text
    * @param text
    * Text to write
    * @exception IOException
    */
    public static void writeEntireFile ( File toFile, String text )
    throws IOException
    {
    // default encoding
    FileWriter fw = new FileWriter( toFile );
    // W R I T E
    fw.write( text );
    // C L O S E
    fw.close();
    } // end writeEntireFile

    /**
    * Write all the text in a file
    *
    * @param toFile
    * file where to write the text
    * @param text
    * Text to write
    * @param encoding
    * name of the encoding to use to translate chars to bytes
    e.g.
    * "windows-1252" "UTF-8" "UTF-16"
    * @exception IOException
    * @exception UnsupportedEncodingException
    * If the named encoding is not supported
    */
    public static void writeEntireFile ( File toFile, String text,
    String encoding ) throws IOException
    {
    // supplied encoding
    FileOutputStream fos = new FileOutputStream( toFile );
    OutputStreamWriter osw = new OutputStreamWriter( fos, encoding
    );
    // W R I T E
    osw.write( text );
    // C L O S E
    osw.close();
    } // end writeEntireFile

    /**
    * Write all the text in a file
    *
    * @param toFile
    * Name of file where to write the text
    * @param text
    * Text to write
    * @exception IOException
    */
    public static void writeEntireFile ( String toFile, String text )
    throws IOException
    {
    writeEntireFile( new File( toFile ), text );

    } // end writeEntireFile

    /**
    * Write all the text in a file
    *
    * @param toFile
    * file where to write the text
    * @param text
    * Text to write
    * @param encoding
    * name of the encoding to use to translate chars to bytes
    e.g.
    * "windows-1252" "UTF-8" "UTF-16"
    * @exception IOException
    * @exception UnsupportedEncodingException
    * If the named encoding is not supported
    */
    public static void writeEntireFile ( String toFile, String text,
    String encoding ) throws IOException
    {
    writeEntireFile( new File( toFile ), text, encoding );
    } // end writeEntireFile

    } // end HunkIO

    --
    Canadian Mind Products, Roedy Green.
    http://mindprod.com Java custom programming, consulting and coaching.
     
    Roedy Green, Feb 3, 2006
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Erik Sandblom

    open with encoding(utf8) takes forever

    Erik Sandblom, May 28, 2004, in forum: Perl
    Replies:
    0
    Views:
    553
    Erik Sandblom
    May 28, 2004
  2. Helixpoint

    This takes forever!!!!!!!!

    Helixpoint, Jul 29, 2003, in forum: ASP .Net
    Replies:
    1
    Views:
    503
    Helixpoint
    Jul 29, 2003
  3. =?Utf-8?B?RGF2aWQgVGhpZWxlbg==?=

    Takes forever to create a new project

    =?Utf-8?B?RGF2aWQgVGhpZWxlbg==?=, Feb 12, 2005, in forum: ASP .Net
    Replies:
    6
    Views:
    385
    Aquila Deus
    Feb 14, 2005
  4. Roedy Green
    Replies:
    2
    Views:
    334
    Roedy Green
    Aug 5, 2003
  5. John Saunders

    First-Time JIT Compile Takes Forever

    John Saunders, Feb 9, 2007, in forum: ASP .Net
    Replies:
    0
    Views:
    380
    John Saunders
    Feb 9, 2007
Loading...

Share This Page