OutOfMemoryException

Discussion in 'Java' started by srinivas.veeranki@gmail.com, Dec 20, 2006.

  1. Guest

    Hi All,

    While I am retrieving from the data base I am getting more than 45k
    Records. I am unable to stored all those records in to the List. Here I
    am getting OutOfMemoryException. How Can i store all those records in
    the List.

    Regards,

    Srinivas.
     
    , Dec 20, 2006
    #1
    1. Advertising

  2. bugbear Guest

    wrote:
    > Hi All,
    >
    > While I am retrieving from the data base I am getting more than 45k
    > Records. I am unable to stored all those records in to the List. Here I
    > am getting OutOfMemoryException. How Can i store all those records in
    > the List.


    Use less memory per record. Which may involve
    simple data packing or compression.

    BugBear
     
    bugbear, Dec 20, 2006
    #2
    1. Advertising

  3. Oliver Wong Guest

    "bugbear" <bugbear@trim_papermule.co.uk_trim> wrote in message
    news:45892c1f$0$8747$...
    > wrote:
    >> Hi All,
    >>
    >> While I am retrieving from the data base I am getting more than 45k
    >> Records. I am unable to stored all those records in to the List. Here I
    >> am getting OutOfMemoryException. How Can i store all those records in
    >> the List.

    >
    > Use less memory per record. Which may involve
    > simple data packing or compression.


    Alternatively, assign more memory to your Java program, or don't
    actually store all the data in the list. See the Proxy design pattern, for
    an example of the latter.

    - Oliver
     
    Oliver Wong, Dec 20, 2006
    #3
  4. bugbear Guest

    Oliver Wong wrote:
    >>Use less memory per record. Which may involve
    >>simple data packing or compression.

    >
    >
    > Alternatively, assign more memory to your Java program, or don't
    > actually store all the data in the list. See the Proxy design pattern, for
    > an example of the latter.


    At the risk of topic drift, does any one
    know of a List implementation
    which doesn't suffer from ArrayList's
    "growth problem", causing (IMHO) premature
    out of memory reports?

    The problem is that when the current memory allocation
    is exceeded, an ArrayList tries to double in size;

    assuming you're currently using (e.g.) 64 Mb,
    it will try and allocate 128 Mb, then copy the data
    from the 64 to the 128, and only then does the 64
    become available for GC.

    Assuming you actual data size were 65 Mb, this
    means you would require 196 Mb to "survive" the handover.

    A better data structure (w.r.t growth) would be a list
    of BLOCKs of some size. Additional blocks can be added,
    and serial (iterator) access would be fast.

    Random access is also "quite" fast, if the blocks are a
    "reasonable" size.

    Does anyone know an implementation of this, or something like it?

    BugBear
     
    bugbear, Dec 20, 2006
    #4
  5. Simon Brooke Guest

    in message <>,
    ('') wrote:

    > Hi All,
    >
    > While I am retrieving from the data base I am getting more than 45k
    > Records. I am unable to stored all those records in to the List. Here I
    > am getting OutOfMemoryException. How Can i store all those records in
    > the List.


    Records belong in the database. Retrieve them a few at a time. As a general
    rule, trying to schlurp the whole result of a database query into memory
    is a bad idea, because the data can be arbitrarily big.

    --
    (Simon Brooke) http://www.jasmine.org.uk/~simon/

    ;; Life would be much easier if I had the source code.
     
    Simon Brooke, Dec 20, 2006
    #5
  6. Daniel Pitts Guest

    bugbear wrote:
    > Oliver Wong wrote:
    > >>Use less memory per record. Which may involve
    > >>simple data packing or compression.

    > >
    > >
    > > Alternatively, assign more memory to your Java program, or don't
    > > actually store all the data in the list. See the Proxy design pattern, for
    > > an example of the latter.

    >
    > At the risk of topic drift, does any one
    > know of a List implementation
    > which doesn't suffer from ArrayList's
    > "growth problem", causing (IMHO) premature
    > out of memory reports?
    >
    > The problem is that when the current memory allocation
    > is exceeded, an ArrayList tries to double in size;
    >
    > assuming you're currently using (e.g.) 64 Mb,
    > it will try and allocate 128 Mb, then copy the data
    > from the 64 to the 128, and only then does the 64
    > become available for GC.
    >
    > Assuming you actual data size were 65 Mb, this
    > means you would require 196 Mb to "survive" the handover.
    >
    > A better data structure (w.r.t growth) would be a list
    > of BLOCKs of some size. Additional blocks can be added,
    > and serial (iterator) access would be fast.
    >
    > Random access is also "quite" fast, if the blocks are a
    > "reasonable" size.
    >
    > Does anyone know an implementation of this, or something like it?
    >
    > BugBear


    If you know before hand that you'll have a lot of data, you can tell
    the ArrayList to allocate a capacity that is as large (or larger) than
    you need.
     
    Daniel Pitts, Dec 20, 2006
    #6
  7. Daniel Pitts Guest

    wrote:
    > Hi All,
    >
    > While I am retrieving from the data base I am getting more than 45k
    > Records. I am unable to stored all those records in to the List. Here I
    > am getting OutOfMemoryException. How Can i store all those records in
    > the List.
    >
    > Regards,
    >
    > Srinivas.


    If possilbe, try to retrieve only a subset (using LIMIT in MySQL)

    If you go through the records, and test them for some condition, and
    THEN work with them, do that test in the SQL statement, not Java.

    Depending on how you are accessing the database, there may be ways to
    "paginate" your result list, so that only a portion is loaded in memory
    at any one time.
     
    Daniel Pitts, Dec 20, 2006
    #7
  8. Simon Brooke <> writes:

    > in message <>,
    > ('') wrote:
    >
    >> Hi All,
    >>
    >> While I am retrieving from the data base I am getting more than 45k
    >> Records. I am unable to stored all those records in to the List. Here I
    >> am getting OutOfMemoryException. How Can i store all those records in
    >> the List.

    >
    > Records belong in the database. Retrieve them a few at a time. As a general
    > rule, trying to schlurp the whole result of a database query into memory
    > is a bad idea, because the data can be arbitrarily big.



    More concretely:

    If you're getting a java.sql.ResultSet from your database,
    you're already retrieving them a few at a time. (I have few
    nice things to say about this interface, but it is handy
    that you get this for free.) You can step through the results
    by repeatedly calling next(), and never have to worry about
    memory, as long as you can make your computation without
    keeping a reference to all those rows.



    (You could even wrap up this behavior into your own List
    subclass, which would provide an Iterator that would call
    ResultSet.next(). I think this would be a terrible idea.)

    --
    Mark Jeffcoat
    Austin, TX
     
    Mark Jeffcoat, Dec 21, 2006
    #8
  9. EJP Guest

    bugbear wrote:
    > At the risk of topic drift, does any one
    > know of a List implementation
    > which doesn't suffer from ArrayList's
    > "growth problem", causing (IMHO) premature
    > out of memory reports?


    java.util.LinkedList.

    > The problem is that when the current memory allocation
    > is exceeded, an ArrayList tries to double in size;


    No. 'The details of the growth policy are not specified beyond the fact
    that adding an element has constant amortized time cost.' The JDK 1.5
    implementation increases the size by 50%.
     
    EJP, Dec 21, 2006
    #9
  10. "bugbear" <bugbear@trim_papermule.co.uk_trim> wrote in message
    news:458973b5$0$8759$...
    > Oliver Wong wrote:
    >>>Use less memory per record. Which may involve
    >>>simple data packing or compression.

    >>
    >>
    >> Alternatively, assign more memory to your Java program, or don't
    >> actually store all the data in the list. See the Proxy design pattern,
    >> for an example of the latter.

    >
    > At the risk of topic drift, does any one
    > know of a List implementation
    > which doesn't suffer from ArrayList's
    > "growth problem", causing (IMHO) premature
    > out of memory reports?
    >
    > The problem is that when the current memory allocation
    > is exceeded, an ArrayList tries to double in size;
    >
    > assuming you're currently using (e.g.) 64 Mb,
    > it will try and allocate 128 Mb, then copy the data
    > from the 64 to the 128, and only then does the 64
    > become available for GC.
    >
    > Assuming you actual data size were 65 Mb, this
    > means you would require 196 Mb to "survive" the handover.
    >


    No, the only thing that increases (doubling or otherwise) is the array of
    references it uses; the data those references point to isn't duplicated. If
    you have 64K objects and the reference array doubles in sixe, you'd add 128K
    references, which, depending on the size of a reference, would probably
    total between 512KB and 1MB.
     
    Mike Schilling, Dec 21, 2006
    #10
  11. In article <fOkih.11122$>,
    EJP <> wrote:
    >bugbear wrote:
    >> At the risk of topic drift, does any one
    >> know of a List implementation
    >> which doesn't suffer from ArrayList's
    >> "growth problem", causing (IMHO) premature
    >> out of memory reports?

    >
    >java.util.LinkedList.


    Of course, if, as in the proposed scenario, you have 65MB of
    _references_ alone (which seems a lot), then a singly linked list
    would increase the resident requirement to 130MB and a doubly linked
    one to 196MB. This may or may not be considered an improvement over
    ArrayList's performance.

    I don't know the details of how java.util.LinkedList is implemented
    though - it might use more than this, or it might use less (well,
    130MB seems like a minimum any way you do it).

    It really needs to be said that with 65MB of references, you are
    probably more worried about the space needed by the actual objects
    being referred to - unless there are multiple repetitions of the same
    reference within those 65MB.

    >No. 'The details of the growth policy are not specified beyond the fact
    >that adding an element has constant amortized time cost.' The JDK 1.5
    >implementation increases the size by 50%.


    Wouldn't this be a useful thing to have a protected method decide so
    that the policy can be modified by subclasses?

    Cheers
    Bent D
    --
    Bent Dalager - - http://www.pvv.org/~bcd
    powered by emacs
     
    Bent C Dalager, Dec 21, 2006
    #11
  12. bugbear Guest

    EJP wrote:
    > bugbear wrote:
    >
    >> At the risk of topic drift, does any one
    >> know of a List implementation
    >> which doesn't suffer from ArrayList's
    >> "growth problem", causing (IMHO) premature
    >> out of memory reports?

    >
    >
    > java.util.LinkedList.
    >
    >> The problem is that when the current memory allocation
    >> is exceeded, an ArrayList tries to double in size;

    >
    >
    > No. 'The details of the growth policy are not specified beyond the fact
    > that adding an element has constant amortized time cost.' The JDK 1.5
    > implementation increases the size by 50%.


    Actually, even if it only increment by 5% the instantaneous
    memory use is still double (old copy, slightly larger new copy)
    the actual requirment.

    BugBear
     
    bugbear, Dec 21, 2006
    #12
  13. EJP wrote:
    > bugbear wrote:
    >> At the risk of topic drift, does any one
    >> know of a List implementation
    >> which doesn't suffer from ArrayList's
    >> "growth problem", causing (IMHO) premature
    >> out of memory reports?

    >
    > java.util.LinkedList.


    Technically true, but the LinkedList will already be using vastly more
    memory than ArrayList by the time there is a problem. LinkedList is a
    good answer to very few questions.

    Tom Hawtin
     
    Thomas Hawtin, Dec 21, 2006
    #13
  14. Thomas Hawtin wrote:
    > Technically true, but the LinkedList will already be using vastly more
    > memory than ArrayList by the time there is a problem. LinkedList is a
    > good answer to very few questions.


    One of those "very few" questions happens to arise frequently, however,
    namely "What do I do when I add only at one or both ends and remove only
    at the ends or when iterating"? LinkedList can do all of those in O(1),
    assuming a non-stupid implementation of the iterator. It's the perfect
    back end for various kinds of queues and temporary lists. That other
    thread where someone wants to read in variable-length arrays as arrays,
    for example -- various people suggested ArrayList, but the actual
    maximum efficiency is achieved by building LinkedLists and calling
    toArray at the end of building each one. (ArrayLists would wind up with
    some extra capacity, unless you did two passes over the file instead of
    one, which would be an inefficiency of its own.)
     
    John Ersatznom, Dec 21, 2006
    #14
  15. Mark Rafn Guest

    memory-friendly List implementations

    bugbear <bugbear@trim_papermule.co.uk_trim> wrote:
    >At the risk of topic drift, does any one
    >know of a List implementation
    >which doesn't suffer from ArrayList's
    >"growth problem", causing (IMHO) premature
    >out of memory reports?


    When possible, please change the title when you cause or notice a topic drift.
    It makes things far easier to find later.

    >The problem is that when the current memory allocation
    >is exceeded, an ArrayList tries to double in size;


    Triple, actually. It creates a new array twice the size of the old array, and
    copies the contents from the old array before the old array can be GC'd.

    >A better data structure (w.r.t growth) would be a list
    >of BLOCKs of some size. Additional blocks can be added,
    >and serial (iterator) access would be fast.


    Yup. You then lose the constant-time indexed access, though. get(int) now
    becomes linear based on the number of blocks.

    You could also use a tree (or a tree of arrays) structure to balance space,
    sequential access, and indexed access patterns.

    >Does anyone know an implementation of this, or something like it?


    It's easy to implement your own Collection or List. For a fairly simple
    structure, I'd rather write my own than to try to find a match, understand it
    well enough to know it's right for me, and deal with an external dependency.
    This is one of those fine lines where reusing code isn't worth the complexity
    of finding and incorporating it.

    That said, jakarta commons has a collections package that provides some useful
    implementations, and more importantly some useful abstract base classes that
    may be easier to extend than writing from scratch.

    And THAT said, the previous responses which advised to change the system not
    to NEED such an operation are better than making a list which can do it.
    --
    Mark Rafn <http://www.dagon.net/>
     
    Mark Rafn, Dec 21, 2006
    #15
  16. Re: memory-friendly List implementations

    Mark Rafn wrote:
    > That said, jakarta commons has a collections package that provides
    > some useful implementations, and more importantly some useful
    > abstract base classes that may be easier to extend than writing from
    > scratch.


    As does the JDK; AbstractList and AbstractSequentialList are damned useful
    things.
     
    Mike Schilling, Dec 21, 2006
    #16
  17. Oliver Wong Guest

    Re: memory-friendly List implementations

    "Mark Rafn" <> wrote in message
    news:...
    > bugbear <bugbear@trim_papermule.co.uk_trim> wrote:
    >>A better data structure (w.r.t growth) would be a list
    >>of BLOCKs of some size. Additional blocks can be added,
    >>and serial (iterator) access would be fast.

    >
    > Yup. You then lose the constant-time indexed access, though. get(int)
    > now
    > becomes linear based on the number of blocks.


    Unless you can randomly access the blocks too, in which case the index
    access is O(2) (i.e. one operation to find the relevant block, and another
    operation to find the relevant item within that block) which is O(1). When
    expanding the list, you'd have to allocate space for more blocks, and copy
    the references to the block, which is a bit faster than copying the
    references to each element within the block. It's basically the same
    asymptotic runtime performance, but you're just playing around with the
    constants.

    If you allow blocks of blocks, then you basically have a tree, and so
    index access becomes O(log(n)).

    - Oliver
     
    Oliver Wong, Dec 21, 2006
    #17
  18. Re: memory-friendly List implementations

    Mark Rafn wrote:
    > Yup. You then lose the constant-time indexed access, though. get(int) now
    > becomes linear based on the number of blocks.


    That can be improved to logarithmic by recursively nesting the blocks.
    In the extreme case, the list is implemented as a binary tree under the
    hood, with the bit sequence of the index giving a series of left/right
    turns to make from the root down to the correct object. The number of
    links to follow is O(log n) in this case. When the list reaches 2^foo
    size, a new root node is created with the old one as left child and new
    list items develop the right side of the tree. When it's doubled again,
    this process happens again, and so forth. Of course, the index bit
    sequence used to navigate to an item has to be read from left to right
    starting with the nth bit and ending with the zeroth, with n the tree's
    height minus 1. The height is easy to track, though, since you can just
    stuff a zero in it for an empty list and add one every time a new root
    node is created. Memory can be saved by leaving unused parts of the tree
    absent, with a null child reference. An empty list has a null root node,
    a height of zero, and a grow threshold of one. Go to add an item and the
    threshold is reached, so a new root node is made, the old (null) added
    to it as its left child, and the object stored at index 0. The height's
    now 1, so the height minus 1 is 0 and no navigation is done and the
    object's stored at the root. The theshold is doubled to 2. Add another
    item and the threshold is reached again, so we end up with a root node
    with a left child node holding the object. The index 0 is now read to 1
    bit to find that object, and that bit is a 0, so you make one left from
    the root. The new object's index of 1 has the bit a 1, so it's stored by
    creating a right child of the root node and storing the object there.

    Turning the index into a path is cheap and O(log n) as well: you can
    just take a copy of the grow threshold, halve it, and & the index with
    this to get the first left/right choice; halve it and & the index for
    the second; and so forth. For the two-element list above, the threshold
    is 2 (meaning it's full) so we'd halve that (1) and look at index & 1,
    i.e. the zeroth bit. For a three- or four-element list the threshold has
    become 4, so we'd check index & 2 and then index & 1 in turn.

    > You could also use a tree (or a tree of arrays) structure to balance space,
    > sequential access, and indexed access patterns.


    See above. You can get O(log n) indexed access out of that too.
    Sequential access is had by either the usual traversal algorithm or
    putting the linked list prev and next refs into the leaf node objects.
    There's probably also a way to halve the size of these trees by storing
    objects in all the nodes rather than only the leaf nodes too, but I
    can't be arsed to reinvent it from scratch right now. :)

    > That said, jakarta commons has a collections package that provides some useful
    > implementations, and more importantly some useful abstract base classes that
    > may be easier to extend than writing from scratch.


    Linkie?

    > And THAT said, the previous responses which advised to change the system not
    > to NEED such an operation are better than making a list which can do it.


    That depends on the system's ultimate requirements. It may or may not be
    possible, separately for each system where this question arises.
     
    John Ersatznom, Dec 22, 2006
    #18
  19. bugbear Guest

    Re: memory-friendly List implementations

    Oliver Wong wrote:
    > "Mark Rafn" <> wrote in message
    > news:...
    >
    >>bugbear <bugbear@trim_papermule.co.uk_trim> wrote:
    >>
    >>>A better data structure (w.r.t growth) would be a list
    >>>of BLOCKs of some size. Additional blocks can be added,
    >>>and serial (iterator) access would be fast.

    >>
    >>Yup. You then lose the constant-time indexed access, though. get(int)
    >>now
    >>becomes linear based on the number of blocks.

    >
    >
    > Unless you can randomly access the blocks too, in which case the index
    > access is O(2) (i.e. one operation to find the relevant block, and another
    > operation to find the relevant item within that block) which is O(1). When
    > expanding the list, you'd have to allocate space for more blocks, and copy
    > the references to the block, which is a bit faster than copying the
    > references to each element within the block. It's basically the same
    > asymptotic runtime performance, but you're just playing around with the
    > constants.
    >
    > If you allow blocks of blocks, then you basically have a tree, and so
    > index access becomes O(log(n)).


    Hmm. What about (designing whilst posting) an array
    of between 10 and 20 blocks references?

    If the blocks start at (say...) 50 references each, the structure
    initially grows by adding new blocks, until we have 20 blocks.

    When this point is reached, I suggest a new operation takes
    place; a merge into blocks of double the size.

    After this merge the 20 blocks are now 10. Note that this merge
    takes a "very moderate" amount of extra memory.

    Growth now proceeds as before, by adding new blocks.

    This avoids the need for the LARGE amount of temporary
    memory use when growing an ArrayList, and still has
    (as far as I can work out) the same O() performance
    as ArrayList.

    BugBear
     
    bugbear, Dec 22, 2006
    #19
  20. Guest

    Hi All,

    I am retrieving the values fromt he database using the
    SpringFramework.
    In SpringFramework we have an execute method which returns the list and
    the query whcih we pass to the execute method returns 45k records

    The code where i am getting out of memory is pasted below


    String slSelectAdministradores = " SELECT codigo_administrador
    codigo,"
    + " razon_social descripcion,"
    + " clase_documento,identificacion,"
    + " estado_administrador estado"
    + " FROM administradores noholdlock"
    + " WHERE razon_social >= ? and"
    + " razon_social <= ? ";
    SelectAdministradore objSelectAdministradore = new
    SelectAdministradore(dscDataSource, slSelectAdministradores);
    objSelectAdministradore.declareParameter(new
    SqlParameter(Types.VARCHAR));
    objSelectAdministradore.declareParameter(new
    SqlParameter(Types.VARCHAR));
    Object[] oalAdministradore = new Object[] {
    nombreInicial, nombreFinal };
    objSelectAdministradore.compile();

    List li1 = objSelectAdministradore.execute(oalAdministradore);

    /**
    * Inner Class
    *
    */
    class SelectAdministradore extends MappingSqlQuery {

    CodDescripcionObj obj;

    SelectAdministradore(DataSource dsaDataSource,
    String saSelectAdministradores) {
    super(dsaDataSource, saSelectAdministradores);
    }

    protected Object mapRow(ResultSet rs, int pRowNum) throws
    SQLException {

    obj = new CodDescripcionObj();
    obj.setCodigo(rs.getString("codigo"));
    obj.setDescripcion(rs.getString("descripcion"));
    obj.setClase_documento(rs.getString("clase_documento"));
    obj.setIdentificacion(rs.getString("identificacion"));
    obj.setEstado(rs.getInt("estado"));
    return obj;
    }
    }


    wrote:
    > Hi All,
    >
    > While I am retrieving from the data base I am getting more than 45k
    > Records. I am unable to stored all those records in to the List. Here I
    > am getting OutOfMemoryException. How Can i store all those records in
    > the List.
    >
    > Regards,
    >
    > Srinivas.
     
    , Dec 26, 2006
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?UmlwdWwgSGFuZGE=?=

    Base Exception:System.OutOfMemoryException error message

    =?Utf-8?B?UmlwdWwgSGFuZGE=?=, Nov 17, 2003, in forum: ASP .Net
    Replies:
    1
    Views:
    999
    Natty Gur
    Nov 18, 2003
  2. =?Utf-8?B?amF6?=

    OutOfMemoryException

    =?Utf-8?B?amF6?=, Dec 10, 2003, in forum: ASP .Net
    Replies:
    2
    Views:
    352
    =?Utf-8?B?SmF6?=
    Dec 16, 2003
  3. Pyramis
    Replies:
    0
    Views:
    403
    Pyramis
    Jan 25, 2004
  4. =?Utf-8?B?U3Rhbg==?=

    Exception of type System.OutOfMemoryException is thrown

    =?Utf-8?B?U3Rhbg==?=, Feb 16, 2004, in forum: ASP .Net
    Replies:
    11
    Views:
    27,972
    Ken Cox [Microsoft MVP]
    Dec 28, 2004
  5. Versteijn
    Replies:
    7
    Views:
    20,124
    Denzien
    Mar 7, 2008
Loading...

Share This Page