java.lang.OutOfMemoryError: Java heap space

Discussion in 'Java' started by Goofball, Oct 29, 2006.

  1. Goofball

    Goofball Guest

    Does anyone have any ideas on why that happened:

    [java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap
    space

    Explaining the situation. I have such code:

    for (int i=0; i<10000-1; i++) {
    for (int j=i+1; j<10000; j++) {
    compareElements(arr, arr[j]);
    }
    }

    Element of the array is a simple class that has two fields (one is
    string and another is BitVector) and getters and setters for those
    fields. Function compareElements compares the BitVectors of two
    elements.

    Now the problem: when I run the code, program runs only till then 1334
    element in the first cycle. Then I receive the above exception.

    Does anyone know, what can that be? Thanks for any help.
     
    Goofball, Oct 29, 2006
    #1
    1. Advertising

  2. Goofball wrote:
    > Does anyone have any ideas on why that happened:
    >
    > [java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap
    > space
    >
    > Explaining the situation. I have such code:
    >
    > for (int i=0; i<10000-1; i++) {
    > for (int j=i+1; j<10000; j++) {
    > compareElements(arr, arr[j]);
    > }
    > }
    >
    > Element of the array is a simple class that has two fields (one is
    > string and another is BitVector) and getters and setters for those
    > fields. Function compareElements compares the BitVectors of two
    > elements.
    >
    > Now the problem: when I run the code, program runs only till then 1334
    > element in the first cycle. Then I receive the above exception.
    >
    > Does anyone know, what can that be? Thanks for any help.
    >


    Yeah, your JVM is allocated a certain amount of memory from the OS and
    you are creating so many objects at a certain size that your JVM is
    running out of memory, hence the OutOfMemoryError exception. You can
    tune the size of the memory for the JVM by passing in various parameters
    to the java command. I believe they look similar to -X256m or something
    like that. You could also think about how your programming is allocating
    the data and possibly restructure your algorithm or, if possibly, when
    you are done working with the data then get rid of it so the GC can
    reclaim the memory.
     
    Brandon McCombs, Oct 29, 2006
    #2
    1. Advertising

  3. Goofball

    Daniel Pitts Guest

    Goofball wrote:
    > Does anyone have any ideas on why that happened:
    >
    > [java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap
    > space
    >
    > Explaining the situation. I have such code:
    >
    > for (int i=0; i<10000-1; i++) {
    > for (int j=i+1; j<10000; j++) {
    > compareElements(arr, arr[j]);
    > }
    > }
    >
    > Element of the array is a simple class that has two fields (one is
    > string and another is BitVector) and getters and setters for those
    > fields. Function compareElements compares the BitVectors of two
    > elements.
    >
    > Now the problem: when I run the code, program runs only till then 1334
    > element in the first cycle. Then I receive the above exception.
    >
    > Does anyone know, what can that be? Thanks for any help.


    If you post an SSCCE (http://www.physci.org/codes/sscce/), we would be
    better able to help you.
     
    Daniel Pitts, Oct 30, 2006
    #3
  4. Goofball

    Goofball Guest

    I was experimenting with gc() in different ways. I was callling it
    after processing 1000 elements. However, that didn't help and the
    problem remained the same: after processing 1334 (or 1335) elements I
    received the OutOfMemory error.

    Brandon McCombs wrote:
    > Goofball wrote:
    > > Does anyone have any ideas on why that happened:
    > >
    > > [java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap
    > > space
    > >
    > > Explaining the situation. I have such code:
    > >
    > > for (int i=0; i<10000-1; i++) {
    > > for (int j=i+1; j<10000; j++) {
    > > compareElements(arr, arr[j]);
    > > }
    > > }
    > >
    > > Element of the array is a simple class that has two fields (one is
    > > string and another is BitVector) and getters and setters for those
    > > fields. Function compareElements compares the BitVectors of two
    > > elements.
    > >
    > > Now the problem: when I run the code, program runs only till then 1334
    > > element in the first cycle. Then I receive the above exception.
    > >
    > > Does anyone know, what can that be? Thanks for any help.
    > >

    >
    > Yeah, your JVM is allocated a certain amount of memory from the OS and
    > you are creating so many objects at a certain size that your JVM is
    > running out of memory, hence the OutOfMemoryError exception. You can
    > tune the size of the memory for the JVM by passing in various parameters
    > to the java command. I believe they look similar to -X256m or something
    > like that. You could also think about how your programming is allocating
    > the data and possibly restructure your algorithm or, if possibly, when
    > you are done working with the data then get rid of it so the GC can
    > reclaim the memory.
     
    Goofball, Oct 30, 2006
    #4
  5. Goofball

    Goofball Guest

    Don't know, how much is that an SSCCE, but I am posting you my example:

    import java.util.ArrayList;
    import java.util.BitSet;
    import java.util.Collection;
    import java.util.HashMap;
    import java.util.List;
    import java.util.Map;

    public class Example {

    private class ArrElem {

    private String id;
    private BitSet fp;

    public BitSet getFp() { return fp; }
    public void setFp(BitSet fp) { this.fp = fp; }
    public String getId() { return id; }
    public void setId(String id) { this.id = id;}
    }

    private List<ArrElem> arr = new ArrayList<ArrElem>();

    private Map<ArrElem, Collection<ArrElem>> rez = new HashMap<ArrElem,
    Collection<ArrElem>>();

    public void processArr() {
    //fillArr(10000);
    for (int i = 0; i < 10000-1; i++) {
    for (int j = i+1; j < 10000; j++) {
    compareElements(arr.get(i), arr.get(j));
    }
    }
    }

    private void compareElements(ArrElem elem1, ArrElem elem2) {
    int bitM = elem1.getFp().cardinality();
    int bitN = elem2.getFp().cardinality();

    BitSet commonBits = elem1.getFp();
    commonBits.and(elem2.getFp());
    int bitCommon = commonBits.cardinality();

    double index = bitCommon/(bitM+bitN+bitCommon);
    if (index < 0.8) {
    if (! rez.containsKey(elem1))
    rez.put(elem1, new ArrayList<ArrElem>());
    rez.get(elem1).add(elem2);
    }
    }
    }


    Daniel Pitts wrote:
    > Goofball wrote:
    > > Does anyone have any ideas on why that happened:
    > >
    > > [java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap
    > > space
    > >
    > > Explaining the situation. I have such code:
    > >
    > > for (int i=0; i<10000-1; i++) {
    > > for (int j=i+1; j<10000; j++) {
    > > compareElements(arr, arr[j]);
    > > }
    > > }
    > >
    > > Element of the array is a simple class that has two fields (one is
    > > string and another is BitVector) and getters and setters for those
    > > fields. Function compareElements compares the BitVectors of two
    > > elements.
    > >
    > > Now the problem: when I run the code, program runs only till then 1334
    > > element in the first cycle. Then I receive the above exception.
    > >
    > > Does anyone know, what can that be? Thanks for any help.

    >
    > If you post an SSCCE (http://www.physci.org/codes/sscce/), we would be
    > better able to help you.
     
    Goofball, Oct 30, 2006
    #5
  6. Goofball

    Ingo Menger Guest

    Goofball wrote:

    > private void compareElements(ArrElem elem1, ArrElem elem2) {
    > int bitM = elem1.getFp().cardinality();
    > int bitN = elem2.getFp().cardinality();
    >
    > BitSet commonBits = elem1.getFp();
    > commonBits.and(elem2.getFp());
    > int bitCommon = commonBits.cardinality();
    >
    > double index = bitCommon/(bitM+bitN+bitCommon);
    > if (index < 0.8) {
    > if (! rez.containsKey(elem1))
    > rez.put(elem1, new ArrayList<ArrElem>());
    > rez.get(elem1).add(elem2);
    > }
    > }


    There you have it.
    The index will be 0.0 unless bitM and bitN are both 0. So you end up
    creating at least a new ArrayList element in each cycle.
     
    Ingo Menger, Oct 30, 2006
    #6
  7. Goofball

    Red Orchid Guest

    "Goofball" <> wrote or quoted in
    Message-ID: <>:

    > ---------- snip -------------
    >
    > private List<ArrElem> arr = new ArrayList<ArrElem>();
    >
    > private Map<ArrElem, Collection<ArrElem>> rez = new HashMap<ArrElem,
    > Collection<ArrElem>>();
    >
    > public void processArr() {
    > //fillArr(10000);
    > for (int i = 0; i < 10000-1; i++) {
    > for (int j = i+1; j < 10000; j++) {
    > compareElements(arr.get(i), arr.get(j));
    > }
    > }
    > }
    >



    It is mere guesswork.


    The above code calls 'compareElements' (10000 * 10000) times.
    Namely, 100 M times.





    > private void compareElements(ArrElem elem1, ArrElem elem2) {
    > int bitM = elem1.getFp().cardinality();
    > int bitN = elem2.getFp().cardinality();
    >
    > BitSet commonBits = elem1.getFp();
    > commonBits.and(elem2.getFp());
    > int bitCommon = commonBits.cardinality();
    >
    > double index = bitCommon/(bitM+bitN+bitCommon);
    >




    Min of 'index' is 0.
    Max of 'index' is smaller than 1.



    >
    > if (index < 0.8) {
    >



    If 'cardinality()' is random,
    the probability of calling the bellow code will be about 8/10.

    That is, 80 M times (10000 * 10000 * 0.8)




    > if (! rez.containsKey(elem1))
    > rez.put(elem1, new ArrayList<ArrElem>());
    > rez.get(elem1).add(elem2);
    > }
    > }
    > }
    >
    >



    The above code adds an element to 'rez' whenever she is executed.
    A total of 80 M elements.

    320 MBytes memory (80M * 4) will be required at least.
     
    Red Orchid, Oct 30, 2006
    #7
  8. Goofball

    Ingo Menger Guest

    Red Orchid wrote:

    > > double index = bitCommon/(bitM+bitN+bitCommon);
    > >

    > Min of 'index' is 0.
    > Max of 'index' is smaller than 1.


    Yes, it's 0!
    Proof:
    case 1) bitM and bitM are 0. Then so will bitComnmon. This will result
    in zero divide. (BTW, this is a programming error)
    case 2) bitM or bitM are > 0. Then the integer division will result in
    0.

    Thus, index will never be anything else than 0.0. And the programm will
    crash occasionally.

    > > if (index < 0.8) {


    Thus, this is always true.

    > >

    >
    >
    > If 'cardinality()' is random,
    > the probability of calling the bellow code will be about 8/10.


    1, to be exact.


    > The above code adds an element to 'rez' whenever she is executed.
    > A total of 80 M elements.


    It adds an element everytime. A total of 100M elements.
     
    Ingo Menger, Oct 30, 2006
    #8
  9. Goofball

    Dijon Yu Guest

    Goofball wrote:
    > I was experimenting with gc() in different ways. I was callling it
    > after processing 1000 elements. However, that didn't help and the
    > problem remained the same: after processing 1334 (or 1335) elements I
    > received the OutOfMemory error.


    I have meet the problem too.
    Don't depend on gc(), whatever 。
    If you want resolve the problem, you can set the JVM.

    You can use java -Xmx256m to avoid the problem.
    But I think maybe you can refactor and redesign the algorithm.
     
    Dijon Yu, Nov 1, 2006
    #9
  10. Goofball

    Goofball Guest

    Thanks a lot. That helped. I redesigned the alsorithm. Now it works
    fine. But when I test it on about 1 million of elements - it crashes.
    Starting JVM with more memory helps, but that sometimes cannot be done
    (because of the requirements). So, I decided to write my own
    implementations of List and Map interfaces, that will flush the results
    on disk. My requirements allow that. :)
    Thanks again.

    Ingo Menger wrote:
    > Red Orchid wrote:
    >
    > > > double index = bitCommon/(bitM+bitN+bitCommon);
    > > >

    > > Min of 'index' is 0.
    > > Max of 'index' is smaller than 1.

    >
    > Yes, it's 0!
    > Proof:
    > case 1) bitM and bitM are 0. Then so will bitComnmon. This will result
    > in zero divide. (BTW, this is a programming error)
    > case 2) bitM or bitM are > 0. Then the integer division will result in
    > 0.
    >
    > Thus, index will never be anything else than 0.0. And the programm will
    > crash occasionally.
    >
    > > > if (index < 0.8) {

    >
    > Thus, this is always true.
    >
    > > >

    > >
    > >
    > > If 'cardinality()' is random,
    > > the probability of calling the bellow code will be about 8/10.

    >
    > 1, to be exact.
    >
    >
    > > The above code adds an element to 'rez' whenever she is executed.
    > > A total of 80 M elements.

    >
    > It adds an element everytime. A total of 100M elements.
     
    Goofball, Nov 8, 2006
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ian Suttle
    Replies:
    2
    Views:
    2,347
    Wolfram Rittmeyer
    Aug 27, 2003
  2. Shuo Xiang

    Stack space, global space, heap space

    Shuo Xiang, Jul 9, 2003, in forum: C Programming
    Replies:
    10
    Views:
    2,955
    Bryan Bullard
    Jul 11, 2003
  3. BogusException
    Replies:
    1
    Views:
    2,846
    Vincent van Beveren
    Aug 10, 2006
  4. Ajay

    How to increase stack space/heap space

    Ajay, May 11, 2006, in forum: C Programming
    Replies:
    9
    Views:
    579
  5. Alessandro
    Replies:
    2
    Views:
    741
    Tom Anderson
    Feb 26, 2009
Loading...

Share This Page