java.lang.OutOfMemoryError: Java heap space

G

Goofball

Does anyone have any ideas on why that happened:

[java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap
space

Explaining the situation. I have such code:

for (int i=0; i<10000-1; i++) {
for (int j=i+1; j<10000; j++) {
compareElements(arr, arr[j]);
}
}

Element of the array is a simple class that has two fields (one is
string and another is BitVector) and getters and setters for those
fields. Function compareElements compares the BitVectors of two
elements.

Now the problem: when I run the code, program runs only till then 1334
element in the first cycle. Then I receive the above exception.

Does anyone know, what can that be? Thanks for any help.
 
B

Brandon McCombs

Goofball said:
Does anyone have any ideas on why that happened:

[java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap
space

Explaining the situation. I have such code:

for (int i=0; i<10000-1; i++) {
for (int j=i+1; j<10000; j++) {
compareElements(arr, arr[j]);
}
}

Element of the array is a simple class that has two fields (one is
string and another is BitVector) and getters and setters for those
fields. Function compareElements compares the BitVectors of two
elements.

Now the problem: when I run the code, program runs only till then 1334
element in the first cycle. Then I receive the above exception.

Does anyone know, what can that be? Thanks for any help.


Yeah, your JVM is allocated a certain amount of memory from the OS and
you are creating so many objects at a certain size that your JVM is
running out of memory, hence the OutOfMemoryError exception. You can
tune the size of the memory for the JVM by passing in various parameters
to the java command. I believe they look similar to -X256m or something
like that. You could also think about how your programming is allocating
the data and possibly restructure your algorithm or, if possibly, when
you are done working with the data then get rid of it so the GC can
reclaim the memory.
 
D

Daniel Pitts

Goofball said:
Does anyone have any ideas on why that happened:

[java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap
space

Explaining the situation. I have such code:

for (int i=0; i<10000-1; i++) {
for (int j=i+1; j<10000; j++) {
compareElements(arr, arr[j]);
}
}

Element of the array is a simple class that has two fields (one is
string and another is BitVector) and getters and setters for those
fields. Function compareElements compares the BitVectors of two
elements.

Now the problem: when I run the code, program runs only till then 1334
element in the first cycle. Then I receive the above exception.

Does anyone know, what can that be? Thanks for any help.


If you post an SSCCE (http://www.physci.org/codes/sscce/), we would be
better able to help you.
 
G

Goofball

I was experimenting with gc() in different ways. I was callling it
after processing 1000 elements. However, that didn't help and the
problem remained the same: after processing 1334 (or 1335) elements I
received the OutOfMemory error.

Brandon said:
Goofball said:
Does anyone have any ideas on why that happened:

[java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap
space

Explaining the situation. I have such code:

for (int i=0; i<10000-1; i++) {
for (int j=i+1; j<10000; j++) {
compareElements(arr, arr[j]);
}
}

Element of the array is a simple class that has two fields (one is
string and another is BitVector) and getters and setters for those
fields. Function compareElements compares the BitVectors of two
elements.

Now the problem: when I run the code, program runs only till then 1334
element in the first cycle. Then I receive the above exception.

Does anyone know, what can that be? Thanks for any help.


Yeah, your JVM is allocated a certain amount of memory from the OS and
you are creating so many objects at a certain size that your JVM is
running out of memory, hence the OutOfMemoryError exception. You can
tune the size of the memory for the JVM by passing in various parameters
to the java command. I believe they look similar to -X256m or something
like that. You could also think about how your programming is allocating
the data and possibly restructure your algorithm or, if possibly, when
you are done working with the data then get rid of it so the GC can
reclaim the memory.
 
G

Goofball

Don't know, how much is that an SSCCE, but I am posting you my example:

import java.util.ArrayList;
import java.util.BitSet;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class Example {

private class ArrElem {

private String id;
private BitSet fp;

public BitSet getFp() { return fp; }
public void setFp(BitSet fp) { this.fp = fp; }
public String getId() { return id; }
public void setId(String id) { this.id = id;}
}

private List<ArrElem> arr = new ArrayList<ArrElem>();

private Map<ArrElem, Collection<ArrElem>> rez = new HashMap<ArrElem,
Collection<ArrElem>>();

public void processArr() {
//fillArr(10000);
for (int i = 0; i < 10000-1; i++) {
for (int j = i+1; j < 10000; j++) {
compareElements(arr.get(i), arr.get(j));
}
}
}

private void compareElements(ArrElem elem1, ArrElem elem2) {
int bitM = elem1.getFp().cardinality();
int bitN = elem2.getFp().cardinality();

BitSet commonBits = elem1.getFp();
commonBits.and(elem2.getFp());
int bitCommon = commonBits.cardinality();

double index = bitCommon/(bitM+bitN+bitCommon);
if (index < 0.8) {
if (! rez.containsKey(elem1))
rez.put(elem1, new ArrayList<ArrElem>());
rez.get(elem1).add(elem2);
}
}
}


Daniel said:
Goofball said:
Does anyone have any ideas on why that happened:

[java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap
space

Explaining the situation. I have such code:

for (int i=0; i<10000-1; i++) {
for (int j=i+1; j<10000; j++) {
compareElements(arr, arr[j]);
}
}

Element of the array is a simple class that has two fields (one is
string and another is BitVector) and getters and setters for those
fields. Function compareElements compares the BitVectors of two
elements.

Now the problem: when I run the code, program runs only till then 1334
element in the first cycle. Then I receive the above exception.

Does anyone know, what can that be? Thanks for any help.


If you post an SSCCE (http://www.physci.org/codes/sscce/), we would be
better able to help you.
 
I

Ingo Menger

Goofball said:
private void compareElements(ArrElem elem1, ArrElem elem2) {
int bitM = elem1.getFp().cardinality();
int bitN = elem2.getFp().cardinality();

BitSet commonBits = elem1.getFp();
commonBits.and(elem2.getFp());
int bitCommon = commonBits.cardinality();

double index = bitCommon/(bitM+bitN+bitCommon);
if (index < 0.8) {
if (! rez.containsKey(elem1))
rez.put(elem1, new ArrayList<ArrElem>());
rez.get(elem1).add(elem2);
}
}

There you have it.
The index will be 0.0 unless bitM and bitN are both 0. So you end up
creating at least a new ArrayList element in each cycle.
 
R

Red Orchid

Message-ID: said:
---------- snip -------------

private List<ArrElem> arr = new ArrayList<ArrElem>();

private Map<ArrElem, Collection<ArrElem>> rez = new HashMap<ArrElem,
Collection<ArrElem>>();

public void processArr() {
//fillArr(10000);
for (int i = 0; i < 10000-1; i++) {
for (int j = i+1; j < 10000; j++) {
compareElements(arr.get(i), arr.get(j));
}
}
}


It is mere guesswork.


The above code calls 'compareElements' (10000 * 10000) times.
Namely, 100 M times.




private void compareElements(ArrElem elem1, ArrElem elem2) {
int bitM = elem1.getFp().cardinality();
int bitN = elem2.getFp().cardinality();

BitSet commonBits = elem1.getFp();
commonBits.and(elem2.getFp());
int bitCommon = commonBits.cardinality();

double index = bitCommon/(bitM+bitN+bitCommon);



Min of 'index' is 0.
Max of 'index' is smaller than 1.


if (index < 0.8) {


If 'cardinality()' is random,
the probability of calling the bellow code will be about 8/10.

That is, 80 M times (10000 * 10000 * 0.8)



if (! rez.containsKey(elem1))
rez.put(elem1, new ArrayList<ArrElem>());
rez.get(elem1).add(elem2);
}
}
}


The above code adds an element to 'rez' whenever she is executed.
A total of 80 M elements.

320 MBytes memory (80M * 4) will be required at least.
 
I

Ingo Menger

Red said:
Min of 'index' is 0.
Max of 'index' is smaller than 1.

Yes, it's 0!
Proof:
case 1) bitM and bitM are 0. Then so will bitComnmon. This will result
in zero divide. (BTW, this is a programming error)
case 2) bitM or bitM are > 0. Then the integer division will result in
0.

Thus, index will never be anything else than 0.0. And the programm will
crash occasionally.

Thus, this is always true.
If 'cardinality()' is random,
the probability of calling the bellow code will be about 8/10.

1, to be exact.

The above code adds an element to 'rez' whenever she is executed.
A total of 80 M elements.

It adds an element everytime. A total of 100M elements.
 
D

Dijon Yu

Goofball said:
I was experimenting with gc() in different ways. I was callling it
after processing 1000 elements. However, that didn't help and the
problem remained the same: after processing 1334 (or 1335) elements I
received the OutOfMemory error.

I have meet the problem too.
Don't depend on gc(), whatever 。
If you want resolve the problem, you can set the JVM.

You can use java -Xmx256m to avoid the problem.
But I think maybe you can refactor and redesign the algorithm.
 
G

Goofball

Thanks a lot. That helped. I redesigned the alsorithm. Now it works
fine. But when I test it on about 1 million of elements - it crashes.
Starting JVM with more memory helps, but that sometimes cannot be done
(because of the requirements). So, I decided to write my own
implementations of List and Map interfaces, that will flush the results
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,733
Messages
2,569,440
Members
44,830
Latest member
ZADIva7383

Latest Threads

Top