Threads and Virtual Memory

[Stig Christiansen]

Hi all,
I must be missing something obvious, but I really have a problem with
(small) threads using virtual memory. I've read people saying that
more than 1000 threads is possible in java, but when I run this
(simple) code:

public class tst implements Runnable
{
public static void main(String args[])
{
for (int g = 0 ; g < 1000 ; g++)
{
Thread t1 = new Thread(new tst()) ;
System.out.println("x: " + g) ;
t1.start() ;
}
}
public void run()
{
System.out.println("t.1") ;
}
}


I get max 280 threads before getting a "out of memory" error:

Exception in thread "main" java.lang.OutOfMemoryError: unable to
create new native thread
at java.lang.Thread.start(Native Method)
at tst.main(tst.java:10)


At that time, the program has used 3GB of virtual memory - why??

I've tried setting the -Xss and -Xms command line options and they do
have an influence, but I still get 280 threads max.

My OS is: Linux onc-dl-1 2.4.21-4.EL #1 Fri Oct 3 18:13:58 EDT 2003
i686 i686 i386 GNU/Linux
and I'm using:
java version "1.3.1_13"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1_13-b03)
Java HotSpot(TM) Client VM (build 1.3.1_13-b03, mixed mode)

from Sun.


Does anyone know what I'm missing here? As I see it, the problem must
be the virtual memory usage, but how can I influence this? Do I need
to look at some OS settings?

Any help will be greatly appreciated
Thanks
Stig Christiansen

 

[xarax]

Hi all,
I must be missing something obvious, but I really have a problem with
(small) threads using virtual memory. I've read people saying that
more than 1000 threads is possible in java, but when I run this
(simple) code:

public class tst implements Runnable
{
public static void main(String args[])
{
for (int g = 0 ; g < 1000 ; g++)
{
Thread t1 = new Thread(new tst()) ;
System.out.println("x: " + g) ;
t1.start() ;
}
}
public void run()
{
System.out.println("t.1") ;
}
}


I get max 280 threads before getting a "out of memory" error:

Exception in thread "main" java.lang.OutOfMemoryError: unable to
create new native thread
at java.lang.Thread.start(Native Method)
at tst.main(tst.java:10)


At that time, the program has used 3GB of virtual memory - why??

I've tried setting the -Xss and -Xms command line options and they do
have an influence, but I still get 280 threads max.

Did you try: -Xms1500m -Xmx1500m

That will start the JVM with 1500MB and set the
max heap at 1500MB. I don't think the 32-bit Sun JVM
can go much higher than that.

 

[Chris]

Hi all,
I must be missing something obvious, but I really have a problem with
(small) threads using virtual memory. I've read people saying that
more than 1000 threads is possible in java, but when I run this
(simple) code:

I get max 280 threads before getting a "out of memory" error:

Do you have a compelling reason to create 1,000 threads? Performance
declines when you use a large number of threads because of all the
context-switching. Not knowing your app I can't recommend an alternative,
but you might consider creating a smaller pool of threads and grabbing them
as needed. If your need for threads is IO-related, you might also look at
Selectors in the java.nio.* package.

 

[Erich Reimberg N.]

Stig,

Did you try using 1.5.0 or any other newer version? Your code works
perfect in my jdk1.5.0. All the 1000 threads are created and started
without any problem.

Erich

Hi all,
I must be missing something obvious, but I really have a problem with
(small) threads using virtual memory. I've read people saying that
more than 1000 threads is possible in java, but when I run this
(simple) code:

public class tst implements Runnable
{
public static void main(String args[])
{
for (int g = 0 ; g < 1000 ; g++)
{
Thread t1 = new Thread(new tst()) ;
System.out.println("x: " + g) ;
t1.start() ;
}
}
public void run()
{
System.out.println("t.1") ;
}
}

I get max 280 threads before getting a "out of memory" error:

Exception in thread "main" java.lang.OutOfMemoryError: unable to
create new native thread
at java.lang.Thread.start(Native Method)
at tst.main(tst.java:10)

At that time, the program has used 3GB of virtual memory - why??

I've tried setting the -Xss and -Xms command line options and they do
have an influence, but I still get 280 threads max.

My OS is: Linux onc-dl-1 2.4.21-4.EL #1 Fri Oct 3 18:13:58 EDT 2003
i686 i686 i386 GNU/Linux
and I'm using:
java version "1.3.1_13"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1_13-b03)
Java HotSpot(TM) Client VM (build 1.3.1_13-b03, mixed mode)

 

[Mike Schilling]

Hi all,
I must be missing something obvious, but I really have a problem with
(small) threads using virtual memory. I've read people saying that
more than 1000 threads is possible in java, but when I run this
(simple) code:

public class tst implements Runnable
{
public static void main(String args[])
{
for (int g = 0 ; g < 1000 ; g++)
{
Thread t1 = new Thread(new tst()) ;
System.out.println("x: " + g) ;
t1.start() ;
}
}
public void run()
{
System.out.println("t.1") ;
}
}


I get max 280 threads before getting a "out of memory" error:

Exception in thread "main" java.lang.OutOfMemoryError: unable to
create new native thread
at java.lang.Thread.start(Native Method)
at tst.main(tst.java:10)


At that time, the program has used 3GB of virtual memory - why??

I've tried setting the -Xss and -Xms command line options and they do
have an influence, but I still get 280 threads max.

My OS is: Linux onc-dl-1 2.4.21-4.EL #1 Fri Oct 3 18:13:58 EDT 2003
i686 i686 i386 GNU/Linux
and I'm using:
java version "1.3.1_13"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1_13-b03)
Java HotSpot(TM) Client VM (build 1.3.1_13-b03, mixed mode)

from Sun.


Does anyone know what I'm missing here? As I see it, the problem must
be the virtual memory usage, but how can I influence this? Do I need
to look at some OS settings?

Very odd. When I run your program, unmodified, on Windows XP using JDK
1.4.1_04, all 1000 thread are created and the memory usage never exceeds 2
MB (yes, MB, not GB).

The way this program works, the threads can exit almost immediately, so
there aren't anything like 1000 simultaneously active threads. But even if
I keep the threads around by having them wait forever (and making them
daemons, so the process exists when the main thread does) it still never
exceeds 2 MB. Likewise with JDK 1.4.0 under Solaris .

 

[Harish]

Hi all,

At that time, the program has used 3GB of virtual memory - why??

are you saying the process reached 3gb memory and threw an out of memory
error?
or because of the exception you assume the process would've reached 3gb?

do you have a ulimit on memory or number of threads?

or how about the OS as such, does it have any configurable parameter which
limits the total number of threads?
does linux, depending the machine hardware, memory and other resources
dynamically put a limit on number of threads?

 

[Stig Christiansen]

(e-mail address removed) (Stig Christiansen) wrote in message
First of all, thanks to all the people taking time to answer my
question.

Google gives me an error when I try to answer each of your individual
postings, so I'll try to answer you all in this one posting although
it's a bit rude to you all:

xarax:
<<Did you try: -Xms1500m -Xmx1500m>>
yes, that only worsen things. Actually, the lower I set it, the more
threads I can create. I believe its a race between the "native heap"
and the "java heap" - the more memory to the "java heap" the less to
"native heap" and therefore fewer threads.

Chris:
Well, it's going to be used in a download servlet (Tomcat), designed
to have a thread for getting and zipping a stream on-the-fly while
another thread is returning the bytes to the requestor (using a tmp.
file as intermediate storage).
I don't know how many concurrent downloads we're going to get, but I'd
like to know that the threads isn't going to be a problem. Right now,
I'd say about 130 would make the service fail, I'd like to be sure
that at least 150 would be possible.

Erich Reimberg N.:
No, I only tried to versions of 1.3.1 because this is what our
production uses. Upgrading may be a possibility - especially if it
works.
What OS are you using?

Mike Schilling:
2MB! so, there must really be something wrong somewhere on my
installation (??) - I'll try using another server to see if it helps.
Can this really be JVM related? You're using XP and Solaris - it's
nice to *know* that it can work. I'll try using another OS and JVM.
Actually, you caught me, I too had a Thread.sleep() in the created
threads in order to have time to see how much memory it uses.
But - you are checking virtual memory, right? Because the "real"
memory usage I see is also very small.

Harish:
I'm using this Linux command to see the virtual memory usage:
ps -e -o pid,vsz,rss,comm |grep java
and it reports just under 3GB every time the program stops.
If I change the Xss and Xms parameters the number of threads vary, but
the "vsz" always reports just under 3GB.
The OS, yes. "ulimit -a" gives me:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) 4
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 7168
virtual memory (kbytes, -v) unlimited

Doesn't this look ok?
I'm not a Linux expert - far from - should I look for limits anywhere
else?


All:
I quickly tried running the program on an XP with JDK 1.4.2_02 and it
works! As far as I see it, for a short while it uses 24M of virtual
memory, then it sinks to 10MB (I modified the example code to
Thread.sleep() in each thread so I have time to see the mem usage
before the program exits)
I also tried using JDK 1.3.1 on the same PC - same result!
I really didn't expect this - a bit naive, I thought the OS shouldn't
matter. I should have done this before posting the question here -
sorry.


SO - would it be fair to say this must be a OS/JVM issue? Not a native
java problem? And that I should be looking for the problem in the
OS/settings of the OS??
Do any of you have any suggestions to where I should go now? where
should I be looking? I'll try installing a new JDK on the server (if
the admin allows me to) to see if it helps.

Thanks!!

Stig Christiansen

 

[Andrew Thompson]

Google gives me an error when I try to answer each of your individual
postings, ..

Please report it to them, if you would be so kind.

 

[Chris Uppal]

Well, it's going to be used in a download servlet (Tomcat), designed
to have a thread for getting and zipping a stream on-the-fly while
another thread is returning the bytes to the requestor (using a tmp.
file as intermediate storage).

I doubt if that's the best way to implement it. A smallish pool of zipping
threads would not be much work to implement and would scale more gracefully and
more controllably.

Basically your problem is that you are using an architecture that assumes that
threads are cheap. Unfortunately, in typical JVM implementations, threads are
/not/ cheap.

Using a 64-bit JVM would probably help (much more address space available for
threads' stacks). Using the -ms JVM option may help (IIRC that's the flag for
setting stack sizes). Using a JVM with a different thread implementation may
help (but maybe not, since much of the work will be done in native ZLIB
compression code -- which needs a "real" C-style stack). Changing your
architecture /will/ help...

-- chris

 

[Thomas G. Marshall]

Chris coughed up:

Do you have a compelling reason to create 1,000 threads? Performance
declines when you use a large number of threads because of all the
context-switching. Not knowing your app I can't recommend an
alternative, but you might consider creating a smaller pool of
threads and grabbing them as needed. If your need for threads is
IO-related, you might also look at Selectors in the java.nio.*
package.

Phlip pointed out something very important a while ago, which should be
tattooed on the foreheads of most of us.

Believe me now or believe me later, but threading should not be jumped to
until you've exhausted using your own (possibly round-robin) techniques
within a single (or /very very few/) threads.

Threading gets even seasoned folks into huge trouble for a myriad of
reasons. The biggest reason is that threading seems to be something that
everyone thinks they can master, and few can. I can't tell you how many
times I've been brought in to fix code and found things like this:

thing.set(thing.get() + 1); // bring thing up by one

....which the coders believed to be safe simply because the set() and get()
methods were synchronized. Laugh now, but I've seen this over and over.
Usually involving "thread safe" synchronized ArrayLists, but that's a
detail.

Maintainability of threads is just not generally easy.

 

[Thomas G. Marshall]

Chris Uppal coughed up:

Well, it's going to be used in a download servlet (Tomcat), designed
to have a thread for getting and zipping a stream on-the-fly while
another thread is returning the bytes to the requestor (using a tmp.
file as intermediate storage).

I doubt if that's the best way to implement it. A smallish pool of
zipping threads would not be much work to implement and would scale
more gracefully and more controllably.

Basically your problem is that you are using an architecture that
assumes that threads are cheap [...]

....and commonly maintainable by most engineers that will later view your
code...

 

[Erich Reimberg N.]

Stig,

I tested it using Windows 2000 SP4

ER

 

[Stig Christiansen]

(e-mail address removed) (Stig Christiansen) wrote in message
Hi all,
Once again, thanks for your feedback!

Still unable to reply to your individual postings, so once again a
collective answer:

Andrew:
Yes, good idea - I've just reported it.

Chris:
Yes, your'e right! - I did assume threads were cheap. I'll redesign
the approach. Thanks for pointing out that I was going down a wrong
road.
(But, it's still a problem not being able to open many threads if Java
is supposed to...)
Your point about the ZLIB is also good - even if threads were cheap
this chould become a problem.


All:
I installed JDK 1.4 on the server: it works with no problem.
I then installed the latest release of the 1.3.1 JDK and it still uses
3GB. So, a problem with JDK 1.3.1 on this version of the OS??
I should get access to a new Linux install tonight or tomorrow, then
I'll see if the problem persists or if it's related to the server
installation.

I'm still very interested in any suggestions to what to look for in
the OS - I'm not a Linux expert, so this is a bit of a mystery to
me...


Regards
Stig

 

[Harish]

http://java.sun.com/docs/hotspot/threads/threads.html
the default thread model in 1.3 and 1.4 is different.
but this is in solaris, may be its true for linux.
"Many-to-Many thread based synchronization" is default in 1.3, and 1.4 has
"Many-to-Many lwp based synchronization"
can you try changing the threading model and see if this makes any
differences?

 

[Mike Schilling]

(e-mail address removed) (Stig Christiansen) wrote in message
Mike Schilling:
2MB! so, there must really be something wrong somewhere on my
installation (??) - I'll try using another server to see if it helps.
Can this really be JVM related? You're using XP and Solaris - it's
nice to *know* that it can work. I'll try using another OS and JVM.
Actually, you caught me, I too had a Thread.sleep() in the created
threads in order to have time to see how much memory it uses.
But - you are checking virtual memory, right? Because the "real"
memory usage I see is also very small.

2MB is actually the size of the Java heap, as reported by verbose:gc.
Virtual memory usage would be somewhat higher, but not by 3 orders of
magnitude.

 

[stigc]

Please report it to them, if you would be so kind.

I did, was told it was a known error from international domains, they
suggested to us Google Groups beta - which I now do, hope it works!
Regards
Stig

 

[stigc]

Thanks Erich

From what I hear here and have now tested myself, it seems to be a

problem with 1.3.1 on our servers (1.4 works ok).

Regards
Stig

 

[stigc]

Hi Thomas,
so true. I don't claim to master it either, but I do think I'm beyond
the "thing.set()" example you show....
Thing is, that letting functions run in their own world, doing their
thing and only that, seems so tempting. In this case, 2 functions each
doing different things, no need to clutter the logic in either
functions.
But then again... as you say, things do get ugly when trying to
coordinate to 2.
I'm going to re-think the strategy.

Thanks
Stig

 

[stigc]

Thanks Mike,
I've done some testing myself now on XP - as you say, no significant
memory usage. And on the Linux server the problem also disappears when
using 1.4.

Regards
Stig

 

[Andrew Thompson]

On 11 Jan 2005 07:14:36 -0800, Stig Christiansen wrote:

(Report Google problems)

..I've just reported it.

Thank you.

[ Hope you get your technical matter sorted. I'm reading
the thread to pick up tips.. ;-) ]