Available memory and limit on thread creation

[Joseph Dionne]

Think in terms of how an HTTP server works. Requests come in on
various sockets, but usually a request is a stateless GET. There is no
need to remember the state of what that socket was doing previously.
There does not need to be a thread waiting for that socket. As soon
as the i/o completes, the packet can be put into a queue and serviced
by a pool of threads.

There is no polling involved. The hardware gives you an interrupt
when the i/o completes. The queue of packets to be processed can be
empty, but even then that can be handled with a synchronization lock,
no polling required.

The i/o is not the problem. The polling I speak of is monitoring the
external, detached applications for their completion. Then I will need
to send the results back to the correct socket.

There are so many things you could potentially do with sockets. We
don't yet know what he is up to, so all we can do is toss out
possibilities.

I have no control of the application I will be calling. It does not use
stdin,stdout, or stderr for returning results. The clients that will be
connecting are other vendors applications, mostly legacy code that has
basic TCP/IP functionality, and definitely does not have HTTP logic.

Since my application is the middle man, without any say into the design
of the app that will be my connected clients, nor say into the
functionality of the back end application, I have little control of
anything other than my application "bridge."

Joseph

 

[Sudsy]

Joseph Dionne wrote:

Because I have a frugal customer who is already not terribly excited
about paying my bill? The real world sucks!

Read Roedy's reply carefully. He was able to succinctly describe the
limitations of the HTTP protocol, namely that it's stateless. What I
would suggest you consider is a hybrid of available technologies.
Given what you've described to-date, I would lean towards a JMS
solution. This would provide "loose coupling" and incorporates some
of Roedy's suggestions in terms of queueing requests/responses.
Insofar as J2EE servers, have you had a look at JBoss? While I
wouldn't recommend it for commercial, mission-critical applications,
it can serve as a powerful development platform when linked with the
Eclipse IDE. It might be enough to demonstrate to your client that
there IS a better way.
I'm just throwing out suggestions as what you're describing is quite
complex and would involve a major investment in time for development.
That will translate to $$, no matter how much or little you charge
per hour.

 

[Roedy Green]

The i/o is not the problem. The polling I speak of is monitoring the
external, detached applications for their completion. Then I will need
to send the results back to the correct socket.

If the problem is too many threads, you can reduce them by as the last
step of the exec stream you send a packet back to a common port where
the packet contain nothing but the id number of the exec. You might
even use a UDP protocol to avoid the overhead of setting up sockets.

When the "i'm done" packet comes in, you can look up the corresponding
object with a HashMap -- no thread or or socket at this point, and
move it to the next stage queue.

This sort of transaction processing was what we did back in the days
of minis with less than a meg of RAM. IBM's CICS was one the first of
these.

 

[Joseph Dionne]

Joseph Dionne wrote:



Read Roedy's reply carefully. He was able to succinctly describe the
limitations of the HTTP protocol, namely that it's stateless. What I
would suggest you consider is a hybrid of available technologies.
Given what you've described to-date, I would lean towards a JMS
solution. This would provide "loose coupling" and incorporates some
of Roedy's suggestions in terms of queueing requests/responses.
Insofar as J2EE servers, have you had a look at JBoss? While I
wouldn't recommend it for commercial, mission-critical applications,
it can serve as a powerful development platform when linked with the
Eclipse IDE. It might be enough to demonstrate to your client that
there IS a better way.
I'm just throwing out suggestions as what you're describing is quite
complex and would involve a major investment in time for development.
That will translate to $$, no matter how much or little you charge
per hour.

Thank you for your thoughtful reply. I assure you that I consider all
the advice Roedy offers. I find him a wealth of information.

I must confess, I am new to Java, and only have a cursory knowledge of
web development. Unfortunately my delivery date is very tight, tighter
than the learning curve needed to explore too much new technology. My
customer base gives me access to many different hardware and OS
platforms, and I have found this limit I am experiencing seems isolated
to Linux. However, every problem has a solution, and I have several
options to overcome this current opportunity.

Throughput is the highest concern. I estimate hits in the thousands so
I must be careful not to add to much processing overhead. The choice of
language was part of the contract, and I always love a challenge. Besides,

Thanks to everyone who offered advice, it is greatly appreciated.

Joseph

 

[Joseph Dionne]

If the problem is too many threads, you can reduce them by as the last
step of the exec stream you send a packet back to a common port where
the packet contain nothing but the id number of the exec. You might
even use a UDP protocol to avoid the overhead of setting up sockets.

When the "i'm done" packet comes in, you can look up the corresponding
object with a HashMap -- no thread or or socket at this point, and
move it to the next stage queue.

This sort of transaction processing was what we did back in the days
of minis with less than a meg of RAM. IBM's CICS was one the first of
these.

First, I need to test the use of Linux virtual ip as a potential
solution. This will allow me to have four VMs accepting connections on
a single CPU, and I can sell the customer a multiple NIC card to improve
IP traffic even more.

Second, using the typical consumer/producer model, allocate general
purpose socket readers in a pool, another pool to process the received
packets, and a third pool to manage the external application execution.

One question that I could use advice, is it possible to use Channels for
Process management? My thought is to monitor the output stream of the
Process for OP_READ, and wait for EOF when the process terminates. Any
flaws in my thought?

Joseph

 

[Roedy Green]

One question that I could use advice, is it possible to use Channels for
Process management? My thought is to monitor the output stream of the
Process for OP_READ, and wait for EOF when the process terminates. Any
flaws in my thought?

One area I still have not explored is nio. I have done such things at
very low level poking hardware devices, but not with the nio package.

My thinking is warped by what the hardware can typically do.

I think back to a ferry reservation system for the BC Government. I
worked with guy who was one of the designers of IBM's channel
architecture and decided to implement the system with astounding
throughput by making the channel (a sort of smart DMA controller) do
much of the work. I thought he was mad. Yet I had to admit, he got
throughput orders of magnitude better than he could have with a more
general solution on what we would consider a tiny machine today. He
did not do everything this way, just the few crucial transactions that
made the overwhelming bulk.

 

[Roedy Green]

I must confess, I am new to Java, and only have a cursory knowledge of
web development. Unfortunately my delivery date is very tight, tighter
than the learning curve needed to explore too much new technology.

My reaction in this deadline environment is to go a little LOWER level
than I would otherwise. I am better off to write code that uses a
smaller subset of tools, but ones that I fully understand.

Using a high level tool presents endless surprises because it takes
years to fully understand it. Further, there is no guarantee it will
sit still. You may have to KEEP dancing to KEEP your code working.
Further the package could deteriorate or lose support. So often you
are working with a black box. Even with source, only the original
developers or someone who makes a full time job of learning the source
can find their way around to track problems. If the code is buggy, you
are better off with the smaller more specialised solution you wrote
yourself. On the other hand, with a commonly used tool, there are
others who can help you track problems.

So, when I want to be sure it will work, I like to have as much of the
code under my personal control as possible. Other people's code gives
me at least ten times more trouble than my own. The main reason is I
can understand my own documentation and I tend to document that which
I know I will want to know later.

If you are starting out on what you know will be a very large project
that will go on for years, maintained by many different people, then
you want to take some time to find some good high level tools,
preferably ones with alternate sources.

 

[Joseph Dionne]

One area I still have not explored is nio. I have done such things at
very low level poking hardware devices, but not with the nio package.

[snip]

I have been watching Java develop since '96. I avoided it even though I
admired the effort for several reasons. But 1.4 has changed my
thinking, and the nio package has addressed my last concern, moving back
towards multiplex technology. The package allows an alternative to
launching threads for each new client, a time consuming process.

I am very impressed with Java, and for my business, a language that
allows me to develop on my favorite OS/hardware and deploy on so many
others without porting is ideal.

I need to get up to speed on JSP and Struts, which opens up even more
opportunities. I commend Sun for bring to the community of developers a
platform that encourages the exchange of ideas, replacing the isolation
we have had for so many years.

Joseph

 

[Roedy Green]

Throughput is the highest concern.

I always wanted some sort of animation tool so you could watch data
flowing. You would watch a typical transaction flowing through the
system. Visual representations of Objects would bloat up to represent
their relative RAM use. Time delay would match CPU or other resource
usage.

Watching endless repetitions of this would quickly point out where you
are being "silly".

It would be sort of like taking a tour of a Coke bottling factory,
watching the entire process from various vantage points.

The sort of things you would notice is carefully setting up a socket
then throwing it away and creating another just like it. Huge numbers
of objects lying around doing nothing. Huge amounts of information
endlessly repeated.

You can get a bit of a feel by doing an assembler level trace and
watching the incredible overhead of method calls, the endless buck
passing.

There are two kinds of programmers, those anal retentive types to try
to optimise everything whether it needs it or not, in the process
making the code hard to read, and programmers terrified to do ANYTHING
out of the usual, even if one line done a slightly odd way will give
you a 100 fold increase in speed. They are anally retentive in wanting
everything done in a uniform way, and feel than any sort of
optimisation is evil unless done at gunpoint and only after you did it
some klutzy way first. They flatly refuse to think about what is
happening under the covers and pride themselves on making non-optimal
choices even when both are equally easy to code. They misunderstand
Knuth's admonition that premature optimisation is the root of all
evil. They think that means you should not think about optimisation
until after the product has shipped. It has a place before you even
write a line of code, in choosing the best data structures and
algorithms. You just should not do anything tricky without proof it
is needed.

There are few non-anal retentives in programming. They are soon fired.

 

[Joseph Dionne]

Roedy Green wrote:

[snip]

There are two kinds of programmers, those anal retentive types to try
to optimise everything whether it needs it or not, in the process
making the code hard to read, and programmers terrified to do ANYTHING
out of the usual, even if one line done a slightly odd way will give
you a 100 fold increase in speed.

I guess I fall into a place between the two, an acute case of
hemorrhoids. I began life creating circuit cards, and years working in
assembler, until c arrived on the scene. Even in c I had to do some
optimization, buy only at the points where I would get the greatest bang
for the buck. The reality of software development is to consider the
future maintenance. Code that is difficult to understand will die a
quite death, neglected when the author loses interest. Code that is
easy to understand will live on, being extended to fit the needs of the day.

Joseph

 

[kjc]

JBoss is free

Because I have a frugal customer who is already not terribly excited
about paying my bill? The real world sucks!

 

[kjc]

Sounds like you're going to be writing an AppServer.

Roedy Green wrote:

[snip]

There are two kinds of programmers, those anal retentive types to try
to optimise everything whether it needs it or not, in the process
making the code hard to read, and programmers terrified to do ANYTHING
out of the usual, even if one line done a slightly odd way will give
you a 100 fold increase in speed.

I guess I fall into a place between the two, an acute case of
hemorrhoids. I began life creating circuit cards, and years working in
assembler, until c arrived on the scene. Even in c I had to do some
optimization, buy only at the points where I would get the greatest bang
for the buck. The reality of software development is to consider the
future maintenance. Code that is difficult to understand will die a
quite death, neglected when the author loses interest. Code that is
easy to understand will live on, being extended to fit the needs of the
day.

Joseph

 

[John C. Bollinger]

One area I still have not explored is nio. I have done such things at
very low level poking hardware devices, but not with the nio package.

[snip]

I have been watching Java develop since '96. I avoided it even though I
admired the effort for several reasons. But 1.4 has changed my
thinking, and the nio package has addressed my last concern, moving back
towards multiplex technology. The package allows an alternative to
launching threads for each new client, a time consuming process.

So, then, why aren't you multiplexing your socket listening? That would
save you a whole lot of threads. You would want to look into the
Selector class, which, with some associated classes, provides more or
less an OO version of C's select() function.

You wrote earlier that the back-end app does not make use of stdin /
stdout / stderr to communicate with the caller. Can you wrap it in a
script or other wrapper program that will adapt it to do so? I think
that would allow you to use NIO to multiplex monitoring of the state of
the back-end process invocations as well, hence giving you an entry into
implementing some of the suggestions Roedy has been offering.

It really should not be necessary to spawn hundreds of threads for an
application like this. Much as Roedy already said, it is probably not
very useful to launch more than a handful of threads if you have a way
to use that handful to perform all the needed work.


John Bollinger
(e-mail address removed)

 

[Joseph Dionne]

One question that I could use advice, is it possible to use Channels
for Process management? My thought is to monitor the output stream
of the Process for OP_READ, and wait for EOF when the process
terminates. Any flaws in my thought?




One area I still have not explored is nio. I have done such things at
very low level poking hardware devices, but not with the nio package.

[snip]

I have been watching Java develop since '96. I avoided it even though
I admired the effort for several reasons. But 1.4 has changed my
thinking, and the nio package has addressed my last concern, moving
back towards multiplex technology. The package allows an alternative
to launching threads for each new client, a time consuming process.

So, then, why aren't you multiplexing your socket listening? That would
save you a whole lot of threads. You would want to look into the
Selector class, which, with some associated classes, provides more or
less an OO version of C's select() function.

Actually, as I have said before, I am using channels for socket
management, and assigning new connections to a connection pool of
created client threads.

You wrote earlier that the back-end app does not make use of stdin /
stdout / stderr to communicate with the caller. Can you wrap it in a
script or other wrapper program that will adapt it to do so? I think
that would allow you to use NIO to multiplex monitoring of the state of
the back-end process invocations as well, hence giving you an entry into
implementing some of the suggestions Roedy has been offering.

Now, I have explored the NIO library, but have not found a
ProcessChannel that monitors a Process, nor a implementation of a
StreamChannel. To implement either as a channel for SelectableChannel I
would need more threads to watch the Process streams. I want to use
blocking threads, to not needlessly eat CPU time, so I would need a
thread per "transaction," or connection in my case. Thus I would need
even more threads.

It really should not be necessary to spawn hundreds of threads for an
application like this. Much as Roedy already said, it is probably not
very useful to launch more than a handful of threads if you have a way
to use that handful to perform all the needed work.

And, you are correct. one thread can handle any number of requests, if
the remote clients are willing to wait to be service. However, as I
have also said, throughput is critical. The remote clients will not
wait long, perhaps five to ten seconds, before the connection is aborted
from the peer. Since the remote client MUST communicate via my
application, to get real work done, aborted connections will result in a
new, quickly generated, connect attempt, and another socket to service,
while the old socket, perhaps waiting in a queue for processing, is now
dead. This potential will only add to the burden of the application I
am creating.

But, if you have suggestion for using NIO channels to monitor Process
object, I would be greatly interested in your input. I have confirmed
that on most OS/hardware I have access to, I can create the desired
number of threads. The problem seems to be localized to Linux, and I
believe MIGHT be a kernel setting, however I have already ruled out
threads-max. On some Linux systems I have tested, I am unable to even
launch the 1015 threads I can on my development system.

Additionally, I have tried JVM 1.5.0(beta), and found that by using the
"-Xprof" option, the limit did not appear, yet without that option, I
still got a 1015 thread limit. I was hoping someone from Sun monitors
this group, and would perhaps offer a reason for the limit I am noticing
on Linux ONLY.

 

[Joseph Dionne]

I am using Linux 2.4 with JVM 1.4.2_04. I have noticed a limit to the
number of threads that can be created. I tried many settings of "-Xmx"
and "-Xss" and the largest number of threads I can launch is 1015. So,
I have several questions.

First, does available memory limit number of threads?
Second, if available memory does limit the number of threads, is there a
rule of thumb on how to calculate that limit?
Finally, exactly what affect do "-Xmx" and "-Xss" have to the VM and do
they affect thread memory at all?

thanks
Joseph

For other people having problems starting thousands of threads from Java
with Linux, the following kernel parameters will limit the number of
threads, and result in an OutOfMemoryError.

/proc/sys/kernel/threads-max <int value>
/proc/sys/kernel/rtsig-max <int value>

While threads-max might be large enough, I have found rtsig-max set very
low, on my system it was set to 1024 even though my threads-max was set
to 8184. I do not know what, is any, relationship exists between the
two values. On some linux systems, installed "out-of-the-box" so to
speak, these values can be very low.

Thanks to all that provided input to my request. Your thoughts and
suggestions are very appreciated.

Joseph

 

[Andrew Thompson]

I was hoping someone from Sun monitors
this group, and would perhaps offer a reason
for the limit I am noticing on Linux ONLY.

Posting to this group sounds like a strange
way to to go about getting a response from Sun.

They have specialized forums of their
own, bug reports, contact forms..
This, in contrast, is 'the jungle'.

 

[marcus]

OK, I did not read all the replies (most of them over my head!), so this
might have been covered . . .

Your idea of using virtual IP's or another mechanism to run multiple
JVM's seems sound for various reasons. You mentioned server farms,
which indicates you will need to do some amount of distributed
processing eventually anyway, so the logic of distributing the load will
not change dramatically from multiple JVM's or multiple machines.

I use apache to handle connect to tomcat through mod_jk, which specifies
an internal port (JkMount /shop/servlet/* ajp12). It is easy enough for
apache to mount to multiple internal ports using their virtual server
based on the domain of the request, so I wouldn't have to mess with IP's
or even ports (mydomain.com:8080).

Since you already have load balancing in the final equasion, it seems to
me the load balancer could be configured to make requests on
loader1.mydomain.com, loader2.mydomain.com, etc, which transparently
point to separate boxes or separate JVM's on a given box, and you can
deal with the fine points of balancing and maxamising server usage (also
over my head *grin*) when the load is actually present.

(My internal editor suggests the load balancer might have to resort to
rewriting to specific ports like loader1.mydomain.com:8081 and you may
have to configure various apps to start and run listening on
corresponding ports, rather than having another app like apache
distributing requests, but that seems trivial.)

-- clh

 

[John C. Bollinger]

Now, I have explored the NIO library, but have not found a
ProcessChannel that monitors a Process, nor a implementation of a
StreamChannel. To implement either as a channel for SelectableChannel I
would need more threads to watch the Process streams.

You're quite right; I had misremembered some of the NIO details. I was
thinking that you could get a SelectableChannel from System.in, which
would then not have required additional threads, but it doesn't look
like that is possible. My apologies.

The general idea may still be applicable, however: if you wrap the
back-end such that it reports task completion by communicating over a
socket to your middleware application then you certainly can use NIO to
multiplex those notifications, as both SocketChannel and
ServerSocketChannel implement SelectableChannel.

So how _does_ the back end report that it has finished a task, anyway?

I want to use
blocking threads, to not needlessly eat CPU time, so I would need a
thread per "transaction," or connection in my case. Thus I would need
even more threads.

You certainly want to avoid putting threads into idle loops, but having
many blocking threads is not the only alternative. Roedy has already
covered this ground, though, and you don't seem to have a problem with
the concept -- except that you keep insisting that you don't want to do
the job that way.

And, you are correct. one thread can handle any number of requests, if
the remote clients are willing to wait to be service. However, as I
have also said, throughput is critical. The remote clients will not
wait long, perhaps five to ten seconds, before the connection is aborted
from the peer. Since the remote client MUST communicate via my
application, to get real work done, aborted connections will result in a
new, quickly generated, connect attempt, and another socket to service,
while the old socket, perhaps waiting in a queue for processing, is now
dead. This potential will only add to the burden of the application I
am creating.

The server has a limited capacity to perform work. If the server cannot
adequately accommodate the demand on its resources then using more
threads will not fix the problem -- threads that are runnable (i.e. not
blocked) but not currently running do not increase or make any better
use of the server's capacity. Indeed, you may incur a noticeable
overhead in the thread scheduler when there are many runnable threads
contending for CPU time. The number of threads required to sufficiently
minimize the likelihood that they may all be blocked at the same time
will depend on the application, hardware, and OS, but that is the number
of threads you should be shooting for to tune the application's
performance. Much of this discussion has been aimed at approaches to
designing the application so that the required number is as small as
possible while at the same time ensuring that the application itself is
as responsive as possible. These are compatible goals.

But, if you have suggestion for using NIO channels to monitor Process
object, I would be greatly interested in your input.

You don't necessarily need to monitor the Process object itself. Ideal
would be to arrange so that the native process actively notifies your
application when it is done. Some of my comments above were directed at
enabling that. Such a scenario could allow you to multiplex the
monitoring, thus requiring fewer threads and possibly yielding a
performance improvement over your preferred approach of using a large
number of dedicated threads. It evidently would make your application
more portable, too.


John Bollinger
(e-mail address removed)

 

[Michael Borgwardt]

For other people having problems starting thousands of threads from Java
with Linux, the following kernel parameters will limit the number of
threads, and result in an OutOfMemoryError.

/proc/sys/kernel/threads-max <int value>
/proc/sys/kernel/rtsig-max <int value>

While threads-max might be large enough, I have found rtsig-max set very
low, on my system it was set to 1024 even though my threads-max was set
to 8184. I do not know what, is any, relationship exists between the
two values.

Looking around a bit, the second doesn't seem directly related to threads; it
limits the number of "realtime signals " that can be queued. I suppose that
a blocked thread takes up one such queue slot.