Game Company- Java Server Thread Priority

[BlackHawke]

Hello!

This is my second post. Ppl really helped me with the first. I hope there
are answers for this one as well

I own a game company (www.aepoxgames.net) releasing the beta for our first
sci-fi space game in about 2 weeks (www.andromedaonline.net)

We have multiple servers, but the game engine (Game Server Program) is a
Java program (don't ask why) on a windows 2003 enterprise system with dual
xeon p4 processors and 4GB of RAM. This program needs to process orders for
thousands of ships as fast as possible.

When we first set this up, the Game Server Program took exactly 50% of the
CPU cycles when idle, and up to 80% or 90% when working (processing a turn,
for example) and could process a turn of 10,000 ships in 15 seconds.

We revamped some of the major systems (the code and the computer itself),
did a fresh install of Windows 2003 on a different system (still dual Xeon,
but faster MB and CPU), migrated the Game Server Program there, changed the
threadpool priorities in the code, etc.

We've now noticed that on idle (IE not processing turns), the Game Server
Program takes 0% of the CPU load, and when processing a turn goes up to
3-7%. It never exceeds 7%, and takes FOREVER now when processing a turn. It
is the only program running on the machine, and services are at a minimum.
Even with 99% of the CPU resources free (IE on System Idle), the game will
never take more than about 7% of the processor time.

This is extremely serious.

We've tried increasing the priority of Java, but
a) The only way I know to do this is through the Task manager, meaning
it would have to be done every time we run the server
b) I increased the priority to "Real Time" and this had absolutely no
impact on the server process. The CPU load didn't change.

This is the only program intended to run on this computer. We need it
hogging as many resources as it can. Database, development, everything's on
different systems. This machine has one job, run this Java program, and it's
not taking that job seriously.

We don't know if it's a Windows issue, or a Java issue, but I suspect it's a
combination of the two. All we know is it used to run fast as hell, and now
crawls.

This post is going to Java groups and Windows groups on the assumption that
something can be done on each end to improve things (perhaps there's a
command line parameter to tell java to run at a higher priority, perhaps
there's a windows issue not giving proper time to the Java App).

Any thoughts?

Thank you for your time.
Nick Soutter

 

[Mike Brannigan [MSFT]]

If all of the underlying system and setup of Windows is identical to the
first system (with the exception of the hardware changes to faster
components). Then unless it is a hardware issue (some setting in the BIOS
to incorrectly set up the mother board or CPUs); then your statement about
"revamped some of the major systems (the code and the computer itself)," -
would lead you to possibly retest the system using the original code base
for the application that was running on the old server. This will help
eliminate your code changes as being the source of the error. (alternative
path is to put your new code version on the old server and retest)

--
Regards,

Mike
--
Mike Brannigan [Microsoft]

This posting is provided "AS IS" with no warranties, and confers no
rights

Please note I cannot respond to e-mailed questions, please use these
newsgroups

 

[=?iso-8859-1?q?Roger_Lindsj=F6?=]

Hello!

This is my second post. Ppl really helped me with the first. I hope
there are answers for this one as well

When we first set this up, the Game Server Program took exactly 50% of
the CPU cycles when idle, and up to 80% or 90% when working (processing
a turn, for example) and could process a turn of 10,000 ships in 15
seconds.

We revamped some of the major systems (the code and the computer
itself), did a fresh install of Windows 2003 on a different system
(still dual Xeon, but faster MB and CPU), migrated the Game Server
Program there, changed the threadpool priorities in the code, etc.

We've now noticed that on idle (IE not processing turns), the Game
Server Program takes 0% of the CPU load, and when processing a turn goes
up to 3-7%. It never exceeds 7%, and takes FOREVER now when processing a
turn. It is the only program running on the machine, and services are at
a minimum. Even with 99% of the CPU resources free (IE on System Idle),
the game will never take more than about 7% of the processor time.

You say you revamped the code. What does that mean? Was the code changed?
If the Java program can not hog 90%-100% of the CPU then I'd
look at code. I don't use windows, but I doubt it will prioritize the
idle process higher than your program.

Things to look at:
1 Synchronizations. This could kill an application on a multi cpu machine.
2 IO. Are you reading / writing to disk or network? If you are and this is
done synchnonously then you application will probably wait for the disk a
lot of the time.
3. Object creation. Are you creating lots of objects that has to be GCd?
Testing my applications shows that creating small objects that should be
GCd makes minor impact, but as soon as the size of the objects grows (a
few KB or so) the time spent on GC increases a lot. Could be related to
now nurseries are managed and if objects over a certain size are not put
into a nursery. You can probably configure your JVM to tell you how often
it has to do a GC and how much memory was freed.

Roger Lindsjö

 

[Roman]

What is the JDK you are running this Java program on?
Have you considered running this application on a Linux system?

Roman.

 

[Chris Uppal]

[Over-enthusiastic cross-posting removed]

When we first set this up, the Game Server Program took exactly 50% of the
CPU cycles when idle, and up to 80% or 90% when working (processing a
turn, for example) and could process a turn of 10,000 ships in 15 seconds.

50% CPU time on *IDLE* ?? What on earth was it doing ? Was it intended to do
that ? If you don't already know the answers to these questions (and I *guess*
that you don't or you'd have included them in you detailed description) then
you *need* to find the answers...

We revamped some of the major systems (the code and the computer itself),
did a fresh install of Windows 2003 on a different system (still dual
Xeon, but faster MB and CPU), migrated the Game Server Program there,
changed the threadpool priorities in the code, etc.

We've now noticed that on idle (IE not processing turns), the Game Server
Program takes 0% of the CPU load, and when processing a turn goes up to
3-7%. It never exceeds 7%, and takes FOREVER now when processing a turn.

If it's only using 0% CPU on idle now then that's what I'd expect, but it's not
consistent with the first case. You should be able to find out the difference
without considering what it's doing during its "real" workload. That should
make the investigation easier. Also there is no way that changing, even, the
entire OS is going to make you app go from 50% to 0% CPU to do *nothing*, so
you can forget -- for now -- about possible OS links.

However, if the thing is running slowly, but is not consuming CPU then it
*must* be waiting for something. I can think of two possibilities:
network waits
internal blocking

An example of network waits (this is a real example from my own past). An
application was able to serve network requests at X/sec when the test workload
was delivered from the same box. As soon as I generalised the test to run the
client on a colleague's box the workrate plummeted to around 1/20th, when I'd
expected it to go *up* (since the server box wasn't running the client too).
That turned out to be because of my logging code which logged the hostname of
the client rather than the IP address -- reverse look up "localhost" was local
and fast, but for some reason reverse lookup of my colleague's box always
involved a network access (DHCP or something) which slowed it down massively.
Of course, almost any kind of network access could show the same kind of
problem, database access for instance.

Internal blocking could be almost anything. Perhaps you have
over-"synchronised" something so that all processing is now happening on just
one thread. Perhaps you are waiting for some conditions somewhere that never
becomes true (or where the code for signalling that it has become true is
broken), and so your app is halted until something times out, or otherwise
wakes up the sleeping threads.

I would advocate sitting down and *thinking* about what's going on -- messing
around trying to fix the symptoms (e.g. playing with task priorities) is just a
waste of time. (I don't blame you though, it's hard to think clearly in what
sounds like a panic situation). However, *while* you are thinking, it may be
worthwhile running some profiling on the application, that should be able to
find operations that are taking a long time without doing any real work.

HTH

-- chris

 

[thoff]

Sounds like deadlock. Turn on logging. If you don't have logging
then add it so you can solve these kinds of dynamic problems.
Deadlock can be in your communication protocols or threading.

It also sounds like you need spend some time with a profiler.

 

[Y2KYZFR1]

We revamped some of the major systems (the code and the computer itself),

you have introduced a major race condition somewhere, do some research
on profilers and look at where the hotspot is.

"We should forget about small efficiencies, say about 97% of the time:
premature optimization is the root of all evil."

- Knuth

 

[BlackHawke]

I know that there is CPU Throttling in Windows 2003... Is that only for IIS?

We found a way to compile this into native code... It's now revved up to 25%
of the CPU cycles during intense work... I notice that in the "performance"
section of the task manager it lists 4 CPU's (we only have 2)... It's
letting us max out one of the "4" to 100%.... Any way to let it max them
all?

Again, I increased the priority, but no luck....

Nick

 

[BlackHawke]

We have, but honestly no one here is an experienced Linux Admin. We can't
afford an admin (we're WAY over budget), but we all know windows systems
pretty well, so.....

I'd do it if I could...

Nick

 

[BlackHawke]

You know, this started RIGHT AFTER a revamp of all of the communications
protocols (and some threading issues too)....

Can you explain deadlock to me a bit more?

Thank you.

Nick

 

[BlackHawke]

Nevermind, looked it up!

I think maybe in an attempt to avoid deadlock the programmers added some
random element to the timing of threads... Perhaps that is the issue....

Nick

 

[Nigel Wade]

I know that there is CPU Throttling in Windows 2003... Is that only for IIS?

We found a way to compile this into native code... It's now revved up to 25%
of the CPU cycles during intense work... I notice that in the "performance"
section of the task manager it lists 4 CPU's (we only have 2)... It's
letting us max out one of the "4" to 100%.... Any way to let it max them
all?

Again, I increased the priority, but no luck....

Nick

That implies you've got dual Xeon processors with Hyperthreading enabled.
That might be confusing Windows into thinking you've got 4 processors. How
many processors is the verion of w3k on that system licensed for? It could
be ignoring processors for which you don't have a license. Of course, this
is pure speculation on my part...

Try disabling Hyperthreading in the BIOS so Windows only sees the 2
processors you actually have, and see what happens.

 

[Chris Smith]

[Followup set]

Nevermind, looked it up!

I think maybe in an attempt to avoid deadlock the programmers added some
random element to the timing of threads... Perhaps that is the issue....

I seriously hope not. If so, you need to fire the programmers that did
so. Developers who don't understand the difference between a correct
application and a lucky application are *not* a good thing to have
around.

In all, I'd consider this hypothesis entirely unlikely.

As was said the last time you brought up issues about your game here,
why don't you ask a programmer to come and discuss the issue? It's a
waste of everyone's time (including your own) to post here when you
can't provide most of the information needed to solve the problem.

--
www.designacourse.com
The Easiest Way to Train Anyone... Anywhere.

Chris Smith - Lead Software Developer/Technical Trainer
MindIQ Corporation

 

[BlackHawke]

We do have dual Xeon processors...

Out of curiosity, is there a problem with hyperthreading? Will we run faster
if we disable it?

Nick

 

[Nigel Wade]

We do have dual Xeon processors...

Out of curiosity, is there a problem with hyperthreading?

Not that I know of.

Will we run faster
if we disable it?

The easiest way to answer that is to try it and see.

I thought that Windows desktop was licensed for a single processor, base
server for 2 and advanced server for more. I might be wrong though as I use
Linux not Windows.

What Windows does if you run desktop on dual processors, or basic server on
dual hyperthreading processors (which appear as 4) someone else will have to
answer. But if it totally disables your 2nd processor and allows the single,
hyperthread processor then it will most definitely run slower with
hyperthreading enabled.

 

[Grant Wagner]

Just to clarify, CPUs with hyperthreading don't "confuse" Windows into thinking
there are more CPUs, its the way it's supposed to work. You may be interested in
reading the document at:

<url:
http://www.microsoft.com/windows2000/server/evaluation/performance/reports/hyperthread.asp
/>

No, there is no problem with hyperthreading, but read the article at the URL I
provided because it discusses issues about CPU licensing and how incorrect
enumeration of hyperthreaded processors could cause Windows to be using two
logical processors on the same physical CPU instead of two logical processors.

"It is critical that the BIOS count logical processors in the manner described;
otherwise, Windows 2000 or its applications may use logical processors when they
should be using physical processors instead. For example, consider an
application that is licensed to use two processors on the system diagrammed in
Figure 1. Such an application will achieve better performance using two separate
physical processors (such as 1 and 2) than it would using two logical processors
on the same physical processor (such as 1 and 5)."

Also note that if you have a 2 CPU license for your application, the 2nd logical
processor on each CPU simply won't be used, so you might as well disable
hyperthreading to avoid any overhead it causes.

I apologize that it's specific to Windows 2000, I was in a hurry and didn't try
to locate a similar document for Windows 2003, but I'd assume most of the
information in that document applies to Windows 2003 as well.