chron overhead

[Roedy Green]

I have always avoided writing Java programs than run all the time and
sporadically wake up to do something, e.g. test if a website is up,
check for mail, do a backup ...

I figured the overhead would be unacceptable. Though I never tested
it. Has anyone ever measured just how much of a drain such a sleeping
program puts on resources?
--
Roedy Green Canadian Mind Products
http://mindprod.com
Your old road is
Rapidly agin'.
Please get out of the new one
If you can't lend your hand
For the times they are a-changin'.

 

[Owen Jacobson]

I have always avoided writing Java programs than run all the time and
sporadically wake up to do something, e.g. test if a website is up,
check for mail, do a backup ...

I figured the overhead would be unacceptable. Though I never tested
it.  Has anyone ever measured just how much of a drain such a sleeping
program puts on resources?

I run Tomcat to host Hudson and JIRA for my own development on a linux
machine stashed in a corner. When no builds are running and I'm not
actively fiddling with JIRA issues, the machine's load averages are
effectively 0.00 0.00 0.00. I was a little leery of having a Java
server running constantly for pretty much the same reasons, but it
actually works rather well.

For reference:
JAVA_OPTS="-Djava.awt.headless=true -Xmx512M \
-Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true"
JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.10

Care of ps(1), the rss and vsize:
rss = 111800 (111 MB), vsize = 695512 (696 MB)
which makes it by far the largest process (for both resident and
virtual size) on the system, but with several GB of RAM I can afford
100MB.

-o

 

[Tom Anderson]

I'd also avoid it, but not out of concern for efficiency - for robustness.
With a sleeping-looping program, if it crashes or gets wedged or
something, it stops doing its job. With a program that does the job once
and exits, but which is scheduled by cron or similar, the worst a crash or
hang can do is clobber one run. It would take a failure of cron itself to
stop future runs, and cron is something that i'm willing to bet the farm
on.

I guess you could use some external utility to monitor your looping
program and restart it if it crashes. I think you can do that with init on
unix (?), or launchd on a Mac, or you can just write a shell script which
invokes the program in a loop. Doesn't really help with hangs, though. You
could use some sort of watchdog mechanism for that.

I run Tomcat to host Hudson and JIRA for my own development on a linux
machine stashed in a corner. When no builds are running and I'm not
actively fiddling with JIRA issues, the machine's load averages are
effectively 0.00 0.00 0.00. I was a little leery of having a Java
server running constantly for pretty much the same reasons, but it
actually works rather well.

For reference:
JAVA_OPTS="-Djava.awt.headless=true -Xmx512M \
-Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true"
JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.10

Care of ps(1), the rss and vsize:
rss = 111800 (111 MB), vsize = 695512 (696 MB)
which makes it by far the largest process (for both resident and
virtual size) on the system, but with several GB of RAM I can afford
100MB.

I'm slightly surprised it uses 100 MB of memory even when not running. Is
that because there's nothing else running on that machine, or at any rate,
not a lot of memory pressure? Would that get paged out if there was, or is
it some kind of pinned/wired memory?

tom

 

[Owen Jacobson]

I'd also avoid it, but not out of concern for efficiency - for robustness..
With a sleeping-looping program, if it crashes or gets wedged or
something, it stops doing its job. With a program that does the job once
and exits, but which is scheduled by cron or similar, the worst a crash or
hang can do is clobber one run. It would take a failure of cron itself to
stop future runs, and cron is something that i'm willing to bet the farm
on.

I guess you could use some external utility to monitor your looping
program and restart it if it crashes. I think you can do that with init on
unix (?), or launchd on a Mac, or you can just write a shell script which
invokes the program in a loop. Doesn't really help with hangs, though. You
could use some sort of watchdog mechanism for that.







I'm slightly surprised it uses 100 MB of memory even when not running. Is
that because there's nothing else running on that machine, or at any rate,
not a lot of memory pressure? Would that get paged out if there was, or is
it some kind of pinned/wired memory?

tom

There's a lot of crap running on that machine (apache2, tomcat, mysql
for some webapps, postgresql for my own apps, slapd, sshd, and a few
long-standing screen sessions) for one user, but it's not busy by any
stretch. The load average rarely breaks 1.0.

Computing range(1, 0x0FFFFFFF) in python (which allocates a really
huge array) forces Tomcat's resident size down to 5916 (6 MB). It also
beats the crap out of the CPU.

-o

 

[Tom Anderson]

There's a lot of crap running on that machine (apache2, tomcat, mysql
for some webapps, postgresql for my own apps, slapd, sshd, and a few
long-standing screen sessions) for one user, but it's not busy by any
stretch. The load average rarely breaks 1.0.

Computing range(1, 0x0FFFFFFF) in python (which allocates a really huge
array) forces Tomcat's resident size down to 5916 (6 MB).

Okay, interesting and reassuring. Thanks for the info. I wonder what's
preserving the last 6 MB.

It also beats the crap out of the CPU.

I have in the past used exponentiation of bignums in python to dry my
socks!

tom

 

[Martin Gregorie]

I'd also avoid it, but not out of concern for efficiency - for

robustness.

With a sleeping-looping program, if it crashes or gets
wedged or something, it stops doing its job. With a program that does
the job once and exits, but which is scheduled by cron or similar, the
worst a crash or hang can do is clobber one run. It would take a failure
of cron itself to stop future runs, and cron is something that i'm
willing to bet the farm on.

Likewise, though as my Java cron job runs once a day for typically less
than 30 seconds, its arguable whether it deserves to occupy memory space
for the remaining 23 hours 59 minutes and 30 seconds. Running it more
frequently than once a day would gain me very little.

I think there are other reasons for using cron that are more compelling:
the program has been a lot easier to test than a permanently resident
version would be and I didn't see the need to build functionality into it
that's already provided by cron.

I guess you could use some external utility to monitor your looping
program and restart it if it crashes. I think you can do that with init
on unix (?)

True. If it was started as a system daemon at boot time it would be
automatically restarted if it crashed, but this mechanism can't spot it
looping or getting wedged. If your *nix has the SVR4 style daemon
management tools built round /etc/rc.d/* you can use the 'service'
command to stop, start, or restart it and report its status. Dead/alive
status reporting is built in but finding out any more about your server
would require a status querying command line client and a matching
transaction in the server.

 

[Mike Schilling]

I have in the past used exponentiation of bignums in python to dry
my
socks!

Heats up the processor, causing the fan to run continuously?

 

[Arne Vajhøj]

I'm slightly surprised it uses 100 MB of memory even when not running.
Is that because there's nothing else running on that machine, or at any
rate, not a lot of memory pressure? Would that get paged out if there
was, or is it some kind of pinned/wired memory?

Java app servers are notorious memory pigs.

But considering that 100 MB of standard PC RAM is about 1 USD, then ...

Arne

 

[Arne Vajhøj]

I'd also avoid it, but not out of concern for efficiency - for
robustness. With a sleeping-looping program, if it crashes or gets
wedged or something, it stops doing its job. With a program that does
the job once and exits, but which is scheduled by cron or similar, the
worst a crash or hang can do is clobber one run. It would take a failure
of cron itself to stop future runs, and cron is something that i'm
willing to bet the farm on.

I guess you could use some external utility to monitor your looping
program and restart it if it crashes. I think you can do that with init
on unix (?), or launchd on a Mac, or you can just write a shell script
which invokes the program in a loop. Doesn't really help with hangs,
though. You could use some sort of watchdog mechanism for that.

If using a Java app server he would also get robustness through a
separation of the container code and the app code.

Arne

 

[Arne Vajhøj]

I have always avoided writing Java programs than run all the time and
sporadically wake up to do something, e.g. test if a website is up,
check for mail, do a backup ...

I figured the overhead would be unacceptable. Though I never tested
it. Has anyone ever measured just how much of a drain such a sleeping
program puts on resources?

A handwritten Java daemon would use very little resources, but
would be a bit vulnerable.

I would suggest following Owens model and run it in Tomcat.

It would user very little CPU, use some memory but memory
is cheap and it could be done rather robust.

Arne

 

[Tom Anderson]

Heats up the processor, causing the fan to run continuously?

Exactly. Although it was mainly the heat i wanted - my laptop reaches [1]
a good 70-odd celsius when the pedal is to the metal. Start exponentiation
loop, put socks on keyboard, go and have shower, return to toasty warm
socks.

tom

[1] Or possibly 'reached' - it's currently dead.

 

[Tom Anderson]

Java app servers are notorious memory pigs.

True. But i was unsure why that data would be kept in main memory, rather
than paged out to disk, when the server was inactive - and it turned out
it wasn't.

But considering that 100 MB of standard PC RAM is about 1 USD, then ...

Then reducing it to 6 MB means that when you need to run 1000 server
instances at once, you've saved about a grand!

tom

 

[Arne Vajhøj]

True. But i was unsure why that data would be kept in main memory,
rather than paged out to disk, when the server was inactive - and it
turned out it wasn't.


Then reducing it to 6 MB means that when you need to run 1000 server
instances at once, you've saved about a grand!

But in most cases you will run multiple web apps in the same
container and that add very little to memory usage.

Arne

 

[Owen Jacobson]

True. But i was unsure why that data would be kept in main memory, rather
than paged out to disk, when the server was inactive - and it turned out
it wasn't.


Then reducing it to 6 MB means that when you need to run 1000 server
instances at once, you've saved about a grand!

It's probably worth mentioning that it promptly went back to around
20MB after I killed python and stayed there - there's obviously some
kind of background task inside Tomcat (off the top of my head,
probably things related to maintaining the thread pool, checking for
sessions to purge, and Hudson's SCM polling) that causes stuff to get
swapped back in.

In practical terms I've never even noticed it running.

-o

 

[Arne Vajhøj]

It's probably worth mentioning that it promptly went back to around
20MB after I killed python and stayed there - there's obviously some
kind of background task inside Tomcat (off the top of my head,
probably things related to maintaining the thread pool, checking for
sessions to purge, and Hudson's SCM polling) that causes stuff to get
swapped back in.

Jasper checkInterval can also cause activity every 5 minutes.

Arne