chron overhead

Discussion in 'Java' started by Roedy Green, Nov 19, 2008.

  1. Roedy Green

    Roedy Green Guest

    I have always avoided writing Java programs than run all the time and
    sporadically wake up to do something, e.g. test if a website is up,
    check for mail, do a backup ...

    I figured the overhead would be unacceptable. Though I never tested
    it. Has anyone ever measured just how much of a drain such a sleeping
    program puts on resources?
    --
    Roedy Green Canadian Mind Products
    http://mindprod.com
    Your old road is
    Rapidly agin'.
    Please get out of the new one
    If you can't lend your hand
    For the times they are a-changin'.
     
    Roedy Green, Nov 19, 2008
    #1
    1. Advertising

  2. On Nov 19, 11:48 am, Roedy Green <>
    wrote:
    > I have always avoided writing Java programs than run all the time and
    > sporadically wake up to do something, e.g. test if a website is up,
    > check for mail, do a backup ...
    >
    > I figured the overhead would be unacceptable. Though I never tested
    > it.  Has anyone ever measured just how much of a drain such a sleeping
    > program puts on resources?


    I run Tomcat to host Hudson and JIRA for my own development on a linux
    machine stashed in a corner. When no builds are running and I'm not
    actively fiddling with JIRA issues, the machine's load averages are
    effectively 0.00 0.00 0.00. I was a little leery of having a Java
    server running constantly for pretty much the same reasons, but it
    actually works rather well.

    For reference:
    JAVA_OPTS="-Djava.awt.headless=true -Xmx512M \
    -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true"
    JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.10

    Care of ps(1), the rss and vsize:
    rss = 111800 (111 MB), vsize = 695512 (696 MB)
    which makes it by far the largest process (for both resident and
    virtual size) on the system, but with several GB of RAM I can afford
    100MB.

    -o
     
    Owen Jacobson, Nov 19, 2008
    #2
    1. Advertising

  3. Roedy Green

    Tom Anderson Guest

    On Wed, 19 Nov 2008, Owen Jacobson wrote:

    > On Nov 19, 11:48 am, Roedy Green <>
    > wrote:
    >
    >> I have always avoided writing Java programs than run all the time and
    >> sporadically wake up to do something, e.g. test if a website is up,
    >> check for mail, do a backup ...
    >>
    >> I figured the overhead would be unacceptable.


    I'd also avoid it, but not out of concern for efficiency - for robustness.
    With a sleeping-looping program, if it crashes or gets wedged or
    something, it stops doing its job. With a program that does the job once
    and exits, but which is scheduled by cron or similar, the worst a crash or
    hang can do is clobber one run. It would take a failure of cron itself to
    stop future runs, and cron is something that i'm willing to bet the farm
    on.

    I guess you could use some external utility to monitor your looping
    program and restart it if it crashes. I think you can do that with init on
    unix (?), or launchd on a Mac, or you can just write a shell script which
    invokes the program in a loop. Doesn't really help with hangs, though. You
    could use some sort of watchdog mechanism for that.

    >> Though I never tested it.  Has anyone ever measured just how much of a
    >> drain such a sleeping program puts on resources?

    >
    > I run Tomcat to host Hudson and JIRA for my own development on a linux
    > machine stashed in a corner. When no builds are running and I'm not
    > actively fiddling with JIRA issues, the machine's load averages are
    > effectively 0.00 0.00 0.00. I was a little leery of having a Java
    > server running constantly for pretty much the same reasons, but it
    > actually works rather well.
    >
    > For reference:
    > JAVA_OPTS="-Djava.awt.headless=true -Xmx512M \
    > -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true"
    > JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.10
    >
    > Care of ps(1), the rss and vsize:
    > rss = 111800 (111 MB), vsize = 695512 (696 MB)
    > which makes it by far the largest process (for both resident and
    > virtual size) on the system, but with several GB of RAM I can afford
    > 100MB.


    I'm slightly surprised it uses 100 MB of memory even when not running. Is
    that because there's nothing else running on that machine, or at any rate,
    not a lot of memory pressure? Would that get paged out if there was, or is
    it some kind of pinned/wired memory?

    tom

    --
    uk.local groups TO BE RENAMED uk.lunatic.fringe groups
     
    Tom Anderson, Nov 19, 2008
    #3
  4. On Nov 19, 1:12 pm, Tom Anderson <> wrote:
    > On Wed, 19 Nov 2008, Owen Jacobson wrote:
    > > On Nov 19, 11:48 am, Roedy Green <>
    > > wrote:

    >
    > >> I have always avoided writing Java programs than run all the time and
    > >> sporadically wake up to do something, e.g. test if a website is up,
    > >> check for mail, do a backup ...

    >
    > >> I figured the overhead would be unacceptable.

    >
    > I'd also avoid it, but not out of concern for efficiency - for robustness..
    > With a sleeping-looping program, if it crashes or gets wedged or
    > something, it stops doing its job. With a program that does the job once
    > and exits, but which is scheduled by cron or similar, the worst a crash or
    > hang can do is clobber one run. It would take a failure of cron itself to
    > stop future runs, and cron is something that i'm willing to bet the farm
    > on.
    >
    > I guess you could use some external utility to monitor your looping
    > program and restart it if it crashes. I think you can do that with init on
    > unix (?), or launchd on a Mac, or you can just write a shell script which
    > invokes the program in a loop. Doesn't really help with hangs, though. You
    > could use some sort of watchdog mechanism for that.
    >
    >
    >
    > >> Though I never tested it.  Has anyone ever measured just how much of a
    > >> drain such a sleeping program puts on resources?

    >
    > > I run Tomcat to host Hudson and JIRA for my own development on a linux
    > > machine stashed in a corner. When no builds are running and I'm not
    > > actively fiddling with JIRA issues, the machine's load averages are
    > > effectively 0.00 0.00 0.00. I was a little leery of having a Java
    > > server running constantly for pretty much the same reasons, but it
    > > actually works rather well.

    >
    > > For reference:
    > > JAVA_OPTS="-Djava.awt.headless=true -Xmx512M \
    > >  -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true"
    > > JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.10

    >
    > > Care of ps(1), the rss and vsize:
    > > rss = 111800 (111 MB), vsize = 695512 (696 MB)
    > > which makes it by far the largest process (for both resident and
    > > virtual size) on the system, but with several GB of RAM I can afford
    > > 100MB.

    >
    > I'm slightly surprised it uses 100 MB of memory even when not running. Is
    > that because there's nothing else running on that machine, or at any rate,
    > not a lot of memory pressure? Would that get paged out if there was, or is
    > it some kind of pinned/wired memory?
    >
    > tom


    There's a lot of crap running on that machine (apache2, tomcat, mysql
    for some webapps, postgresql for my own apps, slapd, sshd, and a few
    long-standing screen sessions) for one user, but it's not busy by any
    stretch. The load average rarely breaks 1.0.

    Computing range(1, 0x0FFFFFFF) in python (which allocates a really
    huge array) forces Tomcat's resident size down to 5916 (6 MB). It also
    beats the crap out of the CPU. :)

    -o
     
    Owen Jacobson, Nov 19, 2008
    #4
  5. Roedy Green

    Tom Anderson Guest

    On Wed, 19 Nov 2008, Owen Jacobson wrote:

    > On Nov 19, 1:12 pm, Tom Anderson <> wrote:
    >> On Wed, 19 Nov 2008, Owen Jacobson wrote:
    >>> On Nov 19, 11:48 am, Roedy Green <>
    >>> wrote:

    >>
    >>>> I have always avoided writing Java programs than run all the time and
    >>>> sporadically wake up to do something, e.g. test if a website is up,
    >>>> check for mail, do a backup ...
    >>>>
    >>>> I figured the overhead would be unacceptable.
    >>>>
    >>>> Though I never tested it.  Has anyone ever measured just how much of a
    >>>> drain such a sleeping program puts on resources?
    >>>
    >>> I run Tomcat to host Hudson and JIRA for my own development on a linux
    >>> machine stashed in a corner. When no builds are running and I'm not
    >>> actively fiddling with JIRA issues, the machine's load averages are
    >>> effectively 0.00 0.00 0.00. I was a little leery of having a Java
    >>> server running constantly for pretty much the same reasons, but it
    >>> actually works rather well.
    >>>
    >>> Care of ps(1), the rss and vsize: rss = 111800 (111 MB), vsize =
    >>> 695512 (696 MB) which makes it by far the largest process (for both
    >>> resident and virtual size) on the system, but with several GB of RAM I
    >>> can afford 100MB.

    >>
    >> I'm slightly surprised it uses 100 MB of memory even when not running.
    >> Is that because there's nothing else running on that machine, or at any
    >> rate, not a lot of memory pressure? Would that get paged out if there
    >> was, or is it some kind of pinned/wired memory?

    >
    > There's a lot of crap running on that machine (apache2, tomcat, mysql
    > for some webapps, postgresql for my own apps, slapd, sshd, and a few
    > long-standing screen sessions) for one user, but it's not busy by any
    > stretch. The load average rarely breaks 1.0.
    >
    > Computing range(1, 0x0FFFFFFF) in python (which allocates a really huge
    > array) forces Tomcat's resident size down to 5916 (6 MB).


    Okay, interesting and reassuring. Thanks for the info. I wonder what's
    preserving the last 6 MB.

    > It also beats the crap out of the CPU. :)


    I have in the past used exponentiation of bignums in python to dry my
    socks!

    tom

    --
    uk.local groups TO BE RENAMED uk.lunatic.fringe groups
     
    Tom Anderson, Nov 19, 2008
    #5
  6. > I'd also avoid it, but not out of concern for efficiency - for
    > robustness.
    >
    > With a sleeping-looping program, if it crashes or gets
    > wedged or something, it stops doing its job. With a program that does
    > the job once and exits, but which is scheduled by cron or similar, the
    > worst a crash or hang can do is clobber one run. It would take a failure
    > of cron itself to stop future runs, and cron is something that i'm
    > willing to bet the farm on.
    >

    Likewise, though as my Java cron job runs once a day for typically less
    than 30 seconds, its arguable whether it deserves to occupy memory space
    for the remaining 23 hours 59 minutes and 30 seconds. Running it more
    frequently than once a day would gain me very little.

    I think there are other reasons for using cron that are more compelling:
    the program has been a lot easier to test than a permanently resident
    version would be and I didn't see the need to build functionality into it
    that's already provided by cron.

    > I guess you could use some external utility to monitor your looping
    > program and restart it if it crashes. I think you can do that with init
    > on unix (?)
    >

    True. If it was started as a system daemon at boot time it would be
    automatically restarted if it crashed, but this mechanism can't spot it
    looping or getting wedged. If your *nix has the SVR4 style daemon
    management tools built round /etc/rc.d/* you can use the 'service'
    command to stop, start, or restart it and report its status. Dead/alive
    status reporting is built in but finding out any more about your server
    would require a status querying command line client and a matching
    transaction in the server.


    --
    martin@ | Martin Gregorie
    gregorie. | Essex, UK
    org |
     
    Martin Gregorie, Nov 19, 2008
    #6
  7. Tom Anderson wrote:
    >
    > I have in the past used exponentiation of bignums in python to dry
    > my
    > socks!


    Heats up the processor, causing the fan to run continuously?
     
    Mike Schilling, Nov 19, 2008
    #7
  8. Roedy Green

    Arne Vajhøj Guest

    Tom Anderson wrote:
    > On Wed, 19 Nov 2008, Owen Jacobson wrote:
    >> Care of ps(1), the rss and vsize:
    >> rss = 111800 (111 MB), vsize = 695512 (696 MB)
    >> which makes it by far the largest process (for both resident and
    >> virtual size) on the system, but with several GB of RAM I can afford
    >> 100MB.

    >
    > I'm slightly surprised it uses 100 MB of memory even when not running.
    > Is that because there's nothing else running on that machine, or at any
    > rate, not a lot of memory pressure? Would that get paged out if there
    > was, or is it some kind of pinned/wired memory?


    Java app servers are notorious memory pigs.

    But considering that 100 MB of standard PC RAM is about 1 USD, then ...

    Arne
     
    Arne Vajhøj, Nov 19, 2008
    #8
  9. Roedy Green

    Arne Vajhøj Guest

    Tom Anderson wrote:
    > On Wed, 19 Nov 2008, Owen Jacobson wrote:
    >> On Nov 19, 11:48 am, Roedy Green <>
    >> wrote:
    >>> I have always avoided writing Java programs than run all the time and
    >>> sporadically wake up to do something, e.g. test if a website is up,
    >>> check for mail, do a backup ...
    >>>
    >>> I figured the overhead would be unacceptable.

    >
    > I'd also avoid it, but not out of concern for efficiency - for
    > robustness. With a sleeping-looping program, if it crashes or gets
    > wedged or something, it stops doing its job. With a program that does
    > the job once and exits, but which is scheduled by cron or similar, the
    > worst a crash or hang can do is clobber one run. It would take a failure
    > of cron itself to stop future runs, and cron is something that i'm
    > willing to bet the farm on.
    >
    > I guess you could use some external utility to monitor your looping
    > program and restart it if it crashes. I think you can do that with init
    > on unix (?), or launchd on a Mac, or you can just write a shell script
    > which invokes the program in a loop. Doesn't really help with hangs,
    > though. You could use some sort of watchdog mechanism for that.


    If using a Java app server he would also get robustness through a
    separation of the container code and the app code.

    Arne
     
    Arne Vajhøj, Nov 19, 2008
    #9
  10. Roedy Green

    Arne Vajhøj Guest

    Roedy Green wrote:
    > I have always avoided writing Java programs than run all the time and
    > sporadically wake up to do something, e.g. test if a website is up,
    > check for mail, do a backup ...
    >
    > I figured the overhead would be unacceptable. Though I never tested
    > it. Has anyone ever measured just how much of a drain such a sleeping
    > program puts on resources?


    A handwritten Java daemon would use very little resources, but
    would be a bit vulnerable.

    I would suggest following Owens model and run it in Tomcat.

    It would user very little CPU, use some memory but memory
    is cheap and it could be done rather robust.

    Arne
     
    Arne Vajhøj, Nov 19, 2008
    #10
  11. Roedy Green

    Tom Anderson Guest

    On Wed, 19 Nov 2008, Mike Schilling wrote:

    > Tom Anderson wrote:
    >
    >> I have in the past used exponentiation of bignums in python to dry my
    >> socks!

    >
    > Heats up the processor, causing the fan to run continuously?


    Exactly. Although it was mainly the heat i wanted - my laptop reaches [1]
    a good 70-odd celsius when the pedal is to the metal. Start exponentiation
    loop, put socks on keyboard, go and have shower, return to toasty warm
    socks.

    tom

    [1] Or possibly 'reached' - it's currently dead.

    --
    I am become Life, destroyer of worlds
     
    Tom Anderson, Nov 20, 2008
    #11
  12. Roedy Green

    Tom Anderson Guest

    On Wed, 19 Nov 2008, Arne Vajhøj wrote:

    > Tom Anderson wrote:
    >> On Wed, 19 Nov 2008, Owen Jacobson wrote:
    >>> Care of ps(1), the rss and vsize:
    >>> rss = 111800 (111 MB), vsize = 695512 (696 MB)
    >>> which makes it by far the largest process (for both resident and
    >>> virtual size) on the system, but with several GB of RAM I can afford
    >>> 100MB.

    >>
    >> I'm slightly surprised it uses 100 MB of memory even when not running. Is
    >> that because there's nothing else running on that machine, or at any rate,
    >> not a lot of memory pressure? Would that get paged out if there was, or is
    >> it some kind of pinned/wired memory?

    >
    > Java app servers are notorious memory pigs.


    True. But i was unsure why that data would be kept in main memory, rather
    than paged out to disk, when the server was inactive - and it turned out
    it wasn't.

    > But considering that 100 MB of standard PC RAM is about 1 USD, then ...


    Then reducing it to 6 MB means that when you need to run 1000 server
    instances at once, you've saved about a grand!

    tom

    --
    I am become Life, destroyer of worlds
     
    Tom Anderson, Nov 20, 2008
    #12
  13. Roedy Green

    Arne Vajhøj Guest

    Tom Anderson wrote:
    > On Wed, 19 Nov 2008, Arne Vajhøj wrote:
    >> Tom Anderson wrote:
    >>> On Wed, 19 Nov 2008, Owen Jacobson wrote:
    >>>> Care of ps(1), the rss and vsize:
    >>>> rss = 111800 (111 MB), vsize = 695512 (696 MB)
    >>>> which makes it by far the largest process (for both resident and
    >>>> virtual size) on the system, but with several GB of RAM I can afford
    >>>> 100MB.
    >>>
    >>> I'm slightly surprised it uses 100 MB of memory even when not
    >>> running. Is that because there's nothing else running on that
    >>> machine, or at any rate, not a lot of memory pressure? Would that get
    >>> paged out if there was, or is it some kind of pinned/wired memory?

    >>
    >> Java app servers are notorious memory pigs.

    >
    > True. But i was unsure why that data would be kept in main memory,
    > rather than paged out to disk, when the server was inactive - and it
    > turned out it wasn't.
    >
    >> But considering that 100 MB of standard PC RAM is about 1 USD, then ...

    >
    > Then reducing it to 6 MB means that when you need to run 1000 server
    > instances at once, you've saved about a grand!


    But in most cases you will run multiple web apps in the same
    container and that add very little to memory usage.

    Arne
     
    Arne Vajhøj, Nov 20, 2008
    #13
  14. On Nov 19, 7:27 pm, Tom Anderson <> wrote:
    > On Wed, 19 Nov 2008, Arne Vajhøj wrote:
    > > Tom Anderson wrote:
    > >> On Wed, 19 Nov 2008, Owen Jacobson wrote:
    > >>> Care of ps(1), the rss and vsize:
    > >>> rss = 111800 (111 MB), vsize = 695512 (696 MB)
    > >>> which makes it by far the largest process (for both resident and
    > >>> virtual size) on the system, but with several GB of RAM I can afford
    > >>> 100MB.

    >
    > >> I'm slightly surprised it uses 100 MB of memory even when not running. Is
    > >> that because there's nothing else running on that machine, or at any rate,
    > >> not a lot of memory pressure? Would that get paged out if there was, or is
    > >> it some kind of pinned/wired memory?

    >
    > > Java app servers are notorious memory pigs.

    >
    > True. But i was unsure why that data would be kept in main memory, rather
    > than paged out to disk, when the server was inactive - and it turned out
    > it wasn't.
    >
    > > But considering that 100 MB of standard PC RAM is about 1 USD, then ...

    >
    > Then reducing it to 6 MB means that when you need to run 1000 server
    > instances at once, you've saved about a grand!


    It's probably worth mentioning that it promptly went back to around
    20MB after I killed python and stayed there - there's obviously some
    kind of background task inside Tomcat (off the top of my head,
    probably things related to maintaining the thread pool, checking for
    sessions to purge, and Hudson's SCM polling) that causes stuff to get
    swapped back in.

    In practical terms I've never even noticed it running.

    -o
     
    Owen Jacobson, Nov 20, 2008
    #14
  15. Roedy Green

    Arne Vajhøj Guest

    Owen Jacobson wrote:
    > On Nov 19, 7:27 pm, Tom Anderson <> wrote:
    >> On Wed, 19 Nov 2008, Arne Vajhøj wrote:
    >>> Tom Anderson wrote:
    >>>> On Wed, 19 Nov 2008, Owen Jacobson wrote:
    >>>>> Care of ps(1), the rss and vsize:
    >>>>> rss = 111800 (111 MB), vsize = 695512 (696 MB)
    >>>>> which makes it by far the largest process (for both resident and
    >>>>> virtual size) on the system, but with several GB of RAM I can afford
    >>>>> 100MB.
    >>>> I'm slightly surprised it uses 100 MB of memory even when not running. Is
    >>>> that because there's nothing else running on that machine, or at any rate,
    >>>> not a lot of memory pressure? Would that get paged out if there was, or is
    >>>> it some kind of pinned/wired memory?
    >>> Java app servers are notorious memory pigs.

    >> True. But i was unsure why that data would be kept in main memory, rather
    >> than paged out to disk, when the server was inactive - and it turned out
    >> it wasn't.
    >>
    >>> But considering that 100 MB of standard PC RAM is about 1 USD, then ...

    >> Then reducing it to 6 MB means that when you need to run 1000 server
    >> instances at once, you've saved about a grand!

    >
    > It's probably worth mentioning that it promptly went back to around
    > 20MB after I killed python and stayed there - there's obviously some
    > kind of background task inside Tomcat (off the top of my head,
    > probably things related to maintaining the thread pool, checking for
    > sessions to purge, and Hudson's SCM polling) that causes stuff to get
    > swapped back in.


    Jasper checkInterval can also cause activity every 5 minutes.

    Arne
     
    Arne Vajhøj, Nov 20, 2008
    #15
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. John T. Goodman

    Overhead of 4-port over 2-port SRAM

    John T. Goodman, Jan 25, 2005, in forum: VHDL
    Replies:
    0
    Views:
    630
    John T. Goodman
    Jan 25, 2005
  2. Showjumper

    Does Option Strict On add overhead?

    Showjumper, Jul 31, 2003, in forum: ASP .Net
    Replies:
    2
    Views:
    364
    John Saunders
    Jul 31, 2003
  3. Brett Robichaud

    Trace overhead

    Brett Robichaud, May 5, 2004, in forum: ASP .Net
    Replies:
    4
    Views:
    513
    Brett Robichaud
    May 6, 2004
  4. Dave Williamson
    Replies:
    2
    Views:
    456
    Rocky Moore
    Aug 15, 2004
  5. tony
    Replies:
    1
    Views:
    346
    =?Utf-8?B?Q293Ym95IChHcmVnb3J5IEEuIEJlYW1lcikgLSBN
    Jan 19, 2005
Loading...

Share This Page