futex high system utilization

Discussion in 'C++' started by Jaydeep Chovatia, Jul 17, 2012.

  1. Hi,

    In my c++ (Linux) application I am seeing high CPU utilization (almost 90%)and it that sys-cpu:user-cpu ratio is 8:2. I then tried to run “strace” command and found that around 80% of time is being spent in “futex”system call, please see snippet of strace here:

    % time seconds usecs/call calls errors syscall
    ------ ----------- ----------- --------- --------- ----------------
    80.66 100.007743 1567 63806 22761 futex
    9.15 11.340274 96925 117 4 restart_syscall
    5.60 6.939598 4401 1577 poll

    Initially I though this is because of pthread_lock, unlock, condition timedwait, condition wait, etc. system calls I am using in the application, to prove that I have overridden pthread_lock, unlock, condition timedwait, condition wait, unlock, etc. using LD_PRELOAD functionality and concluded that not many threads are waiting here (mutex->__data.__nusers remains less than5 most of the time), so it doesn’t seem that these sys calls are part ofthe problem.

    Now I am running out of clue about how to find out code location/system call which is causing this futex high utilization, any help on this would be appreciated.

    Thank you,
    Jaydeep
    Jaydeep Chovatia, Jul 17, 2012
    #1
    1. Advertising

  2. Jaydeep Chovatia

    Jorgen Grahn Guest

    On Tue, 2012-07-17, Jaydeep Chovatia wrote:
    > Hi,
    >
    > In my c++ (Linux) application I am seeing high CPU utilization
    > (almost 90%) and it that sys-cpu:user-cpu ratio is 8:2. I then tried
    > to run ?strace? command and found that around 80% of time is being
    > spent in ?futex? system call,


    I'm not sure you can use strace to find that out -- strace makes
    syscalls much more expensive.

    > please see snippet of strace here:


    Please break your paragraphs into lines, so I don't have to do it for
    you.

    > % time seconds usecs/call calls errors syscall
    > ------ ----------- ----------- --------- --------- ----------------
    > 80.66 100.007743 1567 63806 22761 futex
    > 9.15 11.340274 96925 117 4 restart_syscall
    > 5.60 6.939598 4401 1577 poll
    >


    > Initially I though this is because of pthread_lock, unlock,
    > condition timedwait, condition wait, etc. system calls I am using in
    > the application, to prove that I have overridden pthread_lock, unlock,
    > condition timedwait, condition wait, unlock, etc. using LD_PRELOAD
    > functionality and concluded that not many threads are waiting here
    > (mutex->__data.__nusers remains less than 5 most of the time), so it
    > doesn?t seem that these sys calls are part of the problem.


    > Now I am running out of clue about how to find out code
    > location/system call which is causing this futex high utilization, any
    > help on this would be appreciated.


    This is a question which you should ask in some Linux newsgroup.

    I have only one suggestion to offer: try ltrace too. It traces
    library calls instead of system calls. Both strace and ltrace are
    very useful on Linux.

    /Jorgen

    --
    // Jorgen Grahn <grahn@ Oo o. . .
    \X/ snipabacken.se> O o .
    Jorgen Grahn, Jul 17, 2012
    #2
    1. Advertising

  3. In article <>,
    Jaydeep Chovatia <> wrote:
    >In my c++ (Linux) application I am seeing high CPU utilization (almost
    >90%) and it that sys-cpu:user-cpu ratio is 8:2. I then tried to run


    Is this a machine that has been running for a few weeks at least,
    and runs ntpd? If so the system might be suffering from a
    leap-second-adjustment bug .. as root run date -s "`date`" or
    reboot, then see if your problem is gone.

    Mike.
    Miquel van Smoorenburg, Jul 17, 2012
    #3
  4. Jaydeep Chovatia

    Jorgen Grahn Guest

    On Tue, 2012-07-17, Miquel van Smoorenburg wrote:
    > In article <>,
    > Jaydeep Chovatia <> wrote:
    >>In my c++ (Linux) application I am seeing high CPU utilization (almost
    >>90%) and it that sys-cpu:user-cpu ratio is 8:2. I then tried to run

    >
    > Is this a machine that has been running for a few weeks at least,
    > and runs ntpd? If so the system might be suffering from a
    > leap-second-adjustment bug .. as root run date -s "`date`" or
    > reboot, then see if your problem is gone.


    Do you have any URL for that? I was aware of the recent leap second,
    but haven't heard about that triggering ntpd bugs. Also I'm always
    interested in new ways to botch time calculations in software :)

    /Jorgen

    --
    // Jorgen Grahn <grahn@ Oo o. . .
    \X/ snipabacken.se> O o .
    Jorgen Grahn, Jul 17, 2012
    #4
  5. In article <>,
    Jorgen Grahn <> wrote:
    >On Tue, 2012-07-17, Miquel van Smoorenburg wrote:
    >> In article <>,
    >> Jaydeep Chovatia <> wrote:
    >>>In my c++ (Linux) application I am seeing high CPU utilization (almost
    >>>90%) and it that sys-cpu:user-cpu ratio is 8:2. I then tried to run

    >>
    >> Is this a machine that has been running for a few weeks at least,
    >> and runs ntpd? If so the system might be suffering from a
    >> leap-second-adjustment bug .. as root run date -s "`date`" or
    >> reboot, then see if your problem is gone.

    >
    >Do you have any URL for that? I was aware of the recent leap second,
    >but haven't heard about that triggering ntpd bugs. Also I'm always
    >interested in new ways to botch time calculations in software :)


    It isn't a bug in ntpd, it's just that if you run ntpd it would
    have told the kernel to handle the extra 2012-06-30 23:59:60 leapsecond,
    and that triggered a bug in the kernels timer implementation.
    It crashed older kernels, and on newer kernels it caused applications
    using futexes to use 100% cpu ..

    See for example

    http://www.devand.com/index.php/technology/12-tech/11-leap

    or Google for "linux leap second bug 2012".

    Mike.
    Miquel van Smoorenburg, Jul 17, 2012
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. mcollier

    w3wp.exe High CPU Utilization

    mcollier, Mar 11, 2005, in forum: ASP .Net
    Replies:
    6
    Views:
    60,910
    jeandemo
    Apr 13, 2011
  2. dm3281
    Replies:
    2
    Views:
    520
    bruce barker \(sqlwork.com\)
    Oct 13, 2006
  3. Replies:
    4
    Views:
    6,019
    Gordon Beaton
    Dec 18, 2007
  4. Madhur

    High CPU utilization with Pylab

    Madhur, Sep 5, 2010, in forum: Python
    Replies:
    0
    Views:
    355
    Madhur
    Sep 5, 2010
  5. Brian Adkins

    High ActiveRecord CPU Utilization

    Brian Adkins, Mar 7, 2007, in forum: Ruby
    Replies:
    34
    Views:
    414
    M. Edward (Ed) Borasky
    Mar 9, 2007
Loading...

Share This Page