Question About using MemcachedClient coming with 100% cpu problem

Discussion in 'Java' started by lightning, Oct 24, 2008.

  1. lightning

    lightning Guest

    When using net.spy.memcached, I come with this problem, I use
    MemcachedClient.get to get from memcached,but sometimes my app logs
    exception info below and the cpu goes 100%.


    the code I use to call mc:

    long userid = -1;
    Object obj = mc1.get(id);
    if(obj ==null){
    obj = mc2.get(id);
    }

    I did not catch the OperationTimeoutException and I just log every
    exception because I don't know what else need to do to make everything
    end peacefully.

    the log is as below:




    2008-10-23 20:20:21.045 INFO net.spy.memcached.MemcachedConnection:
    Reconnecting due to exception on {QA sa=/10.23.245.184:11212
    , #Rops=2, #Wops=0, #iq=0,
    topRop=net.spy.memcached.protocol.ascii.GetOperationImpl@1e5e28f,
    topWop=null, toWrite=43, interested=
    5}
    java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcher.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
    at sun.nio.ch.IOUtil.write(IOUtil.java:75)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:
    302)
    at
    net.spy.memcached.protocol.TCPMemcachedNodeImpl.writeSome(TCPMemcachedNodeImpl.java:
    360)
    at
    net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:
    283)
    at
    net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:
    265)
    at
    net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:
    182)
    at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:
    1259)
    2008-10-23 20:20:21.045 WARN net.spy.memcached.MemcachedConnection:
    Closing, and reopening {QA sa=/10.23.245.184:11212, #Rops=2,
    #Wops=0, #iq=0,
    topRop=net.spy.memcached.protocol.ascii.GetOperationImpl@1e5e28f,
    topWop=null, toWrite=43, interested=5}, attemp
    t 0.
    2008-10-23 20:20:21.046 WARN
    net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
    partially completed op: net.spy
    ..memcached.protocol.ascii.GetOperationImpl@1e5e28f
    2008-10-23 20:20:21.046 WARN
    net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
    partially completed op: net.spy
    ..memcached.protocol.ascii.GetOperationImpl@17d1f5c
    2008-10-23 20:20:21,046 [http-8080-Processor24] ERROR [/fc-mc].
    [action] (StandardWrapperValve.java:260) - Servlet.service() f
    or servlet action threw exception
    java.lang.RuntimeException: Exception waiting for value
    at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:
    769)
    at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:
    784)
    at
    com.baidu.fengchao.earth.memcache.SessionService.getUserid(SessionService.java:
    253)
    at
    com.baidu.fengchao.mercury.web.util.SessionCmd.getAdminid(Unknown
    Source)
    at
    com.baidu.fengchao.mercury.web.util.EnvSetterFilter.doFilter(Unknown
    Source)
    at
    org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
    202)
    at
    org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
    173)
    at
    com.baidu.fengchao.mercury.web.util.PathFilter.doFilter(Unknown
    Source)
    at
    org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
    202)
    at
    org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
    173)
    at
    com.baidu.fengchao.mercury.web.util.SetCharacterEncodingFilter.doFilter(Unknown
    Source)
    at
    org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
    202)
    at
    org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
    173)
    at
    org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:
    213)
    at
    org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:
    178)
    at
    org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:
    126)
    at
    org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:
    105)
    at
    org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:
    107)
    at
    org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:
    148)
    at
    org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
    869)
    at org.apache.coyote.http11.Http11BaseProtocol
    $Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:
    664)
    at
    org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:
    527)
    at
    org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:
    80)
    at org.apache.tomcat.util.threads.ThreadPool
    $ControlRunnable.run(ThreadPool.java:684)
    at java.lang.Thread.run(Thread.java:595)
    Caused by: java.util.concurrent.ExecutionException:
    java.lang.RuntimeException: Cancelled
    at net.spy.memcached.MemcachedClient
    $OperationFuture.get(MemcachedClient.java:1443)
    at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:
    764)
    ... 24 more
    Caused by: java.lang.RuntimeException: Cancelled
    ... 26 more
    2008-10-23 20:20:21.149 INFO net.spy.memcached.MemcachedConnection:
    Reconnecting {QA sa=/10.23.245.184:11212, #Rops=0, #Wops=0,
    #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
    2008-10-23 20:20:21.149 INFO net.spy.memcached.MemcachedConnection:
    Connection state changed for sun.nio.ch.SelectionKeyImpl@195
    fbd6
    2008-10-23 20:28:09,756 [http-8080-Processor19] INFO
    util.EnvSetterFilter ( ?:?) - request userid: 265045
    2008-10-23 20:28:09.757 INFO net.spy.memcached.MemcachedConnection:
    Reconnecting due to exception on {QA sa=/10.23.244.207:11212
    , #Rops=2, #Wops=0, #iq=0,
    topRop=net.spy.memcached.protocol.ascii.StoreOperationImpl@782d55,
    topWop=null, toWrite=317, intereste
    d=5}
    java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcher.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
    at sun.nio.ch.IOUtil.write(IOUtil.java:75)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:
    302)
    at
    net.spy.memcached.protocol.TCPMemcachedNodeImpl.writeSome(TCPMemcachedNodeImpl.java:
    360)
    at
    net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:
    283)
    at
    net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:
    265)
    at
    net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:
    182)
    at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:
    1259)
    2008-10-23 20:28:09.758 WARN net.spy.memcached.MemcachedConnection:
    Closing, and reopening {QA sa=/10.23.244.207:11212, #Rops=2,
    #Wops=0, #iq=0,
    topRop=net.spy.memcached.protocol.ascii.StoreOperationImpl@782d55,
    topWop=null, toWrite=317, interested=5}, atte
    mpt 0.
    2008-10-23 20:28:09.758 WARN
    net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
    partially completed op: net.spy
    ..memcached.protocol.ascii.StoreOperationImpl@782d55
    2008-10-23 20:28:09.758 WARN
    net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
    partially completed op: net.spy
    ..memcached.protocol.ascii.StoreOperationImpl@1e833ff
    2008-10-23 20:28:09,765 [http-8080-Processor19] INFO
    util.EnvSetterFilter ( ?:?) - [fc-star-mecury] -
    172.18.110.
    176 - /fc-mc/spread/AdPreview.do
    2008-10-23 20:28:09.861 INFO net.spy.memcached.MemcachedConnection:
    Reconnecting {QA sa=/10.23.244.207:11212, #Rops=0, #Wops=0,
    #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
    2008-10-23 20:28:09.861 INFO net.spy.memcached.MemcachedConnection:
    Connection state changed for sun.nio.ch.SelectionKeyImpl@1f5
    b5a6
     
    lightning, Oct 24, 2008
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Steve Holden

    PyCon is Coming! PyCon is Coming!

    Steve Holden, Jan 5, 2006, in forum: Python
    Replies:
    0
    Views:
    320
    Steve Holden
    Jan 5, 2006
  2. Alex via .NET 247

    using COM from aspx causes CPU to rise to 100%

    Alex via .NET 247, Aug 4, 2004, in forum: ASP .Net Security
    Replies:
    0
    Views:
    120
    Alex via .NET 247
    Aug 4, 2004
  3. jrbrady

    Web Server Using 100% CPU Time

    jrbrady, Apr 19, 2006, in forum: ASP .Net Web Services
    Replies:
    0
    Views:
    132
    jrbrady
    Apr 19, 2006
  4. fred
    Replies:
    3
    Views:
    285
    Zifud
    Mar 17, 2005
  5. Replies:
    5
    Views:
    891
Loading...

Share This Page