L
lightning
When using net.spy.memcached, I come with this problem, I use
MemcachedClient.get to get from memcached,but sometimes my app logs
exception info below and the cpu goes 100%.
the code I use to call mc:
long userid = -1;
Object obj = mc1.get(id);
if(obj ==null){
obj = mc2.get(id);
}
I did not catch the OperationTimeoutException and I just log every
exception because I don't know what else need to do to make everything
end peacefully.
the log is as below:
2008-10-23 20:20:21.045 INFO net.spy.memcached.MemcachedConnection:
Reconnecting due to exception on {QA sa=/10.23.245.184:11212
, #Rops=2, #Wops=0, #iq=0,
topRop=net.spy.memcached.protocol.ascii.GetOperationImpl@1e5e28f,
topWop=null, toWrite=43, interested=
5}
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
at sun.nio.ch.IOUtil.write(IOUtil.java:75)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:
302)
at
net.spy.memcached.protocol.TCPMemcachedNodeImpl.writeSome(TCPMemcachedNodeImpl.java:
360)
at
net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:
283)
at
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:
265)
at
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:
182)
at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:
1259)
2008-10-23 20:20:21.045 WARN net.spy.memcached.MemcachedConnection:
Closing, and reopening {QA sa=/10.23.245.184:11212, #Rops=2,
#Wops=0, #iq=0,
topRop=net.spy.memcached.protocol.ascii.GetOperationImpl@1e5e28f,
topWop=null, toWrite=43, interested=5}, attemp
t 0.
2008-10-23 20:20:21.046 WARN
net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
partially completed op: net.spy
..memcached.protocol.ascii.GetOperationImpl@1e5e28f
2008-10-23 20:20:21.046 WARN
net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
partially completed op: net.spy
..memcached.protocol.ascii.GetOperationImpl@17d1f5c
2008-10-23 20:20:21,046 [http-8080-Processor24] ERROR [/fc-mc].
[action] (StandardWrapperValve.java:260) - Servlet.service() f
or servlet action threw exception
java.lang.RuntimeException: Exception waiting for value
at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:
769)
at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:
784)
at
com.baidu.fengchao.earth.memcache.SessionService.getUserid(SessionService.java:
253)
at
com.baidu.fengchao.mercury.web.util.SessionCmd.getAdminid(Unknown
Source)
at
com.baidu.fengchao.mercury.web.util.EnvSetterFilter.doFilter(Unknown
Source)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
202)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
173)
at
com.baidu.fengchao.mercury.web.util.PathFilter.doFilter(Unknown
Source)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
202)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
173)
at
com.baidu.fengchao.mercury.web.util.SetCharacterEncodingFilter.doFilter(Unknown
Source)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
202)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:
213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:
178)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:
126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:
105)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:
107)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:
148)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
869)
at org.apache.coyote.http11.Http11BaseProtocol
$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:
664)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:
527)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:
80)
at org.apache.tomcat.util.threads.ThreadPool
$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Cancelled
at net.spy.memcached.MemcachedClient
$OperationFuture.get(MemcachedClient.java:1443)
at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:
764)
... 24 more
Caused by: java.lang.RuntimeException: Cancelled
... 26 more
2008-10-23 20:20:21.149 INFO net.spy.memcached.MemcachedConnection:
Reconnecting {QA sa=/10.23.245.184:11212, #Rops=0, #Wops=0,
#iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
2008-10-23 20:20:21.149 INFO net.spy.memcached.MemcachedConnection:
Connection state changed for sun.nio.ch.SelectionKeyImpl@195
fbd6
2008-10-23 20:28:09,756 [http-8080-Processor19] INFO
util.EnvSetterFilter ( ?:?) - request userid: 265045
2008-10-23 20:28:09.757 INFO net.spy.memcached.MemcachedConnection:
Reconnecting due to exception on {QA sa=/10.23.244.207:11212
, #Rops=2, #Wops=0, #iq=0,
topRop=net.spy.memcached.protocol.ascii.StoreOperationImpl@782d55,
topWop=null, toWrite=317, intereste
d=5}
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
at sun.nio.ch.IOUtil.write(IOUtil.java:75)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:
302)
at
net.spy.memcached.protocol.TCPMemcachedNodeImpl.writeSome(TCPMemcachedNodeImpl.java:
360)
at
net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:
283)
at
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:
265)
at
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:
182)
at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:
1259)
2008-10-23 20:28:09.758 WARN net.spy.memcached.MemcachedConnection:
Closing, and reopening {QA sa=/10.23.244.207:11212, #Rops=2,
#Wops=0, #iq=0,
topRop=net.spy.memcached.protocol.ascii.StoreOperationImpl@782d55,
topWop=null, toWrite=317, interested=5}, atte
mpt 0.
2008-10-23 20:28:09.758 WARN
net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
partially completed op: net.spy
..memcached.protocol.ascii.StoreOperationImpl@782d55
2008-10-23 20:28:09.758 WARN
net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
partially completed op: net.spy
..memcached.protocol.ascii.StoreOperationImpl@1e833ff
2008-10-23 20:28:09,765 [http-8080-Processor19] INFO
util.EnvSetterFilter ( ?:?) - [fc-star-mecury] -
172.18.110.
176 - /fc-mc/spread/AdPreview.do
2008-10-23 20:28:09.861 INFO net.spy.memcached.MemcachedConnection:
Reconnecting {QA sa=/10.23.244.207:11212, #Rops=0, #Wops=0,
#iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
2008-10-23 20:28:09.861 INFO net.spy.memcached.MemcachedConnection:
Connection state changed for sun.nio.ch.SelectionKeyImpl@1f5
b5a6
MemcachedClient.get to get from memcached,but sometimes my app logs
exception info below and the cpu goes 100%.
the code I use to call mc:
long userid = -1;
Object obj = mc1.get(id);
if(obj ==null){
obj = mc2.get(id);
}
I did not catch the OperationTimeoutException and I just log every
exception because I don't know what else need to do to make everything
end peacefully.
the log is as below:
2008-10-23 20:20:21.045 INFO net.spy.memcached.MemcachedConnection:
Reconnecting due to exception on {QA sa=/10.23.245.184:11212
, #Rops=2, #Wops=0, #iq=0,
topRop=net.spy.memcached.protocol.ascii.GetOperationImpl@1e5e28f,
topWop=null, toWrite=43, interested=
5}
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
at sun.nio.ch.IOUtil.write(IOUtil.java:75)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:
302)
at
net.spy.memcached.protocol.TCPMemcachedNodeImpl.writeSome(TCPMemcachedNodeImpl.java:
360)
at
net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:
283)
at
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:
265)
at
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:
182)
at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:
1259)
2008-10-23 20:20:21.045 WARN net.spy.memcached.MemcachedConnection:
Closing, and reopening {QA sa=/10.23.245.184:11212, #Rops=2,
#Wops=0, #iq=0,
topRop=net.spy.memcached.protocol.ascii.GetOperationImpl@1e5e28f,
topWop=null, toWrite=43, interested=5}, attemp
t 0.
2008-10-23 20:20:21.046 WARN
net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
partially completed op: net.spy
..memcached.protocol.ascii.GetOperationImpl@1e5e28f
2008-10-23 20:20:21.046 WARN
net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
partially completed op: net.spy
..memcached.protocol.ascii.GetOperationImpl@17d1f5c
2008-10-23 20:20:21,046 [http-8080-Processor24] ERROR [/fc-mc].
[action] (StandardWrapperValve.java:260) - Servlet.service() f
or servlet action threw exception
java.lang.RuntimeException: Exception waiting for value
at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:
769)
at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:
784)
at
com.baidu.fengchao.earth.memcache.SessionService.getUserid(SessionService.java:
253)
at
com.baidu.fengchao.mercury.web.util.SessionCmd.getAdminid(Unknown
Source)
at
com.baidu.fengchao.mercury.web.util.EnvSetterFilter.doFilter(Unknown
Source)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
202)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
173)
at
com.baidu.fengchao.mercury.web.util.PathFilter.doFilter(Unknown
Source)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
202)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
173)
at
com.baidu.fengchao.mercury.web.util.SetCharacterEncodingFilter.doFilter(Unknown
Source)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
202)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:
213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:
178)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:
126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:
105)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:
107)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:
148)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
869)
at org.apache.coyote.http11.Http11BaseProtocol
$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:
664)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:
527)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:
80)
at org.apache.tomcat.util.threads.ThreadPool
$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Cancelled
at net.spy.memcached.MemcachedClient
$OperationFuture.get(MemcachedClient.java:1443)
at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:
764)
... 24 more
Caused by: java.lang.RuntimeException: Cancelled
... 26 more
2008-10-23 20:20:21.149 INFO net.spy.memcached.MemcachedConnection:
Reconnecting {QA sa=/10.23.245.184:11212, #Rops=0, #Wops=0,
#iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
2008-10-23 20:20:21.149 INFO net.spy.memcached.MemcachedConnection:
Connection state changed for sun.nio.ch.SelectionKeyImpl@195
fbd6
2008-10-23 20:28:09,756 [http-8080-Processor19] INFO
util.EnvSetterFilter ( ?:?) - request userid: 265045
2008-10-23 20:28:09.757 INFO net.spy.memcached.MemcachedConnection:
Reconnecting due to exception on {QA sa=/10.23.244.207:11212
, #Rops=2, #Wops=0, #iq=0,
topRop=net.spy.memcached.protocol.ascii.StoreOperationImpl@782d55,
topWop=null, toWrite=317, intereste
d=5}
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
at sun.nio.ch.IOUtil.write(IOUtil.java:75)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:
302)
at
net.spy.memcached.protocol.TCPMemcachedNodeImpl.writeSome(TCPMemcachedNodeImpl.java:
360)
at
net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:
283)
at
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:
265)
at
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:
182)
at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:
1259)
2008-10-23 20:28:09.758 WARN net.spy.memcached.MemcachedConnection:
Closing, and reopening {QA sa=/10.23.244.207:11212, #Rops=2,
#Wops=0, #iq=0,
topRop=net.spy.memcached.protocol.ascii.StoreOperationImpl@782d55,
topWop=null, toWrite=317, interested=5}, atte
mpt 0.
2008-10-23 20:28:09.758 WARN
net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
partially completed op: net.spy
..memcached.protocol.ascii.StoreOperationImpl@782d55
2008-10-23 20:28:09.758 WARN
net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
partially completed op: net.spy
..memcached.protocol.ascii.StoreOperationImpl@1e833ff
2008-10-23 20:28:09,765 [http-8080-Processor19] INFO
util.EnvSetterFilter ( ?:?) - [fc-star-mecury] -
172.18.110.
176 - /fc-mc/spread/AdPreview.do
2008-10-23 20:28:09.861 INFO net.spy.memcached.MemcachedConnection:
Reconnecting {QA sa=/10.23.244.207:11212, #Rops=0, #Wops=0,
#iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
2008-10-23 20:28:09.861 INFO net.spy.memcached.MemcachedConnection:
Connection state changed for sun.nio.ch.SelectionKeyImpl@1f5
b5a6