I `m trying to put data into openTSDB with RESTful api the openTSDB provided. But the opentsdb process always hang there after a few minutes. It can successfully put some data then becomes slower and slower. Finally it will throw some exceptions in the log.
Please check below log:
2015-08-31 21:56:48,360 INFO [New I/O worker #48] HttpQuery: [id: 0x38971101, /10.75.44.33:34549 :> /10.75.44.33:4242] HTTP /api/put?details done in 388999ms
2015-08-31 21:56:48,360 INFO [New I/O server boss #65] ConnectionManager: [id: 0x835153bf, /10.75.44.33:34618 => /10.75.44.33:4242] OPEN 2015-08-31 21:56:48,361 INFO [New I/O worker #40] ConnectionManager: [id: 0x835153bf, /10.75.44.33:34618 => /10.75.44.33:4242] BOUND: /10.75.44.33:4242 2015-08-31 21:56:48,361 INFO [New I/O server boss #65] ConnectionManager: [id: 0xdf86eb92, /0:0:0:0:0:0:0:1:35607 => /0:0:0:0:0:0:0:1:4242] OPEN 2015-08-31 21:56:48,361 INFO [New I/O worker #40] ConnectionManager: [id: 0x835153bf, /10.75.44.33:34618 => /10.75.44.33:4242] CONNECTED: /10.75.44.33:34618 2015-08-31 21:56:48,361 INFO [New I/O worker #41] ConnectionManager: [id: 0xdf86eb92, /0:0:0:0:0:0:0:1:35607 => /0:0:0:0:0:0:0:1:4242] BOUND: /0:0:0:0:0:0:0:1:4242 2015-08-31 21:56:48,361 INFO [New I/O worker #41] ConnectionManager: [id: 0xdf86eb92, /0:0:0:0:0:0:0:1:35607 => /0:0:0:0:0:0:0:1:4242] CONNECTED: /0:0:0:0:0:0:0:1:35607 2015-08-31 21:56:48,361 INFO [New I/O worker #41] HttpQuery: [id: 0xdf86eb92, /0:0:0:0:0:0:0:1:35607 => /0:0:0:0:0:0:0:1:4242] HTTP /api/version done in 0ms 2015-08-31 21:56:48,361 INFO [New I/O worker #41] ConnectionManager: [id: 0xdf86eb92, /0:0:0:0:0:0:0:1:35607 :> /0:0:0:0:0:0:0:1:4242] DISCONNECTED 2015-08-31 21:56:48,361 INFO [New I/O worker #41] ConnectionManager: [id: 0xdf86eb92, /0:0:0:0:0:0:0:1:35607 :> /0:0:0:0:0:0:0:1:4242] UNBOUND 2015-08-31 21:56:48,362 INFO [New I/O worker #41] ConnectionManager: [id: 0xdf86eb92, /0:0:0:0:0:0:0:1:35607 :> /0:0:0:0:0:0:0:1:4242] CLOSED
2015-08-31 21:58:23,436 ERROR [New I/O worker #40] ConnectionManager: Unexpected exception from downstream for [id: 0x835153bf, /10.75.44.33:34618 => /10.75.44.33:4242] java.lang.OutOfMemoryError: Java heap space at java.nio.HeapCharBuffer.(HeapCharBuffer.java:57) ~[na:1.7.0_85] at java.nio.CharBuffer.allocate(CharBuffer.java:331) ~[na:1.7.0_85] at org.jboss.netty.buffer.ChannelBuffers.decodeString(ChannelBuffers.java:1193) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.buffer.AbstractChannelBuffer.toString(AbstractChannelBuffer.java:551) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.buffer.AbstractChannelBuffer.toString(AbstractChannelBuffer.java:543) ~[netty-3.9.4.Final.jar:na] at net.opentsdb.tsd.HttpQuery.getContent(HttpQuery.java:459) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.tsd.HttpJsonSerializer.parsePutV1(HttpJsonSerializer.java:133) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.tsd.PutDataPointRpc.execute(PutDataPointRpc.java:102) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.tsd.RpcHandler.handleHttpQuery(RpcHandler.java:273) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.tsd.RpcHandler.messageReceived(RpcHandler.java:180) ~[tsdb-2.1.0.jar:c775b5f] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.timeout.IdleStateAwareChannelUpstreamHandler.handleUpstream(IdleStateAwareChannelUpstreamHandler.java:36) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentEncoder.messageReceived(HttpContentEncoder.java:82) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:194) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[netty-3.9.4.Final.jar:na] 2015-08-31 21:58:57,956 INFO [New I/O worker #40] ConnectionManager: [id: 0x835153bf, /10.75.44.33:34618 :> /10.75.44.33:4242] DISCONNECTED 2015-08-31 21:58:57,956 INFO [New I/O worker #40] ConnectionManager: [id: 0x835153bf, /10.75.44.33:34618 :> /10.75.44.33:4242] UNBOUND 2015-08-31 21:58:57,956 INFO [New I/O worker #40] ConnectionManager: [id: 0x835153bf, /10.75.44.33:34618 :> /10.75.44.33:4242] CLOSED 2015-08-31 21:59:05,424 INFO [New I/O server boss #65] ConnectionManager: [id: 0x9a40399c, /10.75.44.33:34625 => /10.75.44.33:4242] OPEN 2015-08-31 21:59:05,424 INFO [New I/O worker #42] ConnectionManager: [id: 0x9a40399c, /10.75.44.33:34625 => /10.75.44.33:4242] BOUND: /10.75.44.33:4242 2015-08-31 21:59:05,424 INFO [New I/O worker #42] ConnectionManager: [id: 0x9a40399c, /10.75.44.33:34625 => /10.75.44.33:4242] CONNECTED: /10.75.44.33:34625 2015-08-31 22:02:21,615 ERROR [New I/O worker #31] ConnectionManager: Unexpected exception from downstream for [id: 0xec533f17, /10.75.44.33:34599 => /10.75.44.33:4242] java.lang.OutOfMemoryError: GC overhead limit exceeded at com.google.protobuf.ZeroCopyLiteralByteString.wrap(ZeroCopyLiteralByteString.java:52) ~[asynchbase-1.6.0.jar:na] at org.hbase.async.Bytes.wrap(Bytes.java:287) ~[asynchbase-1.6.0.jar:na] at org.hbase.async.PutRequest.toMutationProto(PutRequest.java:529) ~[asynchbase-1.6.0.jar:na] at org.hbase.async.MultiAction.serialize(MultiAction.java:229) ~[asynchbase-1.6.0.jar:na] at org.hbase.async.RegionClient.encode(RegionClient.java:1146) ~[asynchbase-1.6.0.jar:na] at org.hbase.async.RegionClient.sendRpc(RegionClient.java:894) ~[asynchbase-1.6.0.jar:na] at org.hbase.async.RegionClient.bufferEdit(RegionClient.java:757) ~[asynchbase-1.6.0.jar:na] at org.hbase.async.RegionClient.sendRpc(RegionClient.java:881) ~[asynchbase-1.6.0.jar:na] at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1698) ~[asynchbase-1.6.0.jar:na] at org.hbase.async.HBaseClient.put(HBaseClient.java:1343) ~[asynchbase-1.6.0.jar:na] at net.opentsdb.core.TSDB.addPointInternal(TSDB.java:681) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.TSDB.addPoint(TSDB.java:573) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.tsd.PutDataPointRpc.execute(PutDataPointRpc.java:146) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.tsd.RpcHandler.handleHttpQuery(RpcHandler.java:273) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.tsd.RpcHandler.messageReceived(RpcHandler.java:180) ~[tsdb-2.1.0.jar:c775b5f] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.timeout.IdleStateAwareChannelUpstreamHandler.handleUpstream(IdleStateAwareChannelUpstreamHandler.java:36) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentEncoder.messageReceived(HttpContentEncoder.java:82) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[netty-3.9.4.Final.jar:na] 2015-08-31 22:02:21,616 INFO [New I/O worker #31] ConnectionManager: [id: 0xec533f17, /10.75.44.33:34599 :> /10.75.44.33:4242] DISCONNECTED 2015-08-31 22:02:21,616 INFO [New I/O worker #31] ConnectionManager: [id: 0xec533f17, /10.75.44.33:34599 :> /10.75.44.33:4242] UNBOUND 2015-08-31 22:02:21,616 INFO [New I/O worker #31] ConnectionManager: [id: 0xec533f17, /10.75.44.33:34599 :> /10.75.44.33:4242] CLOSED 2015-08-31 22:02:21,616 ERROR [New I/O worker #66] RegionClient: Unexpected exception from downstream on [id: 0x49bdc01a, /10.75.44.33:40244 => /10.75.44.36:18020] java.lang.OutOfMemoryError: GC overhead limit exceeded at com.google.protobuf.ZeroCopyLiteralByteString.wrap(ZeroCopyLiteralByteString.java:52) ~[asynchbase-1.6.0.jar:na] at org.hbase.async.Bytes.wrap(Bytes.java:287) ~[asynchbase-1.6.0.jar:na] at org.hbase.async.GetRequest.serialize(GetRequest.java:385) ~[asynchbase-1.6.0.jar:na] at org.hbase.async.RegionClient.encode(RegionClient.java:1146) ~[asynchbase-1.6.0.jar:na] at org.hbase.async.RegionClient.sendRpc(RegionClient.java:894) ~[asynchbase-1.6.0.jar:na] at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1698) ~[asynchbase-1.6.0.jar:na] at org.hbase.async.HBaseClient.get(HBaseClient.java:995) ~[asynchbase-1.6.0.jar:na] at net.opentsdb.core.TSDB.get(TSDB.java:1090) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:185) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f] 2015-08-31 22:02:21,616 INFO [New I/O worker #66] HBaseClient: Lost connection with the .META. region 2015-08-31 22:02:21,617 ERROR [New I/O worker #67] RegionClient: Unexpected exception from downstream on [id: 0x97cc2014, /10.75.44.33:44182 => /10.75.44.35:18020] java.lang.OutOfMemoryError: GC overhead limit exceeded at org.hbase.async.HBaseClient.createRegionSearchKey(HBaseClient.java:1954) ~[asynchbase-1.6.0.jar:na] at org.hbase.async.HBaseClient.getRegion(HBaseClient.java:1984) ~[asynchbase-1.6.0.jar:na] at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1659) ~[asynchbase-1.6.0.jar:na] at org.hbase.async.HBaseClient.get(HBaseClient.java:995) ~[asynchbase-1.6.0.jar:na] at net.opentsdb.core.TSDB.get(TSDB.java:1090) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:185) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:191) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.CompactionQueue.flush(CompactionQueue.java:116) ~[tsdb-2.1.0.jar:c775b5f] at net.opentsdb.core.TSDB.flush(TSDB.java:730) ~[tsdb-2.1.0.jar:c775b5f] 2015-08-31 22:03:09,322 INFO [New I/O server boss #65] ConnectionManager: [id: 0x7c5ac3e7, /10.75.44.33:34633 => /10.75.44.33:4242] OPEN 2015-08-31 22:03:09,322 INFO [New I/O worker #43] ConnectionManager: [id: 0x7c5ac3e7, /10.75.44.33:34633 => /10.75.44.33:4242] BOUND: /10.75.44.33:4242 2015-08-31 22:03:09,322 INFO [New I/O worker #43] ConnectionManager: [id: 0x7c5ac3e7, /10.75.44.33:34633 => /10.75.44.33:4242] CONNECTED: /10.75.44.33:34633 2015-08-31 22:03:09,322 ERROR [New I/O worker #39] ConnectionManager: Unexpected exception from downstream for [id: 0xe5cfc336, /10.75.44.33:34614 => /10.75.44.33:4242] java.lang.OutOfMemoryError: GC overhead limit exceeded 2015-08-31 22:03:09,322 INFO [New I/O worker #39] ConnectionManager: [id: 0xe5cfc336, /10.75.44.33:34614 :> /10.75.44.33:4242] DISCONNECTED 2015-08-31 22:03:09,322 INFO [New I/O worker #39] ConnectionManager: [id: 0xe5cfc336, /10.75.44.33:34614 :> /10.75.44.33:4242] UNBOUND 2015-08-31 22:03:09,322 INFO [New I/O worker #39] ConnectionManager: [id: 0xe5cfc336, /10.75.44.33:34614 :> /10.75.44.33:4242] CLOSED
My question is below
When I put data, the memory usage for openTSDB process is becoming bigger and bigger. Is this normal? Because I think the cost for putting first file and second file should be the same. If the file is in same size. But it seems the memory usage for opentsdb is just going up
The log shows the error is from downstream. My understanding it means hbase. But i did not see any error from hbase. Why? what is the donwstream?
