1

Studying about blocking and non blocking IO, created some doubts in my mind.

I came to read a statement written here (https://mail-archives.apache.org/mod_mbox/tomcat-users/201402.mbox/%3C52F521B7.2060207@apache.org%3E) that "Even the NIO HTTP connector has to use blocking IO to read the request body and write the response:.

Can anyone please explain this?

Secondly, I was running jmeter load testing on tomcat server for one of the API used for fetching some details of a user in our project. Initially I was making 2-3 requests/secs and the application was able to maintain that throughput, but as soon as I tried 15-20 requests/sec, tomcat server's performance time increased exponentially. Thread Dump shows that Nio connector threads are waiting to write the data, please provide the solution so that threads do not get blocked.

"http-nio-8080-exec-1502" - Thread t@103682
   java.lang.Thread.State: TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <400f1e16> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.tomcat.util.net.NioEndpoint$KeyAttachment.awaitLatch(NioEndpoint.java:1381)
        at org.apache.tomcat.util.net.NioEndpoint$KeyAttachment.awaitWriteLatch(NioEndpoint.java:1384)
        at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:114)
        at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:172)
        at org.apache.coyote.http11.InternalNioOutputBuffer.writeToSocket(InternalNioOutputBuffer.java:139)
        at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(InternalNioOutputBuffer.java:197)
        - locked <24b951cc> (a org.apache.coyote.http11.InternalNioOutputBuffer)
        at org.apache.coyote.http11.InternalNioOutputBuffer.access$000(InternalNioOutputBuffer.java:41)
        at org.apache.coyote.http11.InternalNioOutputBuffer$SocketOutputBuffer.doWrite(InternalNioOutputBuffer.java:320)
        at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:118)
        at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:256)
        at org.apache.coyote.Response.doWrite(Response.java:491)
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:391)
        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:426)
        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:339)
        at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:421)
        at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:409)
        at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:97)
        at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2039)
        at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:454)
        at com.fasterxml.jackson.databind.ser.std.StringSerializer.serialize(StringSerializer.java:49)
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704)
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:690)
        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704)
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:690)
        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704)
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:690)
        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704)
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:690)
        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704)
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:690)
        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:292)
        at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1429)
        at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:951)
        at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:264)
        at org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:100)
        at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:167)
        at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:100)
        at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.handleReturnValue(RequestResponseBodyMethodProcessor.java:166)
        at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:80)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:127)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:806)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:729)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:528)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1099)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476)
        - locked <368bdc01> (a org.apache.tomcat.util.net.NioChannel)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
Bhanu Pasrija
  • 11
  • 1
  • 3
  • "tomcat NIO connectors use blocking IO or read/write" <- this does not make much sense. If you do I/O then you do read/writes. So there is no "or" involved. Saying "for read/writes tomcat uses blocking NIO", would make sense, but I do not know whether it's accurate. Also you could improve your question by linking to that "somewhere" where you have found that statement. – Tomáš Pospíšek Mar 03 '18 at 22:39
  • Hi @TomášPospíšek, I have added the link here. – Bhanu Pasrija Mar 04 '18 at 06:08
  • The phrase "Even the NIO HTTP connector has to use blocking IO to read the request body and write the response" means that when the NIO HTTP connector is called in a programm, then that call will block the thread or the process that is doing the call until the data from that call is available and can be returned to the caller. "Block" here means, that the process/thread does not get any CPU time. – Tomáš Pospíšek Mar 05 '18 at 07:45
  • Thanks @TomášPospíšek, now I understand that the program which called the worker thread is getting blocked. Can you please also tell that what is this thread is getting blocked on? Is it not getting selector to write the data into channel? – Bhanu Pasrija Mar 06 '18 at 06:47

1 Answers1

0

I am unable to tell you. In the stack trace I see some http and json libraries. What I'd do is to use a java profiler that'd tell you where the application is spending it's time. That way you should be able to find out which line in your code get's blocked. Alternatively you can insert log statements into your code that record the time when they get called. And so you'll see where the app is waiting. But I guess that tomcat is calling your application (and not the other way around) and so it could be that the waiting is outside of your application.

Tomáš Pospíšek
  • 233
  • 1
  • 7