本文参考了深度解读Tomcat中的NIO模型 和Tomcat NIO 模型的实现两篇文章，在看本文之前要先看一下这两位大神的分析，写的很好。但笔者对上述文章中关于NioBlockingSelector和BlockPoller的部分理解有些不太顺畅，所以下面只针对这一部分按照自己的思路整理出了本文。
Tomcat nio读request body与写response都是阻塞的
即使用了NIO模式，在tomcat中读取request body和写response的时候，根据servlet规范需要使用从Request和Response两个类获取流来进行读写，而ServletInputStream和 ServletOutputStream根据servlet规范是要求阻塞读写的。
比如在Request处理时，工作线程读请求line和header的时候是非阻塞的，而读request body是阻塞的。而由于accept的socket设置blocking false，所以要找到一个办法去让工作线程阻塞的去处理非阻塞的socket。
如何以阻塞的方式向非阻塞的socket进行读写
我们在org.apache.tomcat.util.net.NioEndpoint这个tomcat的NIO模式下连接和线程处理的核心组件中，可以找到Endponit初始化代码：
public void bind() throws Exception {
initServerSocket(); //1、初始化Server Socket
setStopLatch(new CountDownLatch(1));
// Initialize SSL if needed
initialiseSsl();
selectorPool.open(getName()); //实例化shared selector，启动BlockPoller线程
}
直接看一下最后
//selectorPool.open
public void open(String name) throws IOException {
enabled = true;
getSharedSelector(); //NioSelectorPool里的共享selector、区别于Poller里的selector
if (shared) {
blockingSelector = new NioBlockingSelector(); //实例化NioBlockingSelector
blockingSelector.open(name, getSharedSelector()); //把共享selector传给NioBlockingSelector
}
}
NioBlockingSelector.open()
public void open(String name, Selector selector) {
sharedSelector = selector;
poller = new BlockPoller(); //实例化BlockPoller
poller.selector = sharedSelector;
poller.setDaemon(true);
poller.setName(name + "-BlockPoller");
poller.start(); //启动BlockPoller线程
}
NioBlockingSelector 和 BlockPoller，前者提供读写方法、如果一次没读写完则阻塞在一个读写锁上等待通知就绪，后者内部是selector和轮询线程、负责epoll出来当前读写就绪的连接。当读写就绪时，就会打开连接上的读写锁（CountDownLatch实现），让阻塞在锁上的线程继续读写。接下来以写response为例，看一下这个过程。
这里我们可以从我们写的Servlet程序的response.getWriter().write()方法处用单步跟踪调试的办法一点一点的跟踪到NioBlockingSelector.write()，笔者也是这么试过的，但是也可以在“知道了答案的”情况下，偷懒直接通过在NioBlockingSelector.write()方法上打断点，通过观察线程栈的方法来了解这里的调用链路。断点打好之后，调用我们编写的servlet，在response.getWriter().write(）写返回的时候，找到如下的调用栈：
"http-nio-8080-exec-2" #26 daemon prio=5 os_prio=0 tid=0x000000002964d000 nid=0x7390 at breakpoint[0x000000002a08e000]
java.lang.Thread.State: RUNNABLE
at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:85)
at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:152)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1253)
at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:764)
at org.apache.tomcat.util.net.SocketWrapperBase.flushBlocking(SocketWrapperBase.java:717)
at org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:707)
at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.end(Http11OutputBuffer.java:567)
at org.apache.coyote.http11.filters.IdentityOutputFilter.end(IdentityOutputFilter.java:123)
at org.apache.coyote.http11.Http11OutputBuffer.end(Http11OutputBuffer.java:234)
at org.apache.coyote.http11.Http11Processor.finishResponse(Http11Processor.java:1162)
at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:389)
at org.apache.coyote.Response.action(Response.java:209)
at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:261)
at org.apache.catalina.connector.Response.finishResponse(Response.java:443)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:374)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:888)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1597)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000000d73412f0> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
ava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
可以看到在response写了缓冲之后，针对response响应（注意不是socket连接）做了“end and flush”，将数据从缓冲区冲刷到socket，这个冲刷实际上就是NioSocketWrapper.doWrite() -> NioSelectorPool.write() -> NioBlockingSelector.write()这样一个调用链路。所以最后我们来看NioBlockingSelector.write()方法：
public int write(ByteBuffer buf, NioChannel socket, long writeTimeout)
throws IOException {
SelectionKey key = socket.getIOChannel().keyFor(socket.getSocketWrapper().getPoller().getSelector());
NioSocketWrapper att = (NioSocketWrapper) key.attachment();
int written = 0;
boolean timedout = false;
int keycount = 1; //assume we can write
long time = System.currentTimeMillis(); //start the timeout timer
while (!timedout && buf.hasRemaining()) {
if (keycount > 0) { //only write if we were registered for a write
int cnt = socket.write(buf); //write the data 写数据
if (cnt == -1) {
throw new EOFException();
}
written += cnt;
if (cnt > 0) {
time = System.currentTimeMillis(); //reset our timeout timer
continue; //we successfully wrote, try again without a selector
}
}
try {
if (att.getWriteLatch() == null || att.getWriteLatch().getCount() == 0) {
att.startWriteLatch(1);
}
poller.add(att, SelectionKey.OP_WRITE, reference); //注册到BlockPoller
att.awaitWriteLatch(AbstractEndpoint.toTimeout(writeTimeout), TimeUnit.MILLISECONDS); //等待WriteLatch可写就绪
} catch (InterruptedException ignore) {
// Ignore
}
}
return written;
}
上面有个awaitWriteLatch的操作，依靠writeLatch来阻塞在等待写就绪事件上，而写就绪事件已经通过BlockPoller.add注册到BlockPoller了，至此，如何使用一个非阻塞的socket模拟阻塞写的过程搞清楚了:
socket.write()
-> 没写完，OP_WRITE注册到BlockPoller，线程阻塞等待awaitWriteLatch -> BlockPoller selector获得写就绪、writerLatch.countdown()通知
-> 线程继续写。
最后看一下tomcat9 的线程，可以看到跟8.5相比Poller只启动1个、就是那个ClientPoller线程，另外BlockPoller线程的作用上面也分析了。
Tomcat 9的线程
来自jstack的输出
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode):
"DestroyJavaVM" #27 prio=5 os_prio=0 tid=0x000000001dde2800 nid=0x5ce4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"http-nio-8080-Acceptor" #26 daemon prio=5 os_prio=0 tid=0x000000001ddde000 nid=0x75d4 runnable [0x00000000219ae000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
- locked <0x00000000d5f1b720> (a java.lang.Object)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:469)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:71)
at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:106)
at java.lang.Thread.run(Thread.java:745)
"http-nio-8080-ClientPoller" #25 daemon prio=5 os_prio=0 tid=0x000000001dddf000 nid=0x3e6c runnable [0x00000000218ae000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000d601a178> (a sun.nio.ch.Util$2)
- locked <0x00000000d601a168> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000d601a018> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:711)
at java.lang.Thread.run(Thread.java:745)
"http-nio-8080-exec-3" #24 daemon prio=5 os_prio=0 tid=0x000000001bd46800 nid=0xfa4 waiting on condition [0x00000000217af000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000d5fdefa8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
"http-nio-8080-exec-2" #23 daemon prio=5 os_prio=0 tid=0x000000001bd45800 nid=0x7584 waiting on condition [0x000000002145d000]
"http-nio-8080-exec-1" #22 daemon prio=5 os_prio=0 tid=0x000000001bd43800 nid=0x5744 waiting on condition [0x000000002135f000]
"http-nio-8080-BlockPoller" #21 daemon prio=5 os_prio=0 tid=0x000000001bd4a000 nid=0x7410 runnable [0x000000002003f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000d5f1dc20> (a sun.nio.ch.Util$2)
- locked <0x00000000d5f1db98> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000d5f1d9e0> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:313)
"container-0" #19 prio=5 os_prio=0 tid=0x000000001bd49800 nid=0x3558 waiting on condition [0x000000001fc3f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:570)
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer$1.run(TomcatWebServer.java:180)
"Catalina-utility-2" #18 prio=1 os_prio=-2 tid=0x000000001bd47000 nid=0x605c waiting on condition [0x000000001e92f000]
...
"Catalina-utility-1" #17 prio=1 os_prio=-2 tid=0x000000001bd48800 nid=0x1454 waiting on condition [0x000000001e82e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000081ca6288> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
"Service Thread" #10 daemon prio=9 os_prio=0 tid=0x000000001a12e000 nid=0x4b58 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread3" #9 daemon prio=9 os_prio=2 tid=0x000000001a083800 nid=0x22e0 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread2" #8 daemon prio=9 os_prio=2 tid=0x000000001a074800 nid=0x5d90 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #7 daemon prio=9 os_prio=2 tid=0x000000001a073000 nid=0x4d5c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #6 daemon prio=9 os_prio=2 tid=0x000000001a06d000 nid=0x4b0 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x000000001a06a800 nid=0x1ed8 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x000000001a01f800 nid=0x5528 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x0000000017f73800 nid=0x719c in Object.wait() [0x0000000019f7f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000814a9548> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x0000000002fa9000 nid=0x32dc in Object.wait() [0x0000000019e7f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x0000000081464228> (a java.lang.ref.Reference$Lock)
"VM Thread" os_prio=2 tid=0x0000000017f69000 nid=0x7264 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x0000000002ecc000 nid=0x2234 runnable
...
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x0000000002ed9800 nid=0x77e4 runnable
"VM Periodic Task Thread" os_prio=2 tid=0x000000001a1a2000 nid=0x3b4 waiting on condition
JNI global references: 1183
网友评论