20191126 Zuul服务器404问题

作者: 昵称个P | 来源:发表于2019-12-10 20:58 被阅读0次

20191126 Zuul服务器404问题
zuul 404
Spring Cloud Zuul（服务网关）
Nginx404页面http状态码返回200问题解决
Spring Cloud微服务之网关zuul配置项详解
0811&0812-0731-JS25
404 错误___<mvc:annotation-driv
2018-12-04 解决nginx搭建图片服务器访问图片404
SpringCloud-5之-Zuul网关服务
UIWebView获取404、504等错误码

问题出现

时间：2019-11-26 星期二
过程：下午18点推送了一条汽车摇号的push，之后app出现大量错误，包括404，500等。

问题修复

500错误

500错误的消息是当前连接人数太多，请稍后重试。这个错误是sentinel返回的，是应对大并发流量时的正常响应。在并发下降后，500错误消失

404错误

通过监控平台，发现404的错误不是在某个或某些个具体的接口上报出来的，分布在多个接口，涉及到多个组件，没有规律。所以可以排除是具体某个组件的问题。

首先，排除具体组件问题。从监控平台上挑选了几个请求，然后查看了对应的class文件是否存在，发现文件还在。说明不是具体组件问题。可能是某些公共组件的问题，比如nginx或者zuul。
分别查看了两台网关的日志，正常刷新，无异常
打开注册中心，zuul注册了4台，A,B,C,D
查看nginx配置文件，发现网关配置了4台，分别是A,B,C,E，和注册中心的不一样
查看E的zuul日志，发现最后一条在18:10:46分。当时查看时间是20点。所以判断E的zuul有问题
重启E的zuul，404消失。

问题定位

404错误定位

查看E机器的zuul的tomcat的日志access_txt.log，发现zuul在18:10:46之后，还有接收请求，但是响应状态码全部都是404，直到重启才恢复正常。
查看tomcat的gc日志，暂无发现
查看pinpoint的各项指标，未发现异常
查看业务日志，发现有下面很多报错。先是打开文件过多，然后tomcat容器关闭，spring关闭，最后eureka-client主动从注册中心上销毁。

2019-11-26 18:10:00 -  INFO - [d39cd5864fc941c1a3a04963c43848b4] - [http-apr-0.0.0.0-30003-exec-18505] - o.a.h.impl.execchain.RetryExec.execute [97] : I/O exception (java.net.SocketException) caught when processing request to {}->http://10.248.82.204:30004: 打开的文件过多
...

2019-11-26 18:10:19 - ERROR - [] - [http-apr-0.0.0.0-30003-exec-18644] - c.p.s.f.s.e.GlobalExceptionHandler.handleThrowable [233] : Could not parse multipart servlet request; nested exception is java.io.IOException: org.apache.tomcat.util.http.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. /xxx/work/Catalina/localhost/ROOT/upload_e8b7bd3f_de5f_4665_8cd4_2e5b09365f70_01821282.tmp (打开的文件过多)
org.springframework.web.multipart.MultipartException: Could not parse multipart servlet request; nested exception is java.io.IOException: org.apache.tomcat.util.http.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. /xxx/work/Catalina/localhost/ROOT/upload_e8b7bd3f_de5f_4665_8cd4_2e5b09365f70_01821282.tmp (打开的文件过多)
...

2019-11-26 18:10:23 -  WARN - [500bfc4f1d964dfa9184d5c04c1a0949] - [http-apr-0.0.0.0-30003-exec-19010] - o.s.c.n.z.f.p.SendErrorFilter.run [78] : Error during filtering
com.netflix.zuul.exception.ZuulException: Forwarding error
    at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.handleException(RibbonRoutingFilter.java:188)
    at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.forward(RibbonRoutingFilter.java:163)
    at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.run(RibbonRoutingFilter.java:111)
    at com.netflix.zuul.ZuulFilter.runFilter(ZuulFilter.java:112)
    at com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:193)
    at com.netflix.zuul.FilterProcessor.runFilters(FilterProcessor.java:157)
    at com.netflix.zuul.FilterProcessor.route(FilterProcessor.java:118)
    at com.netflix.zuul.ZuulRunner.route(ZuulRunner.java:96)
    at com.netflix.zuul.http.ZuulServlet.route(ZuulServlet.java:116)
    at com.netflix.zuul.http.ZuulServlet.service(ZuulServlet.java:81)
    at org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:157)
    at org.springframework.cloud.netflix.zuul.web.ZuulController.handleRequest(ZuulController.java:44)
    at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:103)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:130)
    at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:66)
    at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:105)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:123)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
    at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:2351)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
Caused by: com.netflix.client.ClientException: null
    at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:118)
    at org.springframework.cloud.netflix.zuul.filters.route.support.AbstractRibbonCommand.run(AbstractRibbonCommand.java:187)
    at org.springframework.cloud.netflix.zuul.filters.route.support.AbstractRibbonCommand.run(AbstractRibbonCommand.java:52)
    at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:302)
    at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:298)
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46)
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.Observable.unsafeSubscribe(Observable.java:10151)
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.Observable.unsafeSubscribe(Observable.java:10151)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    at rx.Observable.unsafeSubscribe(Observable.java:10151)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.Observable.unsafeSubscribe(Observable.java:10151)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    at rx.Observable.unsafeSubscribe(Observable.java:10151)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    at rx.Observable.unsafeSubscribe(Observable.java:10151)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.Observable.unsafeSubscribe(Observable.java:10151)
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
    at rx.Observable.unsafeSubscribe(Observable.java:10151)
    at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
    at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
    at rx.Observable.unsafeSubscribe(Observable.java:10151)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.Observable.unsafeSubscribe(Observable.java:10151)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    at rx.Observable.unsafeSubscribe(Observable.java:10151)
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.Observable.subscribe(Observable.java:10247)
    at rx.Observable.subscribe(Observable.java:10214)
    at rx.internal.operators.BlockingOperatorToFuture.toFuture(BlockingOperatorToFuture.java:51)
    at rx.observables.BlockingObservable.toFuture(BlockingObservable.java:411)
    at com.netflix.hystrix.HystrixCommand.queue(HystrixCommand.java:378)
    at com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:344)
    at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.forward(RibbonRoutingFilter.java:158)
    ... 78 common frames omitted
Caused by: java.lang.RuntimeException: java.net.SocketException: 打开的文件过多
    at rx.exceptions.Exceptions.propagate(Exceptions.java:58)
    at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:464)
    at rx.observables.BlockingObservable.single(BlockingObservable.java:341)
    at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:112)
    ... 140 common frames omitted
Caused by: java.net.SocketException: 打开的文件过多
    at java.net.Socket.createImpl(Socket.java:460)
    at java.net.Socket.getImpl(Socket.java:520)
    at java.net.Socket.setSoTimeout(Socket.java:1141)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:120)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:394)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
    at org.springframework.cloud.netflix.ribbon.apache.RibbonLoadBalancingHttpClient.execute(RibbonLoadBalancingHttpClient.java:82)
    at org.springframework.cloud.netflix.ribbon.apache.RibbonLoadBalancingHttpClient.execute(RibbonLoadBalancingHttpClient.java:42)
    at com.netflix.client.AbstractLoadBalancerAwareClient$1.call(AbstractLoadBalancerAwareClient.java:104)
    at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:303)
    at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:287)
    at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:231)
    at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:228)
    at rx.Observable.unsafeSubscribe(Observable.java:10151)
    at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.drain(OnSubscribeConcatMap.java:286)
    at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.onNext(OnSubscribeConcatMap.java:144)
    at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:185)
    at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:180)
    at rx.Observable.unsafeSubscribe(Observable.java:10151)
    at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:94)
    at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:42)
    at rx.Observable.unsafeSubscribe(Observable.java:10151)
    at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber$1.call(OperatorRetryWithPredicate.java:127)
    at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.enqueue(TrampolineScheduler.java:73)
    at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.schedule(TrampolineScheduler.java:52)
    at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:79)
    at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:45)
    at rx.internal.util.ScalarSynchronousObservable$WeakSingleProducer.request(ScalarSynchronousObservable.java:276)
    at rx.Subscriber.setProducer(Subscriber.java:209)
    at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:138)
    at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:129)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.Observable.subscribe(Observable.java:10247)
    at rx.Observable.subscribe(Observable.java:10214)
    at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:444)
    ... 142 common frames omitted
...

2019-11-26 18:10:27 -  INFO - [] - [ContainerBackgroundProcessor[StandardEngine[Catalina]]] - o.a.c.core.StandardWrapper.log [180] : Waiting for [799] instance(s) to be deallocated for Servlet [dispatcherServlet]
...

2019-11-26 18:10:29 -  INFO - [] - [ContainerBackgroundProcessor[StandardEngine[Catalina]]] - c.n.discovery.DiscoveryClient.unregister [900] : DiscoveryClient_Zuul/xxx.xxx.xx.xx:zuul:xxxx - deregister  status: 200
2019-11-26 18:10:29 -  INFO - [] - [ContainerBackgroundProcessor[StandardEngine[Catalina]]] - c.n.discovery.DiscoveryClient.shutdown [887] : Completed shut down of DiscoveryClient
...

后面继续查看日志，没有发现tomcat容器为啥主动关闭的线索。
查看了系统的句柄数，是默认的配置，1024。在稍微有点并发的情况下，很容易就超过了。

到这里，大概只能得出因为大并发下，1024的句柄数耗尽，导致抛出打开文件过多，之后，达到一定条件后，触发tomcat容器关闭，然后spring容器关闭，eureka客户端从注册中心注销。此时tomcat进程还在，端口也在，也能接收请求，但是因为找不到处理的容器，所以返回404。

Sentinel熔断未生效

Sentinel是依托spring容器生效的，因此在大流量进入tomcat后，转换二进制请求都报错的情况下，请求是无法到达sentinel的，所sentinel没有生效。

问题修复

修改文件数量，改成65565
tomcat容器出问题后，nginx能识别到异常，将请求转发到其他zuul服务器。

20191126 Zuul服务器404问题
问题出现时间：2019-11-26 星期二过程：下午18点推送了一条汽车摇号的push，之后app出现大量错误，...
zuul 404
项目结构：zuul + eureka 构建的微服务配置：服务都启动后，item下的controller可以访问...
Spring Cloud Zuul（服务网关）
Spring Cloud Zuul? zuul 是netflix开源的一个API Gateway 服务器, 本质上...
Nginx404页面http状态码返回200问题解决
问题今天配置服务器404页面的时候发现的一个问题百度上搜索nginx配置404页面很多网站都说 fastcg...
Spring Cloud微服务之网关zuul配置项详解
一，Zuul的应用场景 Zuul是Netflix基于JVM的路由器和服务器端负载均衡器。 Zuul可以通过加载动态...
0811&0812-0731-JS25
问题：阅读教案，访问apache 的课堂资源服务器出现404Step1: 云课堂模拟服务器环境准备老师自己搭建后...
404 错误___<mvc:annotation-driv
** 404**: Not Found 服务器无法找到所请求的页面。原因： 1.url路径输入有问题 2....
2018-12-04 解决nginx搭建图片服务器访问图片404
本人主要解决搭建nginx搭建图片服务器，然后访问图片404的问题 cd /usr/local/nginxvim ...
SpringCloud-5之-Zuul网关服务
使用Ribbon和SpringCloud实现客户端负载均衡-Zuul网关服务 1、介绍Zuul是服务器端的负载均衡...
UIWebView获取404、504等错误码
问题描述在使用webview时，会存在这么一个问题：如果访问服务器返回异常，比如404、504这样的错误...