美文网首页
org.apache.catalina.connector.Cl

org.apache.catalina.connector.Cl

作者: 机灵鬼鬼 | 来源:发表于2022-07-20 18:04 被阅读0次

    <meta charset="utf-8">

    1. 错误日志

    TID:N/A] 2022-07-19 21:38:31.167 DEBUG 7 --- [s_8848-aws-prod] o.s.c.e.PropertySourcesPropertyResolver  : Found key 'logging.file.path' in PropertySource 'configurationProperties' with value of type String
    [TID:N/A] 2022-07-19 21:38:31.186 DEBUG 7 --- [s_8848-aws-prod] o.s.c.e.PropertySourcesPropertyResolver  : Found key 'logging.config' in PropertySource 'bootstrapProperties-log.yaml,IDAAS_CONFIG' with value of type String
    [TID:N/A] 2022-07-19 21:38:31.186 DEBUG 7 --- [s_8848-aws-prod] o.s.c.e.PropertySourcesPropertyResolver  : Found key 'logging.file.path' in PropertySource 'bootstrapProperties-log.yaml,IDAAS_CONFIG' with value of type String
    [TID:a3db622045564c8e91dd77ec5ada1917.2267.16582412059470009] 2022-07-19 22:33:39.517 ERROR 7 --- [nio-8080-exec-5] COMMON-ERROR                             : [global exception handle finished,code:5000]
    org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
            at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353)
            at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:783)
            at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:688)
            at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:388)
            at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:366)
            at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96)
            at org.springframework.util.StreamUtils$NonClosingOutputStream.write(StreamUtils.java:287)
            at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2161)
            at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment2(UTF8JsonGenerator.java:1476)
            at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment(UTF8JsonGenerator.java:1423)
            at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:509)
            at com.fasterxml.jackson.databind.ser.std.StringSerializer.serialize(StringSerializer.java:41)
            at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)
            at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:755)
            at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
            at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
            at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
            at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
            at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)
            at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:755)
            at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
            at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)
            at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:755)
            at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
            at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
            at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
            at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1516)
            at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:1006)
            at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:346)
            at org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:104)
            at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:277)
            at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.handleReturnValue(RequestResponseBodyMethodProcessor.java:181)
            at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:82)
            at org.springframework.hateoas.server.mvc.RepresentationModelProcessorHandlerMethodReturnValueHandler.handleReturnValue(RepresentationModelProcessorHandlerMethodReturnValueHandler.java:108)
            at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:82)
            at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:123)
            at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
            at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
            at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
            at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
            at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
            at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
            at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
            at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
            at com.paraview.bum.user.biz.interfaces.config.filters.UserFilter.doFilter(UserFilter.java:24)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
            at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
            at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
            at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
            at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
            at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
            at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
            at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
            at org.apache.catalina.core.StandardHostValve.invoke$original$HjItMESD(StandardHostValve.java:143)
            at org.apache.catalina.core.StandardHostValve.invoke$original$HjItMESD$accessor$GUzLIHAF(StandardHostValve.java)
            at org.apache.catalina.core.StandardHostValve$auxiliary$qy15M84F.call(Unknown Source)
            at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
            at org.apache.catalina.core.StandardHostValve.invoke$original$8L8NHDuy(StandardHostValve.java)
            at org.apache.catalina.core.StandardHostValve.invoke$original$8L8NHDuy$accessor$imKxwdq5(StandardHostValve.java)
            at org.apache.catalina.core.StandardHostValve$auxiliary$ZgMRXHXV.call(Unknown Source)
            at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java)
            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
            at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:764)
            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
            at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
            at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
            at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
            at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707)
            at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
            at java.base/java.lang.Thread.run(Unknown Source)
    Caused by: java.io.IOException: Broken pipe
            at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
            at java.base/sun.nio.ch.SocketDispatcher.write(Unknown Source)
            at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
            at java.base/sun.nio.ch.IOUtil.write(Unknown Source)
            at java.base/sun.nio.ch.IOUtil.write(Unknown Source)
            at java.base/sun.nio.ch.SocketChannelImpl.write(Unknown Source)
            at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:135)
            at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:118)
            at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:151)
            at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1367)
            at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:766)
            at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:586)
            at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:530)
            at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:546)
            at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:112)
            at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:193)
            at org.apache.coyote.Response.doWrite(Response.java:606)
            at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:340)
            ... 87 common frames omitted
    [TID:a3db622045564c8e91dd77ec5ada1917.2267.16582412059470009] 2022-07-19 22:33:39.524 ERROR 7 --- [nio-8080-exec-5] COMMON-ERROR                             : [global exception handle finished,code:5000 , msg:系统异常 ]
    org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
            at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353)
            at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:783)
            at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:688)
            at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:388)
            at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:366)
            at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96)
            at org.springframework.util.StreamUtils$NonClosingOutputStream.write(StreamUtils.java:287)
            at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2161)
            at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment2(UTF8JsonGenerator.java:1476)
            at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment(UTF8JsonGenerator.java:1423)
            at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:509)
            at com.fasterxml.jackson.databind.ser.std.StringSerializer.serialize(StringSerializer.java:41)
            at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)
            at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:755)
            at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
            at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
            at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
            at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
            at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)
            at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:755)
            at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
            at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)
            at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:755)
            at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
            at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
            at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
            at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1516)
            at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:1006)
            at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:346)
            at org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:104)
            at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:277)
            at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.handleReturnValue(RequestResponseBodyMethodProcessor.java:181)
            at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:82)
            at org.springframework.hateoas.server.mvc.RepresentationModelProcessorHandlerMethodReturnValueHandler.handleReturnValue(RepresentationModelProcessorHandlerMethodReturnValueHandler.java:108)
            at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:82)
            at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:123)
            at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
            at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
            at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
            at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
            at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
            at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
            at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
            at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
            at com.paraview.bum.user.biz.interfaces.config.filters.UserFilter.doFilter(UserFilter.java:24)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
            at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
            at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
            at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
            at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
            at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
            at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
            at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
            at org.apache.catalina.core.StandardHostValve.invoke$original$HjItMESD(StandardHostValve.java:143)
            at org.apache.catalina.core.StandardHostValve.invoke$original$HjItMESD$accessor$GUzLIHAF(StandardHostValve.java)
            at org.apache.catalina.core.StandardHostValve$auxiliary$qy15M84F.call(Unknown Source)
            at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
            at org.apache.catalina.core.StandardHostValve.invoke$original$8L8NHDuy(StandardHostValve.java)
            at org.apache.catalina.core.StandardHostValve.invoke$original$8L8NHDuy$accessor$imKxwdq5(StandardHostValve.java)
            at org.apache.catalina.core.StandardHostValve$auxiliary$ZgMRXHXV.call(Unknown Source)
            at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java)
            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
            at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:764)
            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
            at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
            at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
            at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
            at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707)
            at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
            at java.base/java.lang.Thread.run(Unknown Source)
    Caused by: java.io.IOException: Broken pipe
            at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
            at java.base/sun.nio.ch.SocketDispatcher.write(Unknown Source)
            at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
            at java.base/sun.nio.ch.IOUtil.write(Unknown Source)
            at java.base/sun.nio.ch.IOUtil.write(Unknown Source)
            at java.base/sun.nio.ch.SocketChannelImpl.write(Unknown Source)
            at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:135)
            at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:118)
            at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:151)
            at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1367)
            at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:766)
            at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:586)
            at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:530)
            at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:546)
            at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:112)
            at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:193)
            at org.apache.coyote.Response.doWrite(Response.java:606)
            at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:340)
            ... 87 common frames omitted
    
    1. 产生影响

    数据同步功能全量同步飞书和简道云,sync调用bum的user服务查询用户和组织数据接口无返回,导致数据同步异常。

    1. 分析过程

    第一步:首先怀疑bum数据库链接不够,cpu压力过载。经过show processlist发现数据库正常,没有挂起的事务,压力正常,数据库的嫌疑排除。

    第二步:分析sync错误日志,

    apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
    

    这个日志可以看出是服务端非正常断开连接的异常,此时联想起我们的nacos注册中心的user服务是否正常注册,通过检查nacos服务列表正常注册,nacos问题排除。

    第三步:分析user日志,出现大量的这个日志打印,我们容器用的是tomcat并没有使用netty,此时怀疑是不是mq服务挂起导致的。去查看mq的监控,发现资源一切正常,mq的嫌疑排除。

    [TID:N/A] 2022-07-19 21:37:42.388 DEBUG 7 --- [-worker-ELG-1-2] o.a.s.a.d.i.g.netty.NettyClientHandler   : Window: 1048576
    

    第四步:再回到sync服务观察日志

    image.png

    发现是网络io返回数据做写操作的时候,出现了异常,再结合刚才的ClientAbortException服务断主动断开连接,说明服务端返回的数据没有写完,服务端就主动断开了连接,现在开始怀疑是不是服务断接口超时配置导致自动断开,终断了正在返回的数据。此时想起了一个可以验证此猜测的方式,就是在sync所在的服务器,直接用wget访问user的接口,但由于过不了微服务之间的校验,直接被拦截回来了,看不出耗时。

    关键点来了:skywalking工具可以做链路追踪,是否可以通过skywalking的接口链路日志看出来接口的调用耗时呢,此时,通过查询skywalking的链路信息,发现了端倪。bum服务的sync/getAllPushData接口耗时26387ms严重超出了我们在nacos中为每个服务配置的5000ms的feign超时时间。

    image.png
    1. 锁定原因

    至此,找到问题原因,临时解决方案,把user服务和sync服务的feign的超时时间改长到30000ms,最后重启服务,执行全量同步,终于解决了问题。

    feign:
      httpclient:
        enabled: true
      client:
        config:
          default: #这里就是指的所有被加载的默认FeignClient实现的服务配置都生效
            connectTimeout: 30000
            readTimeout: 30000
      hystrix: 
          enable: true
    
    1. 根因分析

    user服务的sync/getAllPushData这个接口为什么会耗时较长,需要分析具体原因。

    经过分析发现,查询待下发数据的sql,没有分页,存在上万条大数据集的查询,导致sql查询效率被拖慢。

    select count(bsd.task_id) as cnt,bsd.task_id,asti.task_name from test_bsd bsd
    
    left join test_asti asti on bsd.task_id = asti.id
    
    where bsd.tenant_id=xx1 and bsd.status='NO' group by bsd.task_id
    
    ORDER by cnt desc
    
    1. 解决方案

    从两个方面入手,(1)首先看下查询语句是否走了索引,(2)其次就是对查询语句进行优化设计,分页查询,以避免单页数据过大导致的查询效率降低的风险。

    看下当前语句的索引执行情况首先我们观察下执行计划

    explain select * from test_bsd  where tenant_id=‘xx1’ and task_id=‘tt1’ and status='NO'
    

    执行结果如下:

    possible_keys:理论用到的key,key:实际查询用到的key,都是idx_task_id。

    经排查执行计划的type字段返回的是ref,这个意思代表:查询条件使用了索引而且不是主键和union唯一索引,意思是说,虽然查询语句使用了索引,但该索引列的值并不唯一,有重复。扫描到第一个匹配的值后,还会继续小范围搜索,但已经比全表效率高很多了。test_bsd 表索引只有一个task_id,没有把作为条件的tenant_id加上索引。

    代码分页查询优化:

    此处省略200字
    

    相关文章

      网友评论

          本文标题:org.apache.catalina.connector.Cl

          本文链接:https://www.haomeiwen.com/subject/tthyirtx.html