神奇的端口变化

作者: 程序员札记 | 来源:发表于2024-02-25 22:40 被阅读0次

一个app 启动之后，当访问8083端口的URL发现出现redirect 死循环。我们确实在servlet的filter（a filter）里有个逻辑，如果访问的端口不是8083 的话，就回redirect 到8083 （保证这个URL的访问都是内部的）. 所以出现redirect死循环，肯定是端口发生了变化。进来8083，到达filter 时候变成了别的端口。
所以当时第一个猜想就是是不是有另外一个filter （b filter）把8083 redirect 到别的端口，然后这个端口的request 被a filter 重新redirect 到8083. 这个两个filter 就形成了死循环。

我们的思路是先要看这个8083 的request被redirect 到哪个端口，通过加入debug 信息发现端口是80. 由于这个app 用了shiro， Shiro过滤器里有个PortFilter，PortFilter 正好强制把端口转向80。所以我们开始查配置，发现8083请求request 配置了跳过Shiro ，所以跟Shiro无关。

通过下面代码查过了所有的filter 发现没有filter 在干redirect 的事情。

import org.springframework.boot.web.servlet.FilterRegistrationBean;
import org.springframework.context.ApplicationContext;
import org.springframework.stereotype.Component;
import javax.annotation.PostConstruct;
import java.util.Map;
@Component
public class FilterLister {
    private final ApplicationContext applicationContext;
    public FilterLister(ApplicationContext applicationContext) {
        this.applicationContext = applicationContext;
    }
    @PostConstruct
    public void listAllFilters() {
        Map<String, FilterRegistrationBean> filters = applicationContext.getBeansOfType(FilterRegistrationBean.class);
        for (Map.Entry<String, FilterRegistrationBean> entry : filters.entrySet()) {
            System.out.println("Filter name: " + entry.getKey());
            System.out.println("Filter instance: " + entry.getValue().getFilter());
        }
    }
}

那么还有什么情况能改变request 端口呢，我们不得不在org.apache.coyote.Request 里的 setServerPort 方法打印 stack trace, 因为只有这里会 setServerPort。通过这个方式发现了
org.apache.catalina.valves.RemoteIpValve。

同时在配置中也发现用户enable 了这个值

server.tomcat.remoteip.remote-ip-header=x-forwarded-for
server.tomcat.remoteip.protocol-header=x-forwarded-proto

这些配置到底有什么关联呢？首先看在RemoteIpValve是怎么进入到tomcat request pipeline里去的。

配置RemoteIpValve

在创建tomcat容器时的ServletWebServerApplicationContext#createWebServer。

ServletWebServerFactory factory = getWebServerFactory();
this.webServer = factory.getWebServer(getSelfInitializer());

getWebServerFactory()方法会从beanFactory中获取ServletWebServerFactory对象。构造这个bean的过程中，会执行提前注册好的WebServerFactoryCustomizerBeanPostProcessor。
WebServerFactoryCustomizerBeanPostProcessor.postProcessBeforeInitialization会获取所有实现WebServerFactoryCustomizer的bean，并执行他们的customize()方法。
这其中就包括TomcatWebServerFactoryCustomizer，这里增加了tomcat专用的一些features.

Customization for Tomcat-specific features common for both Servlet and Reactive servers.

其中的customizeRemoteIpValve()方法会根据条件向tomcat中加入RemoteIpValve

  private void customizeRemoteIpValve(ConfigurableTomcatWebServerFactory factory) {
      Remoteip remoteIpProperties = this.serverProperties.getTomcat().getRemoteip();
      String protocolHeader = remoteIpProperties.getProtocolHeader();
      String remoteIpHeader = remoteIpProperties.getRemoteIpHeader();
      // For back compatibility the valve is also enabled if protocol-header is set
      if (StringUtils.hasText(protocolHeader) || StringUtils.hasText(remoteIpHeader)
              || getOrDeduceUseForwardHeaders()) {
          RemoteIpValve valve = new RemoteIpValve();
          valve.setProtocolHeader(StringUtils.hasLength(protocolHeader) ? protocolHeader : "X-Forwarded-Proto");
          if (StringUtils.hasLength(remoteIpHeader)) {
              valve.setRemoteIpHeader(remoteIpHeader);
          }
          // The internal proxies default to a list of "safe" internal IP addresses
          valve.setInternalProxies(remoteIpProperties.getInternalProxies());
          try {
              valve.setHostHeader(remoteIpProperties.getHostHeader());
          }
          catch (NoSuchMethodError ex) {
              // Avoid failure with war deployments to Tomcat 8.5 before 8.5.44 and
              // Tomcat 9 before 9.0.23
          }
          valve.setPortHeader(remoteIpProperties.getPortHeader());
          valve.setProtocolHeaderHttpsValue(remoteIpProperties.getProtocolHeaderHttpsValue());
          // ... so it's safe to add this valve by default.
          factory.addEngineValves(valve);
      }
  }

我本地用自己Sample 的项目，加上那两个配置后，debug 走到这，分别是protocolHeader : x-forwarded-proto 和remoteIpHeader: x-forwarded-for ，这就体现application.proprties的配置作用。配了任意一个都会激活RemoteIpValve，被加入到了tocat pipeline 中factory.addEngineValves(valve); 。从90.3 Enable HTTPS When Running behind a Proxy Server ，https://docs.spring.io/spring-boot/docs/2.1.x/reference/html/howto-security.html 也证明了我们的判断。

RemoteIpValve 的作用

tomcat会将原始的http报文封装成org.apache.catalina.connector.Request，我们可以从中获取到remoteAddr和http header。
具体细节可以参考： https://www.cnblogs.com/zhongchang/articles/10345333.html 以及https://blog.csdn.net/weixin_40562288/article/details/96131975

我们这个case 就是在enable RemoteIpValve，如果request header 里有x-forwarded-proto 就会导致severport 被改写。

if (protocolHeader != null) {
                String protocolHeaderValue = request.getHeader(protocolHeader);
                if (protocolHeaderValue == null) {
                    // Don't modify the secure, scheme and serverPort attributes
                    // of the request
                } else if (isForwardedProtoHeaderValueSecure(protocolHeaderValue)) {
                    request.setSecure(true);
                    request.getCoyoteRequest().scheme().setString("https");
                    setPorts(request, httpsServerPort);
                } else {
                    request.setSecure(false);
                    request.getCoyoteRequest().scheme().setString("http");
                    setPorts(request, httpServerPort);
                }
            }

为啥我们的会多出来一个x-forwarded-proto ，就涉及到另外一个问题。我们app 是在istio 的环境中。X-Forwarded-Proto header set to http instead of https on subsequent request https://github.com/istio/istio/issues/36612

这就完全解释了为啥端口会被设置成80，导致8083 a filter 无限发生作用。

总结

如果让端口发生变化，要有两个条件
1 要配置tomcat，让RemoteIpValve配置到pipeline 中去

server.tomcat.remoteip.remote-ip-header=x-forwarded-for
server.tomcat.remoteip.protocol-header=x-forwarded-proto

要在header 传入x-forwarded-proto 或者x-forwarded-for 的值

两者缺一不可。我们的app 正好在code 配置了RemoteIpValve，在istio 里加了x-forwarded-proto ，所以激活了RemoteIpValve set request port。

扩展

Tomcat 改port 的地方一共有三处，都在Http11Processor

server.tomcat.remoteip.remote-ip-header=x-forwarded-for
server.tomcat.remoteip.protocol-header=x-forwarded-proto

这两行配置，访问http://abc:8083时，内存中 request 端口的变化

在prepareRequest， prepareRequest(Http11Processor.java:784)，根据 URL 里面的 "冒号+port", 在 parseHost 去设置从 old ServerPort = -1 变成 newServerPort = 8083

java.base/java.lang.Thread.getStackTrace(Unknown Source)
org.apache.coyote.Request.setServerPort(Request.java:345)
org.apache.coyote.AbstractProcessor.parseHost(AbstractProcessor.java:311)
org.apache.coyote.http11.Http11Processor.prepareRequest(Http11Processor.java:784)
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367)
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928)
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794)
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
java.base/java.lang.Thread.run(Unknown Source)

在(Http11Processor.java:390)，CoyoteAdapter.serviceservicepipeline 中如果有 org.apache.catalina.valves.RemoteIpValve，它又会改port
old ServerPort = 8083
newServerPort = 80

java.base/java.lang.Thread.getStackTrace(Unknown Source)
org.apache.coyote.Request.setServerPort(Request.java:345)
org.apache.coyote.AbstractProcessor.parseHost(AbstractProcessor.java:311)
org.apache.coyote.http11.Http11Processor.prepareRequest(Http11Processor.java:784)
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367)
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928)
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794)
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
java.base/java.lang.Thread.run(Unknown Source)

postParseRequest(CoyoteAdapter.java:585) 在前面两个都不起作用的时候来设置port

org.apache.coyote.Request.setServerPort(Request.java:345) 
org.apache.catalina.connector.CoyoteAdapter.postParseRequest(CoyoteAdapter.java:585) 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:337) 
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390) 
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) 
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928) 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794) 
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) 
org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) 
org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) 
java.base/java.lang.Thread.run(Unknown Source)

例如访问 api: https://tmsapp-staging-2-tmsappcont.istio-staging.svc.130.tess.io/instance-access/10.42.231.252/api/v1/internal/get-request-info?ab=cd时， prepareRequest(Http11Processor.java:784)里面不会 setServerPort，因为这个 URL 里面不带 "冒号+port"，它里面在 parseHost 找不到端口去设置随后在 postParseRequest(CoyoteAdapter.java:585) 时，因为 req.getServerPort() 还是 -1，所以会基于 scheme 去改port。

神奇的端口变化

配置RemoteIpValve

RemoteIpValve 的作用

总结

扩展

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读