美文网首页graphql-java 手册
云原生现场分析 Part4 —— eBPF 跟踪 Istio/E

云原生现场分析 Part4 —— eBPF 跟踪 Istio/E

作者: MarkZhu | 来源:发表于2022-04-07 08:38 被阅读0次
    image.png

    注,原文来自 https://blog.mygraphql.com/zh/posts/low-tec/trace/trace-istio/trace-istio-part4/ 。如你看到的转载图片不清,请回到原文。

    为何要了解 upstream/downstream 事件驱动协作下的 HTTP 反向代理流程

    想不到还真坚持写到 Part4 了。不管你信不信,每一 part 的“为何”一节,是我觉得最难写的 :) 。如果你是第一次读这个系列,不用担心,每一节都是比较独立的。

    不出意外的话,你之前看到的 Envoy 简介是这样描述 Envoy 的特性的:

    • C++ 编写,原生底层,没有 GC stop the world,所以性能优异
    • 异步事件驱动,多路复用,完美解决 C10k problem
    • 由于单线程负责多连接,在大连接量时,减少了大量线程的内存开销,和 CPU 上下文切换开销

    这些描述,当然有其合理性。但很多远看很美丽的事物,在微距放大后,可能有很多有意思,有价值的东西。我相信只要分析得足够细致,总可能针对我们实际运行环境和流量特性对作一些有意义的优化。可能只是修改一个 Envoy / Kernel 的配置,也可能是修改一行 Envoy 的代码。或你的应用的流程相关行为,如每次写 socket 的 buffer 数据包的大小。

    而这些,都需要建立在了解实现细节的基础上。除非觉得运气或经验值特别好,可以瞎猜。

    逆向工程与云原生现场分析 系列介绍

    开始前先做个预告,逆向工程与云原生现场分析 系列(将)包括:

    在系列中,我将演示如何让 bpftrace "读" 懂运行期的由 C++ 11 编写成的 envoy 进程中的对象数据。
    为免吓跑人,还是老套路,多图少代码,不过有的图有点点复杂。程序员大叔开始讲故事了。🚜

    HTTP 反向代理的总流程

    整体看,Socket 事件驱动的 HTTP 反向代理总流程如下:


    image.png

    图中看出,有4种事件驱动了整个流程。后面几节会逐个分析。

    为免一下子进入各个步骤细节而让人迷途,现在先看看所有步骤的总流程:

    image.png

    图:Istio/Envoy 模块协作总图

    Downstream Read Request 模块协作

    image.png

    图:Downstream Read-Ready 模块协作

    大概说明一下流程:

    1. upstream socket 可读回调
    2. Http::ConnectionManagerImpl 读取 socket,增量放入 Http1::ConnectionImpl
    3. Http1::ConnectionImpl 调用 nghttp2 增量解释 HTTP 请求
    4. 如果 nghttp2 认为已经 完整读取了 HTTP Request 请求,则调用 Http::ServerConnection::onMessageCompleteBase()
    5. Http::ServerConnection::onMessageCompleteBase() 首先 停止 downstream ReadReady 监听
    6. Http::ServerConnection::onMessageCompleteBase() 调用 Http::FilterManager ,发起 http filter chain 的 decodeHeaders 迭代流程
    7. 一般,http filter chain 的最后一个 http filter 是 Router::FilterRouter::Filter::decodeHeaders() 被调用
    8. Router::Filter::decodeHeaders() 的逻辑就见下图了。
    image.png

    图:Downstream Request Router 模块协作

    大概说明一下流程:

    1. Router::FilterRouter::Filter::decodeHeaders() 被调用
    2. 根据配置的 Router 规则,匹配到 Cluster
    3. 如 Cluster 连接池对象不存在,则新建
    4. 新建 Envoy::Router::UpstreamRequest 对象。
    5. 调用 Envoy::Router::UpstreamRequest::encodeHeaders(bool end_stream) , encode HTTP header
    6. 经过一系列的负载均衡算法,匹配到 upstream 的 host(endpoint)
    7. 发现到选定的 upstream host 的连接不足,则:
      1. 打开一新的 socket fd(未连接)
      2. 注册 upstream socket FD 的 WriteReady / Connected 事件。 准备在事件回调时写 upstream request
      3. 用 socket fd 发起到 upstream host 的异步连接请求
    8. 关联 downstream 与 upstream fd

    Upstream Write Request 模块协作

    [图片上传失败...(image-5b18c8-1649291572197)]
    大概说明一下流程:

    1. upstream socket write ready 回调
    2. 发现是连接成功回调,关联 upstream socket 到 ConnectionPool::ActiveClient
    3. upstream socket write ready 回调
    4. 发现是连接可写,写入 upstream HTTP request

    Upstream Read Response 模块协作

    image.png

    Downstream Write Response 模块协作

    image.png

    BPF 脚本输出

    以上图片,不是只读源码就得出的。是需要通过 bpf 埋点观察:

    1. 记录 downstream FD(socket 的 file descriptor),可以认为是进程中的 socket id
    2. 埋点进程级和 kernel 级的相关函数入口和出口,观察其出入参与 stack
    3. 关联起 downstream FD 与 upstream FD

    其中当然有很多细节,不过我不打算一个个讲了,实在有需要用,想深入了解的同学可以联系我探讨。下一节的 bpf 代码是最好的细节。

    https://github.com/labilezhu/pub-diy/blob/main/low-tec/trace/trace-istio/trace-istio-part4/trace-envoy-filter-router.3.log

    #### Downstream Read Request: accept downstream socket, 可见 downstream socket 的 FD=41
    
    OS handshaked TCP:
    01:22:11 4215   wrk:worker_0   172.30.207.129                          58708 172.21.206.232                          15006 0/4096
    sys_exit_accept4 fd=41
    
            accept4+96
            Envoy::Network::IoSocketHandleImpl::accept(sockaddr*, unsigned int*)+82
            Envoy::Network::TcpListenerImpl::onSocketEvent(short)+216
            std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
            Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
            0x7fffffffe000
            event_base_loop+1953
            Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
            Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
            start_thread+217
     
    
    ***** elapsed=1621361178: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZNSt3__110__function6__funcIZN5Envoy10Extensions15ListenerFilters12TlsInspector25TlsInspectorConfigFactory36createListenerFilterFactoryFromProtoERKN6google8protobuf7MessageERKNS_10shared_ptrINS2_7Network21ListenerFilterMatcherEEERNS2_6Server13Configuration22ListenerFactoryContextEEUlRNSD_21ListenerFilterManagerEE_NS_9allocatorISO_EEFvSN_EEclESN_
    
    ***** elapsed=1621785490: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN5Envoy10Extensions15ListenerFilters12TlsInspector25TlsInspectorConfigFactory36createListenerFilterFactoryFromProtoERKN6google8protobuf7MessageERKNS_10shared_ptrINS3_7Network21ListenerFilterMatcherEEERNS3_6Server13Configuration22ListenerFactoryContextEEUlRNSE_21ListenerFilterManagerEE_SO_EEEvDpOT_
    
    ***** elapsed=1630898092: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterC1ENSt3__110shared_ptrINS2_6ConfigEEE
    
    ***** elapsed=1630908037: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterC2ENSt3__110shared_ptrINS2_6ConfigEEE
    
    ***** elapsed=1631036930: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter8onAcceptERNS_7Network23ListenerFilterCallbacksE
    
    ***** elapsed=1631045047: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter6onReadEv
    
    ***** elapsed=1631070582: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_recvfrom, fd=41, ret=89
    
            recv+108
    
    
    ***** elapsed=1631085065: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter16parseClientHelloEPKvm
    
    #### Downstream Read Request: 选择 network filter chain
    ***** elapsed=1631145371: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterD0Ev
    comm:wrk:worker_0,tid:5327: Got setFilterChainName=0.0.0.0_8080, lenght=12
    
            Envoy::StreamInfo::StreamInfoImpl::setFilterChainName(absl::string_view)+0
            Envoy::Server::ActiveTcpSocket::newConnection()+377
            Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
    
    #### Downstream Read Request: 监听 upstream FD 事件
    ***** elapsed=1631199672: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=41, events=0x80000005
    EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)
    EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
    EPOLL_CTL_ADD/MOD EdgeTrigger
    
            epoll_ctl+14
            epoll_nochangelist_add+54
            evmap_io_add_+421
            event_add_nolock_+603
            event_add+54
            Envoy::Event::FileEventImpl::FileEventImpl(Envoy::Event::DispatcherImpl&, int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+362
            Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+284
            Envoy::Network::IoSocketHandleImpl::initializeFileEvent(Envoy::Event::Dispatcher&, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+126
            Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)+1026
            Envoy::Network::ServerConnectionImpl::ServerConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)+107
            Envoy::Event::DispatcherImpl::createServerConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&)+70
            Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+307
            Envoy::Server::ActiveTcpSocket::newConnection()+377
            Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
            Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
            Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
            std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
            Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
            0x7fffffffe000
            event_base_loop+1953
            Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
            Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
            start_thread+217
    
    comm:wrk:worker_0    : setsockopt: level=6, fd=41, optname=1, optval=1, optlen=4. 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl9addFilterENSt3__110shared_ptrINS0_6FilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41 
    
            Envoy::Network::FilterManagerImpl::addFilter(std::__1::shared_ptr<Envoy::Network::Filter>)+0
            std::__1::__function::__func<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0, std::__1::allocator<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0>, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+93
            Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector<std::__1::function<void (Envoy::Network::FilterManager&)>, std::__1::allocator<std::__1::function<void (Envoy::Network::FilterManager&)> > > const&)+50
            Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774
            Envoy::Server::ActiveTcpSocket::newConnection()+377
            Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
            Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
            Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
            std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
            Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
            0x7fffffffe000
            event_base_loop+1953
            Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
            Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
            start_thread+217
    
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41 
    
            Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr<Envoy::Network::ReadFilter>)+0
            virtual thunk to Envoy::Network::ConnectionImpl::addFilter(std::__1::shared_ptr<Envoy::Network::Filter>)+62
            std::__1::__function::__func<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0, std::__1::allocator<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0>, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+93
            Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector<std::__1::function<void (Envoy::Network::FilterManager&)>, std::__1::allocator<std::__1::function<void (Envoy::Network::FilterManager&)> > > const&)+50
            Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774
            Envoy::Server::ActiveTcpSocket::newConnection()+377
            Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
            Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
            Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
            std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
            Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
            0x7fffffffe000
            event_base_loop+1953
            Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
            Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
            start_thread+217
    
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41 
    
            Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr<Envoy::Network::ReadFilter>)+0
            std::__1::__function::__func<Envoy::Extensions::NetworkFilters::HttpConnectionManager::HttpConnectionManagerFilterConfigFactory::createFilterFactoryFromProtoTyped(envoy::extensions::filters::network::http_connection_manager::v3::HttpConnectionManager const&, Envoy::Server::Configuration::FactoryContext&)::$_7, std::__1::allocator<Envoy::Extensions::NetworkFilters::HttpConnectionManager::HttpConnectionManagerFilterConfigFactory::createFilterFactoryFromProtoTyped(envoy::extensions::filters::network::http_connection_manager::v3::HttpConnectionManager const&, Envoy::Server::Configuration::FactoryContext&)::$_7>, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+353
            Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector<std::__1::function<void (Envoy::Network::FilterManager&)>, std::__1::allocator<std::__1::function<void (Envoy::Network::FilterManager&)> > > const&)+50
            Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774
            Envoy::Server::ActiveTcpSocket::newConnection()+377
            Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
            Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
            Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
            std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
            Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
            0x7fffffffe000
            event_base_loop+1953
            Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
            Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
            start_thread+217
    
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41 
    
            Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+0
            Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774
            Envoy::Server::ActiveTcpSocket::newConnection()+377
            Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
            Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
            Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
            std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
            Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
            0x7fffffffe000
            event_base_loop+1953
            Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
            Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
            start_thread+217
    
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    
    ***** elapsed=1631443267: tid=5327,comm=wrk:worker_0: END:EventFired
    
    ******* WAKE-ROUND:END Summary *******
    ***** elapsed=1631456738: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=31393726, tid2epollNrFdReady=1
    *** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=100 
    ***************************
    
    #### Downstream Read Request: 读 HTTP Request
    ***** elapsed=1631475016: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
    FileEventImpl*=0x559f98e26ee0, fd=41, events=0x26
    libevent: EV_READ
    libevent: EV_WRITE
    libevent: EV_ET
    
    
    ***** elapsed=1631513821: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=41, ret=89
    
            readv+77
            Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
            Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
            Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
            Envoy::Network::ConnectionImpl::onReadReady()+753
            Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
            std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
            Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
            0x7fffffffe000
            event_base_loop+1953
            Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
            Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
            start_thread+217
    
    
    ***** elapsed=1631527775: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=41, ret=-11
    
            readv+77
            Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
            Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
            Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
            Envoy::Network::ConnectionImpl::onReadReady()+753
            Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
            std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
            Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
            0x7fffffffe000
            event_base_loop+1953
            Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
            Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
            start_thread+217
    
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl6onReadEv,FilterManagerImpl.this=0x559f98d38758,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41 
    
            Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+0
            Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
            std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
            Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
            0x7fffffffe000
            event_base_loop+1953
            Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
            Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
            start_thread+217
    
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98c07200,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98c07200,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    ***** elapsed=1631671079: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_DEL, fd=41, events=0x80000005
    
    #### Downstream Read Request: downstream 的 HTTP Reuest 已经完全读取(onMessageComplete())。停止监听 downstream 的 ReadReady 事件
    ***** elapsed=1631678652: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=41, events=0x80002004
    EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
    EPOLL_CTL_ADD/MOD EdgeTrigger
    
            epoll_ctl+14
            epoll_nochangelist_add+54
            evmap_io_add_+421
            event_add_nolock_+603
            event_add+54
            Envoy::Network::ConnectionImpl::readDisable(bool)+1077
            Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+86
            Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
            Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
            http_parser_execute+7959
            Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
            Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
            Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
            virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
            Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
            Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
            Envoy::Network::ConnectionImpl::onReadReady()+1622
            Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
            std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
            Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
            0x7fffffffe000
            event_base_loop+1953
            Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
            Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
            start_thread+217
    
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    
    #### Downstream Read Request: 监听新建立的 upstream socket 的 FD=44 , 这里 socket 还处于未连接状态。现在监听 upstream socket 的 所有事件。以备后面异步连接成功时收到回调
    ***** elapsed=1632388633: tid=5327,comm=wrk:worker_0: register upstream event trigger:sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=44, events=0x80000005
    EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)
    EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
    EPOLL_CTL_ADD/MOD EdgeTrigger
    
            epoll_ctl+14
            epoll_nochangelist_add+54
            evmap_io_add_+421
            event_add_nolock_+603
            event_add+54
            Envoy::Event::FileEventImpl::FileEventImpl(Envoy::Event::DispatcherImpl&, int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+362
            Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+284
            Envoy::Network::IoSocketHandleImpl::initializeFileEvent(Envoy::Event::Dispatcher&, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+126
            Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)+1026
            Envoy::Network::ClientConnectionImpl::ClientConnectionImpl(Envoy::Event::Dispatcher&, std::__1::shared_ptr<Envoy::Network::Address::Instance const> const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const> const&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&)+283
            Envoy::Event::DispatcherImpl::createClientConnection(std::__1::shared_ptr<Envoy::Network::Address::Instance const>, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&)+75
            Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterInfo const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const> const&, Envoy::Network::TransportSocketFactory&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const>)+561
            Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const>) const+126
            non-virtual thunk to Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const>) const+21
            Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+112
            Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69
            std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr<Envoy::ConnectionPool::ActiveClient, std::__1::default_delete<Envoy::ConnectionPool::ActiveClient> > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40
            Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35
            Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233
            Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001
            non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47
            Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70
            Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120
            Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334
            Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::RequestHeaderMap, std::__1::default_delete<Envoy::Http::RequestHeaderMap> >&&, bool)+6139
            Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279
            Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
            Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
            http_parser_execute+7959
            Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
            Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
            Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
            virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
            Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
            Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
            Envoy::Network::ConnectionImpl::onReadReady()+1622
            Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
            std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
            Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
            0x7fffffffe000
            event_base_loop+1953
            Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
            Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
            start_thread+217
    
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98f08478,fd=41 
    
            Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr<Envoy::Network::ReadFilter>)+0
            Envoy::Http::CodecClient::CodecClient(Envoy::Http::CodecType, std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> >&&, std::__1::shared_ptr<Envoy::Upstream::HostDescription const>, Envoy::Event::Dispatcher&)+357
            Envoy::Http::CodecClientProd::CodecClientProd(Envoy::Http::CodecType, std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> >&&, std::__1::shared_ptr<Envoy::Upstream::HostDescription const>, Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&)+70
            std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1>, std::__1::unique_ptr<Envoy::Http::CodecClient, std::__1::default_delete<Envoy::Http::CodecClient> > (Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*&&)+94
            Envoy::Http::FixedHttpConnPoolImpl::createCodecClient(Envoy::Upstream::Host::CreateConnectionData&)+35
            Envoy::Http::ActiveClient::initialize(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase&)+113
            Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+171
            Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69
            std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr<Envoy::ConnectionPool::ActiveClient, std::__1::default_delete<Envoy::ConnectionPool::ActiveClient> > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40
            Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35
            Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233
            Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001
            non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47
            Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70
            Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120
            Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334
            Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::RequestHeaderMap, std::__1::default_delete<Envoy::Http::RequestHeaderMap> >&&, bool)+6139
            Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279
            Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
            Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
            http_parser_execute+7959
            Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
            Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
            Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
            virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
            Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
            Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
            Envoy::Network::ConnectionImpl::onReadReady()+1622
            Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
            std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
            Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
            0x7fffffffe000
            event_base_loop+1953
            Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
            Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
            start_thread+217
    
    comm:wrk:worker_0    : setsockopt: level=6, fd=44, optname=1, optval=1, optlen=4. 
    
    #### Downstream Read Request: 发现异步 upstream 连接
    connecting to 8080:
    01:23:36 4215     wrk:worker_0     127.0.0.6                               36383  172.21.206.232                          8080  
    
            connect+75
            Envoy::Network::IoSocketHandleImpl::connect(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+104
            Envoy::Network::SocketImpl::connect(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+64
            Envoy::Network::ClientConnectionImpl::connect()+650
            Envoy::Http::CodecClient::connect()+543
            Envoy::Http::CodecClientProd::CodecClientProd(Envoy::Http::CodecType, std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> >&&, std::__1::shared_ptr<Envoy::Upstream::HostDescription const>, Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&)+694
            std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1>, std::__1::unique_ptr<Envoy::Http::CodecClient, std::__1::default_delete<Envoy::Http::CodecClient> > (Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*&&)+94
            Envoy::Http::FixedHttpConnPoolImpl::createCodecClient(Envoy::Upstream::Host::CreateConnectionData&)+35
            Envoy::Http::ActiveClient::initialize(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase&)+113
            Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+171
            Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69
            std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr<Envoy::ConnectionPool::ActiveClient, std::__1::default_delete<Envoy::ConnectionPool::ActiveClient> > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40
            Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35
            Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233
            Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001
            non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47
            Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70
            Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120
            Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334
            Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::RequestHeaderMap, std::__1::default_delete<Envoy::Http::RequestHeaderMap> >&&, bool)+6139
            Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279
            Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
            Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
            http_parser_execute+7959
            Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
            Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
            Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
            virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
            Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
            Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
            Envoy::Network::ConnectionImpl::onReadReady()+1622
            Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
            std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
            Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
            0x7fffffffe000
            event_base_loop+1953
            Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
            Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
            start_thread+217
    
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
    
    ***** elapsed=1632608064: tid=5327,comm=wrk:worker_0: END:EventFired
    
    ******* WAKE-ROUND:END Summary *******
    ***** elapsed=1632618881: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=1145219, tid2epollNrFdReady=1
    *** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=92 
    ***************************
    
    
    ***** elapsed=1632635526: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
    FileEventImpl*=0x559f98e26ee0, fd=41, events=0x24
    libevent: EV_WRITE
    libevent: EV_ET
    
    
    ***** elapsed=1632654517: tid=5327,comm=wrk:worker_0: END:EventFired
    
    #### Upstream Write Request: 异步 upstream 连接成功,回调。
    ***** elapsed=1632659321: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
    FileEventImpl*=0x559f98eb6540, fd=44, events=0x24
    libevent: EV_WRITE
    libevent: EV_ET
    
    comm:wrk:worker_0,tid:5327: ConnPoolImplBase*attachStreamToClient: uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy14ConnectionPool16ConnPoolImplBase20attachStreamToClientERNS0_12ActiveClientERNS0_13AttachContextE 
    
            Envoy::ConnectionPool::ConnPoolImplBase::attachStreamToClient(Envoy::ConnectionPool::ActiveClient&, Envoy::ConnectionPool::AttachContext&)+0
            Envoy::ConnectionPool::ConnPoolImplBase::onConnectionEvent(Envoy::ConnectionPool::ActiveClient&, absl::string_view, Envoy::Network::ConnectionEvent)+497
            Envoy::Network::ConnectionImplBase::raiseConnectionEvent(Envoy::Network::ConnectionEvent)+59
            Envoy::Network::ConnectionImpl::raiseEvent(Envoy::Network::ConnectionEvent)+511
            Envoy::Network::ConnectionImpl::onWriteReady()+1648
            Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+818
            std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
            Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
            0x7fffffffe000
            event_base_loop+1953
            Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
            Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
            start_thread+217
    
    #### Upstream Write Request: 向 upstream socket 写 HTTP Request
    ***** elapsed=1632741863: tid=5327,comm=wrk:worker_0: socket_write, probe=tracepoint:syscalls:sys_exit_writev, fd=44, ret=262
    
            writev+77
            Envoy::Network::IoSocketHandleImpl::writev(Envoy::Buffer::RawSlice const*, unsigned long)+263
            Envoy::Network::IoSocketHandleImpl::write(Envoy::Buffer::Instance&)+107
            Envoy::Network::RawBufferSocket::doWrite(Envoy::Buffer::Instance&, bool)+121
            Envoy::Network::ConnectionImpl::onWriteReady()+1876
            Envoy::Network::ConnectionImpl::raiseEvent(Envoy::Network::ConnectionEvent)+529
            Envoy::Network::ConnectionImpl::onWriteReady()+1648
            Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+818
            std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
            Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
            0x7fffffffe000
            event_base_loop+1953
            Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
            Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
            start_thread+217
    
    
    ***** elapsed=1632756069: tid=5327,comm=wrk:worker_0: END:EventFired
    
    ******* WAKE-ROUND:END Summary *******
    ***** elapsed=1632764448: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=129648, tid2epollNrFdReady=2
    *** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=60 
    ***************************
    
    #### Upstream Read Response: upstream socket ReadReady 回调
    ***** elapsed=1637821708: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
    FileEventImpl*=0x559f98eb6540, fd=44, events=0x26
    libevent: EV_READ
    libevent: EV_WRITE
    libevent: EV_ET
    
    #### Upstream Read Response: 读 upstream HTTP Response
    ***** elapsed=1637898855: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=44, ret=4511
    
            readv+77
            Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
            Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
            Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
            Envoy::Network::ConnectionImpl::onReadReady()+753
            Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
            std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
            Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
            0x7fffffffe000
            event_base_loop+1953
            Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
            Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
            start_thread+217
    
    
    ***** elapsed=1637915998: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=44, ret=-11
    
            readv+77
            Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
            Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
            Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
            Envoy::Network::ConnectionImpl::onReadReady()+753
            Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
            std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
            Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
            0x7fffffffe000
            event_base_loop+1953
            Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
            Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
            start_thread+217
    
    ***** elapsed=1638220184: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_DEL, fd=41, events=0x80002004
    
    #### Upstream Read Response: 已经完整读取 upstream HTTP Response,开始监听 downstream 的 WriteReady 事件,准备写 downstream HTTP Response
    ***** elapsed=1638231319: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=41, events=0x80000005
    EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)
    EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
    EPOLL_CTL_ADD/MOD EdgeTrigger
    
            epoll_ctl+14
            epoll_nochangelist_add+54
            evmap_io_add_+421
            event_add_nolock_+603
            event_add+54
            Envoy::Network::ConnectionImpl::readDisable(bool)+938
            Envoy::Http::Http1::StreamEncoderImpl::~StreamEncoderImpl()+112
            non-virtual thunk to Envoy::Http::Http1::ServerConnectionImpl::onEncodeComplete()+54
            Envoy::Http::Http1::StreamEncoderImpl::endEncode()+166
            Envoy::Http::Http1::StreamEncoderImpl::encodeData(Envoy::Buffer::Instance&, bool)+340
            Envoy::Http::ConnectionManagerImpl::ActiveStream::encodeData(Envoy::Buffer::Instance&, bool)+679
            Envoy::Http::FilterManager::encodeData(Envoy::Http::ActiveStreamEncoderFilter*, Envoy::Buffer::Instance&, bool, Envoy::Http::FilterManager::FilterIterationStartState)+2138
            Envoy::Router::UpstreamRequest::decodeData(Envoy::Buffer::Instance&, bool)+230
            Envoy::Http::ResponseDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool)+59
            Envoy::Http::ResponseDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool)+59
            Envoy::Http::Http1::ClientConnectionImpl::onMessageCompleteBase()+619
            Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
            Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
            http_parser_execute+7705
            Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
            Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
            Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
            Envoy::Http::Http1::ClientConnectionImpl::dispatch(Envoy::Buffer::Instance&)+29
            Envoy::Http::CodecClient::onData(Envoy::Buffer::Instance&)+48
            Envoy::Http::CodecClient::CodecReadFilter::onData(Envoy::Buffer::Instance&, bool)+21
            Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
            Envoy::Network::ConnectionImpl::onReadReady()+1622
            Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
            std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
            Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
    
    
    ***** elapsed=1638898463: tid=5327,comm=wrk:worker_0: END:EventFired
    
    ******* WAKE-ROUND:END Summary *******
    ***** elapsed=1638934709: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=1123611, tid2epollNrFdReady=1
    *** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=56 
    ***************************
    
    
    ***** elapsed=1638955421: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
    FileEventImpl*=0x559f98eb6540, fd=44, events=0x24
    libevent: EV_WRITE
    libevent: EV_ET
    
    
    ***** elapsed=1638975366: tid=5327,comm=wrk:worker_0: END:EventFired
    
    #### Downstream Write Response: downstream 的 WriteReady 事件回调。
    ***** elapsed=1638980295: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
    FileEventImpl*=0x559f98e26ee0, fd=41, events=0x24
    libevent: EV_WRITE
    libevent: EV_ET
    
    #### Downstream Write Response: 写 downstream HTTP Response
    ***** elapsed=1639253752: tid=5327,comm=wrk:worker_0: socket_write, probe=tracepoint:syscalls:sys_exit_writev, fd=41, ret=4631
    
            writev+77
            Envoy::Network::IoSocketHandleImpl::writev(Envoy::Buffer::RawSlice const*, unsigned long)+263
            Envoy::Network::IoSocketHandleImpl::write(Envoy::Buffer::Instance&)+107
            Envoy::Network::RawBufferSocket::doWrite(Envoy::Buffer::Instance&, bool)+121
            Envoy::Network::ConnectionImpl::onWriteReady()+1876
            Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+818
            std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
            Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
            0x7fffffffe000
            event_base_loop+1953
            Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
            Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
            start_thread+217
    
    
    ***** elapsed=1639293010: tid=5327,comm=wrk:worker_0: END:EventFired
    
    ******* WAKE-ROUND:END Summary *******
    ***** elapsed=1639308346: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=354486, tid2epollNrFdReady=2
    *** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=56 
    ***************************
    
    
    ***** elapsed=1643785802: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
    FileEventImpl*=0x559f98e26ee0, fd=41, events=0x26
    libevent: EV_READ
    libevent: EV_WRITE
    libevent: EV_ET
    
    
    ***** elapsed=1643855427: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=41, ret=0
    
            readv+77
            Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
            Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
            Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
            Envoy::Network::ConnectionImpl::onReadReady()+753
            Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
            std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
            Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
            0x7fffffffe000
            event_base_loop+1953
            Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
            Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
            start_thread+217
    
    ***** elapsed=1643880789: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_DEL, fd=41, events=0x80000005
    sys_enter_close fd=41
    delete fd2filterManagerImpl, fd=41
    delete fd2filterManagerImpl=0x559f98d38758
    delete fd2filterManagerImpl, fd=41
    delete fd2filterManagerImpl=0x559f98996380
    
    ***** elapsed=1644006858: tid=5327,comm=wrk:worker_0: END:EventFired
    
    ******* WAKE-ROUND:END Summary *******
    ***** elapsed=1644056452: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=289169, tid2epollNrFdReady=1
    *** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=52 
    ***************************
    
    ^C
    
    
    @epoll_ctl_op[1]: EPOLL_CTL_ADD
    @epoll_ctl_op[2]: EPOLL_CTL_DEL
    @epoll_ctl_op[3]: EPOLL_CTL_MOD
    
    
    @fd2sockopt[44, 6, 1, 1]: 1
    @fd2sockopt[41, 6, 1, 1]: 1
    
    
    @filterManagerImpl2fd[94143954060408]: 41
    @filterManagerImpl2fd[94143950909952]: 41
    

    BPF 脚本

    https://github.com/labilezhu/pub-diy/blob/main/low-tec/trace/trace-istio/trace-istio-part4/trace-envoy-filter-router.bt

    #!usr/bin/bpftrace
    /*
    IMPORT-ENV: $ENVOY_PID
    args: $1=ENVOY_PID
    */
    
    /*
    
    #export PODID=`sudo crictl  pods | grep fortio-server | egrep '\bReady' | awk '{print $1}'`
    #export CONTAINERID=`sudo crictl ps | grep $PODID | grep istio-proxy |  awk '{print $1}'`
    
    kubectl -n idm-mark exec -it fortio-server -c  istio-proxy  -- bash -c 'ls -l /proc/`pgrep envoy`/ns/pid'
    lrwxrwxrwx 1 istio-proxy istio-proxy 0 Mar 31 09:18 /proc/15/ns/pid -> 'pid:[4026532799]'
    pgrep envoy | tee /dev/fd/2 | xargs -L 1 -I '{}' sudo ls -l /proc/'{}'/ns 
    
    export PID=4215
    export SCRIPT_HOME=`pwd`
    export bpftrace_image=cndt-bcc-ub
    
    docker run -it --rm --init  --privileged --name bpftrace -h bpftrace \
        --pid host \
        --net host \
        -e SCRIPT_HOME=$SCRIPT_HOME \
        -e PID=$PID \
        -e ENVOY_PID=$PID \
        -e BT=trace-envoy-filter-router.bt \
        -v /etc/localtime:/etc/localtime:ro \
        -v /sys:/sys:rw \
        -v /usr/src:/usr/src:rw \
        -v /lib/modules:/lib/modules:ro \
        -v ${SCRIPT_HOME}:${SCRIPT_HOME}:rw \
        $bpftrace_image \
        ${SCRIPT_HOME}/warp-bt.sh $PID
    */
    
    
    /*
    1. BPF Map 说明:
    
    1.1. range tid map
    - @watchedWakeRound[tid]=tid
           - sys_exit_accept4
           - FileEventImpl*assignEvents*
    - @fdFired[tid]=$fd;
           - uprobe: *FileEventImpl*assignEvents*
           - uretprobe: *FileEventImpl*assignEvents*
    
    1.2. fd map
    - @fd2sockpair[$fd]=@sockpair[tid]
           - sys_exit_accept4
           - sys_enter_close
    - @fd2sockopt[$fd, $level, $optname, $optval_int] = 1
           - sys_enter_setsockopt
    
    - @filterManagerImpl2fd[arg0] = @fdFired[tid] // filterManagerImpl to downstream fd
    - @fd2filterManagerImpl[$fd] = arg0 
    - @fd2filterManagerImpl2[$fd] = arg0
    - @downFD2upFD[$downFD] = $fd; //downstream fd to upstream fd
           - sys_enter_epoll_ctl
    - @upFD[$fd] = $fd; //upstream fd list
           - sys_enter_epoll_ctl
    
    1.3. temp tid map
    - @sockpair[tid]=($sk->__sk_common.skc_daddr, $dport, $sk->__sk_common.skc_rcv_saddr, $lport);
    - @sockpair_exist[tid]=1
    - @watchedIo[tid]=fd
    
    */
    
    #include <linux/in.h>
    #include <linux/in6.h>
    #include <linux/socket.h>
    #include <net/sock.h>
    // #include <sys/epoll.h>
    
    // The event argument describes the object linked to the file descriptor fd.  The struct epoll_event is defined as:
    // struct epoll_data_t {
    //     int          fd;
    // };
    
    // struct epoll_event {
    //     //a bit mask composed by ORing together zero or more of the following available event types
    //     uint32_t     events;      /* Epoll events */
    
    //     struct epoll_data_t data;        /* User data variable */
    // };
    
    
    
    BEGIN
    {
        printf("Tracing Envoy. Hit Ctrl-C to end.\n");
           printf("#define EPOLL_CTL_ADD 1\n #define EPOLL_CTL_DEL 2\n #define EPOLL_CTL_MOD 3\n");
    
           @epoll_ctl_op[1]="EPOLL_CTL_ADD";
           @epoll_ctl_op[2]="EPOLL_CTL_DEL";
           @epoll_ctl_op[3]="EPOLL_CTL_MOD";
    
    
    }
    
    /*
    获取 accept 连接时的本地和对端地址。并标记本轮 wakeup 需要监控
     */
    kretprobe:inet_csk_accept
    /pid==$1 /
    {
        $sk = (struct sock *)retval;
        $inet_family = $sk->__sk_common.skc_family;
    
        if ($inet_family == AF_INET || $inet_family == AF_INET6) {
            // initialize variable type:
            $daddr = ntop(0);
            $saddr = ntop(0);
            if ($inet_family == AF_INET) {
                $daddr = ntop($sk->__sk_common.skc_daddr);
                $saddr = ntop($sk->__sk_common.skc_rcv_saddr);
            } else {
                printf("not support IPv6.\n");
                return;
            }
            
            $lport = $sk->__sk_common.skc_num;
    
                  // printf("accept(), port=%d\n", $lport);
    
                  //only watch listen port 15006
            if( 15006 != $lport ) { 
                return;
            }
    
            $dport = $sk->__sk_common.skc_dport;
            $qlen  = $sk->sk_ack_backlog;
            $qmax  = $sk->sk_max_ack_backlog;
    
            // Destination port is big endian, it must be flipped
            $dport = ($dport >> 8) | (($dport << 8) & 0x00FF00);
    
                  printf("OS handshaked TCP:\n");
            time("%H:%M:%S ");
            printf("%-6d %-14s ", pid, comm);
            printf("%-39s %-5d %-39s %-5d ", $daddr, $dport, $saddr,
                $lport);
            printf("%d/%d\n", $qlen, $qmax);
    
            @sockpair[tid]=($sk->__sk_common.skc_daddr, $dport, $sk->__sk_common.skc_rcv_saddr, $lport);
            @sockpair_exist[tid]=1;
    
                  @watchedWakeRound[tid]=tid;
        }
    }
    
    /**
     * 获取新连接的 FD 。并标记本轮 libevent 回调需要监控
     */
    tracepoint:syscalls:sys_exit_accept4
    /pid==$1 && @sockpair_exist[tid] /
    {
        $fd = args->ret;
        if( $fd < 0 ) {
            return;
        }
        printf("sys_exit_accept4 fd=%d\n", $fd);
        @fd2sockpair[$fd]=@sockpair[tid];
    
           @fdFired[tid]=$fd;
    
        delete(@sockpair[tid]);
        delete(@sockpair_exist[tid]);
           printf("%s \n", ustack());
    }
    
    /**
     * 记录 FD 的 sockopt
     */
    tracepoint:syscalls:sys_enter_setsockopt
    /pid==$1/
    {
           // socket opts: https://elixir.bootlin.com/linux/v5.16.3/source/include/uapi/linux/tcp.h#L92     
           $level = args->level;
           $fd = args->fd;
    
           if( @fd2sockpair[$fd].0 || @upFD[$fd] ) {
                  $optname = args->optname;
                  $optval = args->optval;
                  $optval_int = *$optval;
                  $optlen = args->optlen;
                  // printf("\n########## setsockopt() ##########\n");
                  printf("comm:%-16s: setsockopt: level=%d, fd=%d, optname=%d, optval=%d, optlen=%d. \n", comm, $level, $fd, $optname, $optval_int, $optlen);
                  @fd2sockopt[$fd, $level, $optname, $optval_int] = 1;
           }
    }
    
    
    
    /*
    记录 epoll 监听的 FD
    
    cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_epoll_ctl/format
     */
    tracepoint:syscalls:sys_enter_epoll_ctl
    /pid==$1/
    {
           $fd=args->fd;
           $epollEvents = (uint32) (args->event->events);
           $op = args->op;
           if( @fd2sockpair[$fd].0 ) {
    
                  printf("***** elapsed=%d: tid=%d,comm=%s: sys_enter_epoll_ctl, epfd=%d, op=%s, fd=%d, events=0x%x\n", 
                         elapsed, tid, comm, args->epfd, @epoll_ctl_op[$op], $fd, $epollEvents);
    
    
                  if( $op == 1/*EPOLL_CTL_ADD*/ || $op == 3/*EPOLL_CTL_MOD*/ ) { //add watch epoll event
                         if( $epollEvents & (uint32)0x001 /*EPOLLIN*/ ) {
                                printf("EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)\n");
                         }
                         if( $epollEvents & (uint32)0x004 /*EPOLLOUT*/ ) {
                                printf("EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)\n");
                         }
                         if( $epollEvents & (1u << 31) /*EPOLLET*/ ) {
                                printf("EPOLL_CTL_ADD/MOD EdgeTrigger\n");
                         }
                         printf("%s\n", ustack(30));
                  }
                  
           } else {
                  $downFD = @fdFired[tid];
                  if( $downFD ) {//in downstream fd event callback, register upstream event trigger
                         @upFD[$fd] = $fd;
                         @downFD2upFD[$downFD] = $fd;
    
                         printf("***** elapsed=%d: tid=%d,comm=%s: register upstream event trigger:sys_enter_epoll_ctl, epfd=%d, op=%s, fd=%d, events=0x%x\n", 
                                elapsed, tid, comm, args->epfd, @epoll_ctl_op[$op], $fd, $epollEvents);
    
    
                         if( $op == 1/*EPOLL_CTL_ADD*/ || $op == 3/*EPOLL_CTL_MOD*/ ) { //add watch epoll event
                                if( $epollEvents & (uint32)0x001 /*EPOLLIN*/ ) {
                                       printf("EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)\n");
                                }
                                if( $epollEvents & (uint32)0x004 /*EPOLLOUT*/ ) {
                                       printf("EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)\n");
                                }
                                if( $epollEvents & (1u << 31) /*EPOLLET*/ ) {
                                       printf("EPOLL_CTL_ADD/MOD EdgeTrigger\n");
                                }
                                printf("%s\n", ustack(60));
                         }                     
                  }
           }
    }
    
    /*
    结束本轮的 wakeRound/runnableRound,并等待下一轮
    cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_epoll_wait/format
     */
    tracepoint:syscalls:sys_enter_epoll_wait
    /pid==$1/
    {
    
           if( @watchedWakeRound[tid] ) {
                  $runnableStartTime=@tid2Waketime[tid];
                  if( $runnableStartTime ) {
                         $runableDuaration = elapsed - $runnableStartTime;
                         printf("\n******* WAKE-ROUND:END Summary *******\n");
                         printf("***** elapsed=%d: tid=%d,comm=%s: sys_enter_epoll_wait, runableDuaration=%d, tid2epollNrFdReady=%d\n", 
                                elapsed, tid, comm, $runableDuaration, @tid2epollNrFdReady[tid]);
                         $tid_last_epoll_wait_args = @last_epoll_wait_args[tid];
                         if( $tid_last_epoll_wait_args.0  ) {
                                printf("*** last_epoll_wait_args: epfd=%d, events=%d, maxevents=%d, timeout=%d \n", 
                                       $tid_last_epoll_wait_args.0, $tid_last_epoll_wait_args.1, $tid_last_epoll_wait_args.2, $tid_last_epoll_wait_args.3);
                         }
                         printf("***************************\n\n");
                  }
           }
    
           delete(@tid2Waketime[tid]);
           delete(@watchedWakeRound[tid]);
           delete(@tid2epollNrFdReady[tid]);
    
           //read in next sys_enter_epoll_wait
           @last_epoll_wait_args[tid]=(args->epfd, args->events, args->maxevents, args->timeout);
    }
    
    /*
    开始本轮 wakeRound/runnableRound
    cat /sys/kernel/debug/tracing/events/syscalls/sys_exit_epoll_wait/format
     */
    tracepoint:syscalls:sys_exit_epoll_wait
    /pid==$1/
    {
           // printf("\n***** elapsed=%d: tid=%d,comm=%s: sys_enter_epoll_wait\n", elapsed, tid, comm);
    
           // printf("epfd: 0x%08lx, events: 0x%08lx, maxevents: 0x%08lx, timeout: 0x%08lx \n", 
           //        ((args->epfd)), ((args->events)), ((args->maxevents)), ((args->timeout)) );
    
           @tid2Waketime[tid]=elapsed;
           @tid2epollNrFdReady[tid]=args->ret;
    }
    
    /*
    记录本轮 epoll 事件触发的 FD 级回调开始
    
    Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int fd, short events, void* fileEventImplThis)  !!!NOT!!: FileEventImpl::assignEvents()
    C++11 Lambda expressions:
    event_assign( &raw_event_, base, fd_, xxx, [](evutil_socket_t, short what, void* arg)-> void {}, this );
    
    event_assign(): https://libevent.org/doc/event_8h.html#a3e49a8172e00ae82959dfe64684eda11
           event_assign (   struct event *  ev,
                  struct event_base *   base,
                  evutil_socket_t   fd,
                  short     events,
                  event_callback_fn     callback,
                  void *    callback_arg 
           )    
    
           https://libevent.org/doc/event_8h.html#aed2307f3d9b38e07cc10c2607322d758
           typedef void(* event_callback_fn) (evutil_socket_t, short, void *)
                  fd    An fd or signal
                  events    One or more EV_* flags
                  arg   A user-supplied argument.
    */
    uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FileEventImpl*assignEvents*
    /pid == $1/ 
    {
           $fd = arg0;
           if( @fd2sockpair[$fd].0 || @upFD[$fd] ) {
                  $libevent_events = arg1;
    
                  printf("\n***** elapsed=%d: tid=%d,comm=%s: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()\n", elapsed, tid, comm);
                  printf("FileEventImpl*=%p, fd=%d, events=0x%x\n",arg2, $fd, $libevent_events);
    
                  if( $libevent_events & (uint16)0x01 /*EV_TIMEOUT*/ ) {
                         printf("libevent: EV_TIMEOUT\n");
                  }
                  if( $libevent_events & (uint16)0x02 /*EV_TIMEOUT*/ ) {
                         printf("libevent: EV_READ\n");
                  }
                  if( $libevent_events & (uint16)0x04 /*EV_TIMEOUT*/ ) {
                         printf("libevent: EV_WRITE\n");
                  }
                  if( $libevent_events & (uint16)0x20 /*EV_TIMEOUT*/ ) {
                         printf("libevent: EV_ET\n");
                  }
                  if( $libevent_events & (uint16)0x80 /*EV_TIMEOUT*/ ) {
                         printf("libevent: EV_CLOSED\n");
                  }
                  
                  printf("%s\n", kstack);
                  @fdFired[tid]=$fd;
                  @watchedWakeRound[tid]=tid;
           }
    }
    
    /*
    记录本轮 epoll 事件触发的 FD 级回调结束
    */
    uretprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FileEventImpl*assignEvents*
    /pid == $1 && @fdFired[tid] / 
    {
           printf("\n***** elapsed=%d: tid=%d,comm=%s: END:EventFired\n", elapsed, tid, comm);
           delete(@fdFired[tid]);
    }
    
    /*
    埋点 socket 读
    */ 
    tracepoint:syscalls:sys_enter_read,tracepoint:syscalls:sys_enter_readv,tracepoint:syscalls:sys_enter_recv*
    /pid == $1 && (@fd2sockpair[args->fd].0 || @upFD[args->fd] ) / 
    {
           @watchedIo[tid]=args->fd;
    }
    
    tracepoint:syscalls:sys_exit_read,tracepoint:syscalls:sys_exit_readv,tracepoint:syscalls:sys_exit_recv*
    /pid == $1 && @watchedIo[tid]/ 
    {
           printf("\n***** elapsed=%d: tid=%d,comm=%s: socket_read, probe=%s, fd=%d, ret=%d\n", elapsed, tid, comm, probe, @watchedIo[tid],args->ret);
           delete(@watchedIo[tid]);
           printf("%s\n", ustack(20));
    }
    
    /*
    埋点 socket 写
    */
    tracepoint:syscalls:sys_enter_write,tracepoint:syscalls:sys_enter_writev
    /pid == $1 && ( @fd2sockpair[args->fd].0 || @upFD[args->fd] ) / 
    {
           @watchedIo[tid]=args->fd;
    }
    
    tracepoint:syscalls:sys_exit_write,tracepoint:syscalls:sys_exit_writev
    /pid == $1 && @watchedIo[tid]/ 
    {
           printf("\n***** elapsed=%d: tid=%d,comm=%s: socket_write, probe=%s, fd=%d, ret=%d\n", elapsed, tid, comm, probe, @watchedIo[tid], args->ret);
           delete(@watchedIo[tid]);
           printf("%s\n", ustack(20));
    }
    
    
    /*
    打印 TlsInspector 的所有函数调用
    */
    uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*TlsInspector*
    /pid == $1 && @fdFired[tid]/ 
    {
           printf("\n***** elapsed=%d: tid=%d,comm=%s: TlsInspector*, probe=%s\n", elapsed, tid, comm, probe);
    }
    
    /*
    打印连接确认协后的函数调用
    */
    uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*ConnectionSocketImpl*setRequestedApplicationProtocols*
    /pid == $1 && @fdFired[tid]/ 
    {
           printf("\n***** elapsed=%d: tid=%d,comm=%s: ConnectionSocketImpl::setRequestedApplicationProtocols\n", elapsed, tid, comm);
           // printf("%s", ustack);
    }
    
    //SslSocket::SslSocket
    uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocketC2ENSt3__110shared_ptrINS_3Ssl7ContextEEENS2_12InitialStateERKNS5_IKNS_7Network22TransportSocketOptionsEEENS4_8functionIFNS5_INS6_10HandshakerEEENS4_10unique_ptrI6ssl_stN4bssl8internal7DeleterISK_EEEEiPNS6_18HandshakeCallbacksEEEE
    /pid == $1 && @fdFired[tid]/ 
    {
           printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
           printf("%s\n", ustack(10));
    }
    
    // setTransportSocketCallbacks
    uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocket27setTransportSocketCallbacksERNS_7Network24TransportSocketCallbacksE
    /pid == $1 && @fdFired[tid]/ 
    {
           printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
           printf("%s\n", ustack(10));
    }
    
    
    // SslSocket::doRead
    uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocket6doReadERNS_6Buffer8InstanceE
    /pid == $1 && @fdFired[tid]/ 
    {
           printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
           printf("%s\n", ustack(10));
    }
    
    
    // SslSocket::doWrite
    uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocket7doWriteERNS_6Buffer8InstanceEb
    /pid == $1 && @fdFired[tid]/ 
    {
           printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
           printf("%s\n", ustack(10));
    }
    
    // SslSocket::onSuccess
    uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZThn16_N5Envoy10Extensions16TransportSockets3Tls9SslSocket9onSuccessEP6ssl_st
    /pid == $1 && @fdFired[tid]/ 
    {
           printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
           printf("%s\n", ustack(10));
    }
    
    // SslSocket::connection
    uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZThn16_NK5Envoy10Extensions16TransportSockets3Tls9SslSocket10connectionEv
    /pid == $1 && @fdFired[tid]/ 
    {
           printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
           printf("%s\n", ustack(10));
    }
    
    // SslSocket::connection
    uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZThn16_NK5Envoy10Extensions16TransportSockets3Tls9SslSocket10connectionEv
    /pid == $1 && @fdFired[tid]/ 
    {
           printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
           printf("%s\n", ustack(10));
    }
    
    /*
    打印匹配到的 Network Fitler Chain 名字
    void setFilterChainName(absl::string_view filter_chain_name)
    */
    uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10StreamInfo14StreamInfoImpl18setFilterChainNameEN4absl11string_viewE 
    /pid == $1 && @fdFired[tid]/ 
    { 
    /**
    [Assembly 2: Calling convention](https://cs61.seas.harvard.edu/site/2018/Asm2/)
    
    1. A structure argument that fits in a single machine word (64 bits/8 bytes) is passed in a single register.
       
        Example: `struct small { char a1, a2; }`
    
    2. A structure that fits in two to four machine words (16–32 bytes) is passed in sequential registers, as if it were multiple arguments.
       
        Example: `struct medium { long a1, a2; }`
    */
        $filterName = str(reg("si"));
        $filterNameLength = reg("dx");
    
        // printf("tid:%d: Got setFilterChainName=%s, lenght=%d\n %s \n", tid, $filterName, $filterNameLength, ustack ); 
        printf("comm:%s,tid:%d: Got setFilterChainName=%s, lenght=%d\n", comm, tid, $filterName, $filterNameLength ); 
    
        if( $filterNameLength > 0 ) {
            printf("%s\n", ustack(3));
        }
    }
    
    /*
    埋点 FilterManagerImpl::add*Filter()
    */
    uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FilterManagerImpl*add*Filter*,uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FilterManagerImpl*FilterManagerImpl*
    /pid == $1 && @fdFired[tid]/ 
    { 
           if( @upFD[@fdFired[tid]] ) {
                  return;
           }
    
           $fd = @fdFired[tid];
           @filterManagerImpl2fd[arg0] = $fd;
           if( @fd2filterManagerImpl[$fd] ) {
                  @fd2filterManagerImpl2[$fd] = arg0;
           } else {
                  @fd2filterManagerImpl[$fd] = arg0;
           }
           printf("comm:%s,tid:%d: FilterManagerImpl.add*Filter|FilterManagerImpl::%s,FilterManagerImpl.this=%p,fd=%d \n", comm, tid, probe, arg0, $fd);
           printf("%s\n", ustack(40));
    }
    
    /*
    埋点 FilterManagerImpl::on*()
    */
    uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FilterManagerImpl*on*
    /pid == $1 && @fdFired[tid]/ 
    { 
           if( @upFD[@fdFired[tid]] ) {
                  return;
           }
    
           $fd = @fdFired[tid];
           @filterManagerImpl2fd[arg0] = $fd;
           if( @fd2filterManagerImpl[$fd] ) {
                  @fd2filterManagerImpl2[$fd] = arg0;
           } else {
                  @fd2filterManagerImpl[$fd] = arg0;
           }
           printf("comm:%s,tid:%d: FilterManagerImpl.on*::%s,FilterManagerImpl.this=%p,fd=%d \n", comm, tid, probe, arg0, $fd); 
    }
    
    /*
    埋点 ConnPoolImplBase::attachStreamToClient()
    */
    uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*ConnPoolImplBase*attachStreamToClient*
    /pid == $1 && @fdFired[tid]/ 
    { 
           printf("comm:%s,tid:%d: ConnPoolImplBase*attachStreamToClient: %s \n", comm, tid, probe); 
           printf("%s\n", ustack(40));
    }
    
    /*
    埋点 socket connect upstream
    */
    kprobe:tcp_connect
    /pid == $1/
    {
           $sk = ((struct sock *)arg0);
           $inet_family = $sk->__sk_common.skc_family;
    
           if ($inet_family == AF_INET || $inet_family == AF_INET6)
           {
                  if ($inet_family == AF_INET)
                  {
                         $daddr = ntop($sk->__sk_common.skc_daddr);
                         $saddr = ntop($sk->__sk_common.skc_rcv_saddr);
                  }
                  else
                  {
                         $daddr = ntop($sk->__sk_common.skc_v6_daddr.in6_u.u6_addr8);
                         $saddr = ntop($sk->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr8);
                  }
                  $lport = $sk->__sk_common.skc_num;
                  $dport = $sk->__sk_common.skc_dport;
    
                  // Destination port is big endian, it must be flipped
                  $dport = ($dport >> 8) | (($dport << 8) & 0x00FF00);
    
                  // only watch listen port 8080
                  if (8080 != $dport)
                  {
                         return;
                  }
    
                  printf("connecting to 8080:\n");
                  time("%H:%M:%S ");
                  printf("%-8d %-16s ", pid, comm);
                  printf("%-39s %-6d %-39s %-6d\n", $saddr, $lport, $daddr, $dport);
                  printf("%s\n", ustack(40));
           }
    }
    
    
    /*
    清理关闭的 FD 相关的 Map
    cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_close/format
    */
    tracepoint:syscalls:sys_enter_close
    /pid==$1/
    {
        $fd = args->fd;
        if( $fd < 0 ) {
            return;
        }
    
           delete(@upFD[$fd]);
    
           if( @fd2sockpair[$fd].0 ) {
                  printf("sys_enter_close fd=%d\n", $fd);
           }
           else {
                  return;
           }
        delete(@fd2sockpair[$fd]);
    
           /////
    
           $filterManagerImpl = @fd2filterManagerImpl[$fd];
           if( $filterManagerImpl ) {
                  printf("delete fd2filterManagerImpl, fd=%d\n", $fd );
                  delete( @fd2filterManagerImpl[$fd] );
           }
    
           if( @filterManagerImpl2fd[$filterManagerImpl] ) {
                  printf("delete fd2filterManagerImpl=%p\n", $filterManagerImpl );
                  delete(@filterManagerImpl2fd[$filterManagerImpl]);
           }
    
    
           $filterManagerImpl = @fd2filterManagerImpl2[$fd];
           if( $filterManagerImpl ) {
                  printf("delete fd2filterManagerImpl, fd=%d\n", $fd );
                  delete( @fd2filterManagerImpl2[$fd] );
           }
    
           if( @filterManagerImpl2fd[$filterManagerImpl] ) {
                  printf("delete fd2filterManagerImpl=%p\n", $filterManagerImpl );
                  delete(@filterManagerImpl2fd[$filterManagerImpl]);
           }
    
           delete(@upFD[@downFD2upFD[$fd]]);
           delete(@downFD2upFD[$fd]);
    }
    
    END
    {
           clear(@last_epoll_wait_args);
           clear(@tid2Waketime);
           clear(@tid2epollNrFdReady);
    }
    

    结尾

    这 part,从 socket 事件驱动的角度,了解了 envoy 作为反向代理的主要流程。我自己觉得是学习到东西的,你呢?
    写博客的初憧是记录自己。想不到也能通过博客认识到很多志同道合,对技术认真之人(不能叫同志?)。谢谢你们,我才能坚持写下去,也认为这样的写作有一定价值!

    image.png

    这是几年前拍的老照片了,共勉。

    相关文章

      网友评论

        本文标题:云原生现场分析 Part4 —— eBPF 跟踪 Istio/E

        本文链接:https://www.haomeiwen.com/subject/qadjsrtx.html