美文网首页
【渲染逆向】RenderDoc hook与capture流程分析

【渲染逆向】RenderDoc hook与capture流程分析

作者: crossous | 来源:发表于2022-02-17 17:15 被阅读0次

      承接上一篇【渲染逆向】Hook D3D API.

    DX11的Hook流程

      程序的入口点(DllMain)在win32_libentry.cpp中

    BOOL APIENTRY DllMain(HMODULE hModule, DWORD ul_reason_for_call, LPVOID lpReserved)
    {
      if(ul_reason_for_call == DLL_PROCESS_ATTACH)
      {
        BOOL ret = add_hooks();
        SetLastError(0);
        return ret;
      }
    
      return TRUE;
    }
    

      在函数add_hooks的结尾会调用LibraryHooks::RegisterHooks()

    void LibraryHooks::RegisterHooks()
    {
      BeginHookRegistration();
    
      for(LibraryHook *lib : LibList())
        lib->RegisterHooks();
    
      EndHookRegistration();
    }
    

      LibList函数返回LibraryHook对象的列表,而向列表中加入对象,是LibraryHook调用构造函数的时候。

    LibraryHook::LibraryHook()
    {
      LibList().push_back(this);
    }
    

      显然LibraryHook是一个基类,会有超级多需要Hook的库来继承这一类,而我们现在关心的是DX11的库。

    class D3D11Hook : LibraryHook
    {
    public:
      void RegisterHooks();
    private:
      static D3D11Hook d3d11hooks;
    //something....
    }
    

      可以发现D3D11Hooks有一个静态成员,当这个类型初始化时,会自动生成一个D3D11Hook对象,并调用父类LibraryHook的构造函数,将自身加入Lib列表中。
      当函数add_hooks遍历列表,调用RegisterHooks时,会调用D3D11Hook::RegisterHooks

    void RegisterHooks()
    {
      //something....
    
      CreateDevice.Register("d3d11.dll", "D3D11CreateDevice", D3D11CreateDevice_hook);
      CreateDeviceAndSwapChain.Register("d3d11.dll", "D3D11CreateDeviceAndSwapChain",
                                        D3D11CreateDeviceAndSwapChain_hook);
    }
    

      这里会调用HookedFunction::Register,然后调用LibraryHooks::RegisterFunctionHook,将Hook所需的数据(dll库名称、函数名称、原始函数地址、hook函数地址)记录到数据结构中,之后统一用IAT hook进程函数hook,如果有需要,可以在这里修改为其他hook类型,例如:

    void LibraryHooks::RegisterFunctionHook(const char *libraryName, const FunctionHook &hook)
    {
      //something...
    
      if(!_stricmp(libraryName, "d3d11.dll") || !_stricmp(libraryName, "dxgi.dll"))
      {
        std::cout << "inline hook " << libraryName
                  << "\t-\t" << hook.function.c_str() << std::endl;
        *hook.orig = GetProcAddress(GetModuleHandleA(libraryName), hook.function.c_str());
    
         DetourTransactionBegin();
         DetourUpdateThread(GetCurrentThread());
         DetourAttach((PVOID *)hook.orig, hook.hook);
    
         DetourTransactionCommit();
      }
      else
      {
        s_HookData->DllHooks[strlower(rdcstr(libraryName))].FunctionHooks.push_back(hook);
      }
    }
    

      这里我就修改了一下,假如是d3d11.dll或dxgi.dll的函数,则进行inline hook,而其他库函数则保持原样。
      dxgi.dll的Hook同理:

    class DXGIHook : LibraryHook
    {
    public:
      void RegisterHooks()
      {
        LibraryHooks::RegisterLibraryHook("dxgi.dll", NULL);
    
        CreateDXGIFactory.Register("dxgi.dll", "CreateDXGIFactory", CreateDXGIFactory_hook);
        CreateDXGIFactory1.Register("dxgi.dll", "CreateDXGIFactory1", CreateDXGIFactory1_hook);
        CreateDXGIFactory2.Register("dxgi.dll", "CreateDXGIFactory2", CreateDXGIFactory2_hook);
        GetDebugInterface.Register("dxgi.dll", "DXGIGetDebugInterface", DXGIGetDebugInterface_hook);
        GetDebugInterface1.Register("dxgi.dll", "DXGIGetDebugInterface1", DXGIGetDebugInterface1_hook);
      }
    

      这些都是Hook需要查询地址的函数,而类似SetRenderTargets之类的虚表函数的Hook方式,放到下一节来找。

    DX11截帧流程

      RenderDoc按键或调用提供的api截帧都是调用core.h中的RenderDoc::TriggerCapture(uint32_t numFrames)函数记录需要截几帧。

    void TriggerCapture(uint32_t numFrames) { m_Cap = numFrames; }
    

      同文件下又提供了另一个方法bool RenderDoc::ShouldTriggerCapture(uint32_t frameNumber)

    bool RenderDoc::ShouldTriggerCapture(uint32_t frameNumber)
    {
      bool ret = m_Cap > 0;
    
      if(m_Cap > 0)
        m_Cap--;
    //something....
      return ret;
    

      继续查找引用可以找到一个相对有用的函数——WrappedID3D11Device::Present,这个函数在结尾调用的ShouldTriggerCapture,并且这个函数中还可以调整OverlayText(RenderDoc左上角的覆盖层)。

    //在WrappedID3D11Device::Present中
    if(IsBackgroundCapturing(m_State) && RenderDoc::Inst().ShouldTriggerCapture(m_FrameCounter))
    {
      RenderDoc::Inst().StartFrameCapture((ID3D11Device *)this, swapper->GetHWND());
    
      m_AppControlledCapture = false;
      m_CapturedFrames.back().frameNumber = m_FrameCounter;
    }
    
    void RenderDoc::StartFrameCapture(void *dev, void *wnd)
    {
      IFrameCapturer *frameCap = MatchFrameCapturer(dev, wnd);
      if(frameCap)
      {
        frameCap->StartFrameCapture(dev, wnd);
        m_CapturesActive++;
      }
    }
    

      frameCapIFrameCapturer的指针,IFrameCapturer是个接口,而实现这个接口的是WrappedID3D11Device

    void WrappedID3D11Device::StartFrameCapture(void *dev, void *wnd)
    {
    //something...
      m_State = CaptureState::ActiveCapturing;
    //something...
    }
    

      在这里激活了当前抓取状态,core.h中封装了一层函数:

    constexpr inline bool IsActiveCapturing(CaptureState state)
    {
      return state == CaptureState::ActiveCapturing;
    }
    

      当程序调用渲染api时,会进入到RenderDoc定义的Hook函数中,在调用原有函数后,会检查截帧状态,假如处于截帧状态,则记录API,这就是抓帧的实现,贴一个例子,下面是一个Wrap(包装)好的api:

    void WrappedID3D11DeviceContext::IASetPrimitiveTopology(D3D11_PRIMITIVE_TOPOLOGY Topology)
    {
      SCOPED_LOCK_OPTIONAL(m_pDevice->D3DLock(), m_pDevice->D3DThreadSafe());
    
      DrainAnnotationQueue();
    
      m_EmptyCommandList = false;
      //调用原有函数
      SERIALISE_TIME_CALL(m_pRealContext->IASetPrimitiveTopology(Topology));
      //如果处于截帧状态,则序列化API和参数
      if(IsActiveCapturing(m_State))
      {
        USE_SCRATCH_SERIALISER();
        SCOPED_SERIALISE_CHUNK(D3D11Chunk::IASetPrimitiveTopology);
        SERIALISE_ELEMENT(m_ResourceID).Named("Context"_lit).TypedAs("ID3D11DeviceContext *"_lit);
        Serialise_IASetPrimitiveTopology(GET_SERIALISER, Topology);
    
        m_ContextRecord->AddChunk(scope.Get());
      }
    
      m_CurrentPipelineState->Change(m_CurrentPipelineState->IA.Topo, Topology);
      VerifyState();
    }
    

      借此我们对这些Wrap的函数产生兴趣,例如WrappedID3D11Device,查看声明:

    class WrappedID3D11Device : public IFrameCapturer, public ID3DDevice, public ID3D11Device5
    

      这个类竟然实现了ID3DDevice这个DX接口。
      断点跟踪下这个类的构造函数调用堆栈

    -D3D11Hook::D3D11CreateDevice_hook
    -D3D11Hook::D3D11CreateDeviceAndSwapChain_hook
    -D3D11Hook::Create_Internal
    -WrappedID3D11Device::WrappedID3D11Device
    

      第一个函数D3D11Hook::D3D11CreateDevice_hook是RenderDoc的hook函数,本文的第一节已经提到了;在D3D11Hook::Create_Internal中,调用了原始的D3D11CreateDeviceAndSwapChain方法,然后用返回的device构造wrap device,最后将wrap返回,由于wrap device同样实现了ID3D11Device类的接口,因此在使用上和DirectX的Device没有区别:

    HRESULT Create_Internal(RealD3D11CreateFunction real, __in_opt IDXGIAdapter *pAdapter,
                            D3D_DRIVER_TYPE DriverType, HMODULE Software, UINT Flags,
                            __in_ecount_opt(FeatureLevels) CONST D3D_FEATURE_LEVEL *pFeatureLevels,
                            UINT FeatureLevels, UINT SDKVersion,
                            __in_opt CONST DXGI_SWAP_CHAIN_DESC *pSwapChainDesc,
                            __out_opt IDXGISwapChain **ppSwapChain, __out_opt ID3D11Device **ppDevice,
                            __out_opt D3D_FEATURE_LEVEL *pFeatureLevel,
                            __out_opt ID3D11DeviceContext **ppImmediateContext)
    {
    //...
    //调用原有D3D11CreateDeviceAndSwapChain方法
      HRESULT ret = real(pAdapter, DriverType, Software, Flags, pFeatureLevels, FeatureLevels,
                         SDKVersion, pUsedSwapDesc, ppSwapChain, ppDevice, pFeatureLevel, NULL);
    //构造wrap device
      WrappedID3D11Device *wrap = new WrappedID3D11Device(*ppDevice, params);
    //将wrap device的地址通过指针返回
      *ppDevice = wrap;
    
    //...
    }
    

      其他DX组件,例如IDXGIFactory、ID3D11DeviceContext的hook方式与此类似。

      报个坑,Nvidia 驱动安装程序Geforce Experience的游戏内覆盖如果开启,有几率会弹NVAPI_NO_IMPLEMENT错误。

    相关文章

      网友评论

          本文标题:【渲染逆向】RenderDoc hook与capture流程分析

          本文链接:https://www.haomeiwen.com/subject/diedlrtx.html