unable to hook EndScene

Jan 5, 2013 at 12:34 AM

I am trying to Hook EndScene and the injection/hook succeeds but the hooked function never gets called. Can someone help me debug this problem? The EndScene_Hook function never gets called. I am not sure what is going on. I see on debugging that LhInstallHook does replace the pointer to something new. But the function never gets called.

 

Thanks a lot in advance

Pankaj

 

Here is my code:

 

 DXD9Hook::DXD9Hook()
 {
       IDirect3D9* d3d;
       IDirect3DDevice9 *device;
       int NtStatus = ERROR_SUCCESS;

       device = NULL;
       HRESULT hRes = D3D_OK;
       ULONG ACLEntries[1] = {0};
       d3d = Direct3DCreate9(D3D_SDK_VERSION);
       D3DPRESENT_PARAMETERS pp = {};
        pp.BackBufferWidth = 1;
        pp.BackBufferHeight = 1;
        pp.BackBufferFormat = D3DFMT_X8R8G8B8;
        pp.BackBufferCount = 1;
        pp.SwapEffect = D3DSWAPEFFECT_DISCARD;
        pp.Windowed = TRUE;
  
        printf ("This is fucking coming here\n");

        hRes = d3d->CreateDevice(
            D3DADAPTER_DEFAULT,
            D3DDEVTYPE_HAL,
            GetDesktopWindow(),
            D3DCREATE_HARDWARE_VERTEXPROCESSING,
            &pp,
            &device);

  // This hack seems to work
  BYTE *ppVtable = (BYTE *)device;
  BYTE *pVtable = *((BYTE **)ppVtable);
  //BYTE* pVtable =  (BYTE *)(*(BYTE *)device);
       
  TRACED_HOOK_HANDLE hHook = new HOOK_TRACE_INFO();
  HMODULE hd3d9 = LoadLibraryA("d3d9.dll");
  
  device->EndScene();
  FORCE(LhInstallHook(
   pVtable + (EndScene * sizeof(int *)),
   //PVOID(0),
   (void *)(EndScene_Hook ),
   (PVOID)0x12345678,
   hHook));
  FORCE(LhSetExclusiveACL(ACLEntries, 1, hHook));

  //device->EndScene();
 ERROR_ABORT:
  if(hHook != NULL)
   delete hHook;
  return;

 }

 

HRESULT EndScene_Hook(IDirect3DDevice9 *device)

{

DebugBreak();

 

//device->EndScene();

 

return

S_OK;

}

Coordinator
Jan 10, 2013 at 8:31 AM

You may need to try calling EndScene in your test on a different thread and definitely after you have set the ACL's.

Jan 14, 2013 at 11:16 PM

Thanks so much. I think I found the issue. The issue was not in the hooking. The function was getting hooked properly but it was not the correct function. The issue was that the native version of CreateDevice returns a different interface pointer everytime it is called and a different vTable. I was hooking the wrong function. I was porting it directly from a C# code that was using the managed version of EasyHook and the C# SlimDX version of CreateDevice. This returns the same device everytime it is called and thus hooking it enables hooking all endscene calls in C#. But in native code, it does not work the same way.

The workaround in native code is to actually hook the function and not the entry in the vTable. That made it work.