_RTC_CheckEsp exception in x86

Jan 28, 2013 at 9:54 PM

In the x86 version of our hooking code, we see a crash when the hooked function is called. The crash happens during the return from the hooked function. Here is the call stack :

 

0 ConfigSystem!failwithmessage+

1 ConfigSystem!_RTC_CheckEsp

2 ConfigSystem!DXUTRender3DEnvironment9

 

It seems _RTC_CheckEsp is doing some stack verification. This does not happen in the x64 version of the hook. Can you please help us figure out how to solve this problem ?

 

Thanks

Pankaj

Coordinator
Jan 29, 2013 at 2:31 AM

Please provide a simplified reproducible test case project (maybe upload to the issue tracker). Then I may be able to take a look for you.

Jan 30, 2013 at 3:21 AM

It is hard to setup a small test case project but I have setup a remote windbg debug session which is stuck right before the error:
windbg -remote tcp:Port=5000,Server=23.22.118.205 . I will put sources in the debugger  first thing in the morning if you would like.

Present has been hooked to Present_Hook using easyhook.(IT is all x86 , ConfigSystem.exe is the game that we are running and ISwifterInject.dll is the hooking library that is hooking into dxd9.dll's Present call).

The server is stopped inside the first call to Present_Hook right before the call to the original Present.

Here is my analysis -->

After the call to the original Present, the code returns back fine to the Present_Hook function. But the the stack pointers are messed up. After the return from the Present_Hook call, there is a cmp instruction on esi and esp, If they dont match, the error stack written before is shown after an exception.

Some debugging gotchas -->

1. Initially only the Present_Hook stack is correct. Easyhook has confused the rest of the stack for ConfigSystem.exe.

2. If you do a few "u" commands (unassemble), you will see a call to EasyHook!LhBarrierOutro and then a return (retn instruction). You can do repeated "p" command to reach the retn instruction. After the retn instruction, if you do another "p",  there is cmp esi, esp in ConfigSystem.exe . But even now, the ConfigSystem.exe stack is messed up.

3. If you do a "g" or go at this point you will hit the exception. The exception seems to be caused by the fact that esp and esi did not match in the previous step.

Please let me know if you would like to reset the debugger or if there is any problem connecting to it. I will put sources in the morning. If you see anything wrong in the hooking code which is causing this, please let us know.

Jan 30, 2013 at 9:16 PM

I have put sources in the debugger and the debugger is currently stuck right before the error like mentioned in the previous email. It would be awesome if you could take a look.

Coordinator
Feb 1, 2013 at 6:41 AM
Hi Pankaj,

Sorry I have been real busy with work and life etc... I will try to connect soon.

Cheers,
J
Coordinator
Feb 1, 2013 at 7:06 AM
Pankaj,

Ok I haven't looked at the WinDbg session much yet, however I noticed you are running one of the D3D samples, so I ran it here, and everything is working fine (I have a number of projects that hook DX stuff for screen capture and UI overlay purposes).

Before I spend any time in the debugger:
  1. can you reproduce on other system configurations?
  2. does it happen in only this one sample?
  3. What EasyHook version are you using?
  4. what is your system configuration? OS version etc.
The reason I ask is I have been hooking various D3D functions in d3d9, d3d9Ex, d3d10, d3d10_1 and d3d11 for a long time now and haven't had the problem you described...

Cheers,
J
Coordinator
Feb 1, 2013 at 7:07 AM
Pankaj,

What method are you using to call the original Present? Are you access the original pointer, or using SlimDX or SharpDX or some other managed wrapper?

Cheers,
J
Feb 1, 2013 at 8:07 PM
We are using(Actually the DirectX SDK sample is) the native C++ version of DirectX. I dont think they are using SlimDX or SharpDX.

Answers to other questions:
  1. We cannot reproduce on x64 , it only happens on x86. We can hook EndScene okay but we cannot hook present. EndScene has fewer parameters.
  2. I will try other samples
  3. We are using 1.7 I think
  4. We have a Windows 7 x86 PC running DirectX sample and Easyhook with Visual Studio installed on it. We are running the debug version. Not sure if that would matter.
Coordinator
Feb 2, 2013 at 6:40 AM
Edited Feb 2, 2013 at 6:42 AM
My guess is you have the parameters wrong. Please post your hook handler and delegate so I can check.

Here is my version of the delegates (working correctly):
        [UnmanagedFunctionPointer(CallingConvention.StdCall, CharSet = CharSet.Unicode, SetLastError = true)]
        unsafe delegate int Direct3D9Device_PresentDelegate(IntPtr devicePtr, SharpDX.Rectangle* pSourceRect, SharpDX.Rectangle* pDestRect, IntPtr hDestWindowOverride, IntPtr pDirtyRegion);

        [UnmanagedFunctionPointer(CallingConvention.StdCall, CharSet = CharSet.Unicode, SetLastError = true)]
        unsafe delegate int Direct3D9DeviceEx_PresentExDelegate(IntPtr devicePtr, SharpDX.Rectangle* pSourceRect, SharpDX.Rectangle* pDestRect, IntPtr hDestWindowOverride, IntPtr pDirtyRegion, Present dwFlags);
I use SharpDX to call the original PresentHook method, here is a cut down version of the Present:
        unsafe int PresentHook(IntPtr devicePtr, SharpDX.Rectangle* pSourceRect, SharpDX.Rectangle* pDestRect, IntPtr hDestWindowOverride, IntPtr pDirtyRegion)
        {
            Device device = (Device)devicePtr;

            if (pSourceRect == null || *pSourceRect == SharpDX.Rectangle.Empty)
                device.Present();
            else
            {
                if (hDestWindowOverride != IntPtr.Zero)
                    device.Present(*pSourceRect, *pDestRect, hDestWindowOverride);
                else
                    device.Present(*pSourceRect, *pDestRect);
            }
            return SharpDX.Result.Ok.Code;
        }
Feb 2, 2013 at 10:06 PM
I am putting a snippet of our version. We are only using the unmanaged APIs. Some places where I think I did some hacking I have explained in /COMMENT: ... /

/COMMENT: this is the prototype of the function we are using to Hook Present /
HRESULT Present_Hook(IDirect3DDevice9 *device,
const RECT *pSourceRect,
const RECT *pDestRect,
HWND hDestWindowOverride,
const RGNDATA *pDirtyRegion)
{
}

NativeEntryPoint()
{
    /*COMMENT: Unlike SlimDX, it seems this call returns a different "device" pointer every time */
hRes = d3d->CreateDevice(
           D3DADAPTER_DEFAULT,
           D3DDEVTYPE_HAL,
           GetDesktopWindow(),
           D3DCREATE_HARDWARE_VERTEXPROCESSING,
           //D3DCREATE_SOFTWARE_VERTEXPROCESSING | D3DCREATE_FPU_PRESERVE,
           &pp,
           &device);

    /*COMMENT: Even though the "device" pointer is different, the actual functions are at the same location in memory which we can get after some traversal */
    BYTE *pppPresent = (BYTE *)device; 
    BYTE *ppPresent = *((BYTE **)pppPresent);
    BYTE *pPresent = *((BYTE **)(ppPresent + Present*sizeof(void *)));

TRACED_HOOK_HANDLE hHook = new HOOK_TRACE_INFO();

    /* COMMENT: HEre is another fixme where we need to sleep before calling LhInstallHook, but it works after a sleep :) */
    //FIXME: Somehow we need to wait before callign LhInstallHook: TODO investigate why
    Sleep(1000);

     FORCE(LhInstallHook(
         pPresent,
         (void *)(Present_Hook),
         (PVOID)0x12345678,
          hHook));

    FORCE(LhSetExclusiveACL(ACLEntries, 1, hHook));
}
Feb 2, 2013 at 10:16 PM
I am wondering if I can use the unsafe keyword on Present_Hook to solve this problem like your SharpDX code is doing. I am not sure if it even available as it is not a C++/CLI class. I will try it asap.
Coordinator
Feb 3, 2013 at 12:14 AM
pankajk81 wrote:
I am wondering if I can use the unsafe keyword on Present_Hook to solve this problem like your SharpDX code is doing. I am not sure if it even available as it is not a C++/CLI class. I will try it asap.
The reason I used it was that the structure is passed as a pointer and I had to check for if it is null. In C# you cannot have a structure set to null unless you are passing in an IntPtr, or doing something like I am above.

The main reason I did this was so that I could easily pass through the parameters to the SharpDX method - I could have also simply called the function pointer directly then you can just use IntPtr parameters and don't require the unsafe - e.g.:
var callOriginal = (Direct3D9Device_PresentDelegate)(Object)Marshal.GetDelegateForFunctionPointer(intPtrToPresentMethod, typeof(Direct3D9Device_PresentDelegate));