Some Issues

Dec 16, 2010 at 8:34 PM
Edited Dec 16, 2010 at 8:42 PM

Hi, at first I'd liketo thank you for such a great project, but there are some miserable issues^

1. Call stack. I've found no way to take call stack in win32 with EasyHook. I've read your source code and found that you use RtlCaptureBacktrace fuction. This function tries to capture backtrace and make a lot of checks (because of FPO optimization). Every time RtlCaptureBacktrace moves one frame up it checks if frame address is inside DLL range, if not the whole function RtlCaptureBacktrace returns error.
You allocate some memory through VirtualAlloc for hook stubs, so when RtlCaptureBacktrace finds your stub (stub is not in any known dll range) it finishes immideatly with error and zero captured stacks (of course your hook stub must be non-FPO).
The solution is to create huge function in your library (EasyHook) full of noop instructions and allocate hook stubs inside it then RtlCaptureBacktrace sees that hook address is inside EasyHook32 address range, becomes happy and works perfectly.

2. Hook functions reentrace. I'm writing memory profilng application, so I have to hook HeapAlloc. Sometimes I have to allocate some memory right inside hook, this causes hook handle to be called once again recursively. I've protected my code through TLS counter but I found that you maintain Runtime->Current pointer, you reset it when you leave LhBarrierOutro function, so when I return from recurrent HeapAlloc function Runtime->Current pointer is already reset to NULL, so I cannot use any function that relies on it. I've tried to use ACE to disable reentrace, but it does not work. Even if I found the way to use ACE there is limit on thread count in ACE, so idea is no good.

That problems are critical for me, so I cannot use your library, but anyway library is outstanding, wish you luck!

RtlCaptureBacktrace
Dec 21, 2010 at 5:56 PM

Look at http://easyhook.codeplex.com/Thread/View.aspx?ThreadId=231486

The trick i've used is to allocate a private heap before hooking and not to trace calls when the private heap is used (i.e call th eoriginal function and return immediately if heap == my private heap). This allows you to allocate on the private heap for you need and still hook the function and trace other calls.

Dec 21, 2010 at 8:08 PM

So do I. But there is problem: I have to hook VirtualAllocation. 

When heap is growing it might allocate some memory by VirtualAllocEx. I do not want to track my own Virtual allocations that are caused by my own heap allocations, so I try to maintain call reentrace counter. If I pass allocation call through  I increment counter and if that counter is greater than zero in any allocation call that means that it is reentrant call and I dont trace it. I managed to use TLS to store counter, but I found that TLS can call HeapAllocEx in some cases, so I cannot use it too. Now I have no idea how to distinguish my own heap allocations, my own virtual allocations and other allocations