Managed hook and local variables in handlers

Jan 30, 2009 at 1:31 PM
Hello,

I'm trying to hook ExtTextOut and DrawText winapi functions, and thought I try using managed hooking. I was under the impression that I could use managed code internally in my handlers, but this is only true in some cases I found.

If I try to create a new string object in the handler, the target application crashes with  access violations, presumably because there wasn't any memory allocated for it. If I call native win32 functions to do the same, using IntPtr and LocalAlloc it works.

Can I use managed variables in the handlers, or do I have to use native memory functions to allocate variables and manipulate them?

Regards,
Tomas
Jan 30, 2009 at 4:00 PM
Edited Jan 30, 2009 at 4:02 PM
You have to be careful about that...

Of course you should be able to use all the managed functionality in your managed hook handlers, BUT:

- if you pass objects, like strings are, to native code and want that native code to operate with it, you have to allocate native memory! All plain NET objects are managed by the garbage collector. If you pass them to native code as parameter by calling a P-Invoke, NET will of course do the work for you. But in your case the situation is something different. Native code calls the NET framework and the NET framework won't do the job for you as it expects the unmanaged code to do so. But since the unmanaged code doesn't know about NET, it is your handler who has to do the job :-D.

regards
chris
Feb 2, 2009 at 11:06 AM
Hm.

I don't understand how this works. The handler, the injected dll written in C# and compiled with the managed libraries, don't know about .NET? I sort of assumed that the native code called the trampoline function residing in managed space, thus it had all the knowledge of managed functions, like strings and such.

Can you give an example of managed variables in a managed handler function (like declaring a string, adding some text to it, removing some text from it, and then passing it out to the original function)? Don't bother if this is much work, but just tell me it is much work instead :D

Thanks,
Toams
Feb 2, 2009 at 3:06 PM
There is no problem with using managed code within managed handler. You can write it just like you are used to!

> thus it had all the knowledge of managed functions, like strings and such

Just because it calls managed code and potentially has the knowledge about strings and such, this doesn't power the native handler to do some "magic" work and do things that are simply impossible in a generic way! The problem is stated below....

> passing it out to the original function

And exactly this is the point of failure! You just can't do this. If you pass a managed object to unmanaged code by return value, or by any other way than through parameter passing with marshalling, your code will fail. What do you expect the NET Framework to do in this case? The unmanaged code you are going to return the NET object to potentially doesn't know anything about that it just got a managed object. Even if you return a string and NET would marhsal it to an UTF-8 "char*": Then what about memory management? How should NET know when its time to release the allocated memory? The unmanaged code also can't release it, because it was allocated on an unknown heap.

WE HAVE A DIFFERENT SITUATION IN EASYHOOK:

I must admit, that this might be a little bit confusing. But keep in mind, that we are NOT doing usual P-Invoke with EasyHook. We are not calling unmanaged code from managed code, but unmanaged code calls managed code and on the top of this, the unmanaged code knows nothing about that ;-). So now you can imagine that you have to be quite careful.
If you call a usual P-Invoke method, NET knows exactly what to do. Marshal all parameters to unamanged code, when the method returns it will also marshal unmanaged return values back to NET objects and release the unmanaged parameters. If you pass managed handlers to unamanged code, I think the situation gets more complicated. I am not sure but I think you can't return objects in this case, because this would cause the same issues we have with EasyHook.

Feb 2, 2009 at 3:08 PM
To solve your problem, YOU are the one who has to use the InteropServices to properly allocate memory for the unmanaged code and also in a way the code expects it to be allocated and laid out. Then you can convert your managed objects to unmanaged memory and return the reference. This requires that the unmanaged code takes over memory allocation of the returned unamanged memory...
Feb 4, 2009 at 1:18 PM
Thanks for the replies. I've read through them a few times, and I think I understand what you are saying... :)

I'll continue my work with easyhook, and see how far I get :)

Remoting will be the next subject...