Unmanaged code, CloseHandle, Notepad.exe

Aug 19, 2010 at 11:05 PM
Edited Aug 19, 2010 at 11:06 PM

Hi,

I have problem with hooking CloseHandle in Notepad.exe. Here is what i do... I start Notepad.exe. Then I let my app to remotely hook Notepad.exe and to install the CloseHandleHook. Everything works fine... But if I open some txt file, change it, save it and finaly I close the Notepad (doesn't matter by X or by Menu->Close). There is high probability the Notepad just deadlocks and the process itself won't terminate (i see it in taskmgr), even if the gui is already closed. Deadlock doesn't happen everytime, but quite often... Don't know if the problem is in EasyHook or in notepad itself or, with the highest probability, in my CODE :D. But without hooking CloseHandle (I also hook CreateFile, ReadFile, WriteFile... even COM ) everything seem working fine.

some pseudocode in my InjectDLL:

OnNativeEntryPoint:
   GetCurThreadId
   InstallHook ( only one: CloseHandleHook ... it just imediatelly calls the original CloseHandle function )
SetExclusiveACL (with curThreadId)
WaitForMultileObjects (my app specific)
//next code doesn't get called, because Notepad terminates itself
UninstallAllHooks
WaitForPendingRemovals

Hook gets called np, no errors in startup. DLLMain entry is not set (dunno where is declared). Interesting think is that when I set own DLLMain and Notepad terminates propertly ( well I don't know if propertly, but at least it terminates :D ), then DLLMain with DLL_PROCESS_DETACH is not called. But when Notepad hangs DLL_PROCESS_DETACH gets called. :s

System is Win7-32b. Already tried both release and debug versions of dlls. Unmanaged code...

Thanks for help
   Petr Pospisil.