Hooking API functions that use handles fails ?

Jan 18, 2009 at 9:57 PM
Hi there

First, thanks for a great piece of work - i've been using the original detours in a project and now i need 64-bit, so I came upon what you've done.
My problem is, that it seems when I'm calling API functions that use handles i get the "Attempted to read or write protected memory. This is often an indication that other memory is corrupt." error.
I've boiled it down to code that works fine in a standard win-app not working when situated in the hooking function :

Works
        private void button2_Click(object sender, EventArgs e)
        {
            IntPtr ptrPrinter = new IntPtr();


            LibWrap.OpenPrinter("Microsoft XPS Document Writer", ref ptrPrinter, IntPtr.Zero);
        }

    public class LibWrap
    {
        [DllImport("winspool.drv",
            CharSet = CharSet.Auto,
            SetLastError = true,
            CallingConvention = CallingConvention.StdCall)]
        public static extern int OpenPrinter(String pPrinterName, ref IntPtr phPrinter, IntPtr pDefault);
    }

Doesn't work
        [UnmanagedFunctionPointer(CallingConvention.StdCall,
            CharSet = CharSet.Auto,
            SetLastError = true)]
        delegate bool DOpenPrinterW([System.Runtime.InteropServices.MarshalAsAttribute(System.Runtime.InteropServices.UnmanagedType.LPWStr)] String pPrinterName, ref System.IntPtr phPrinter, ref PRINTER_DEFAULTSW pDefault);

        [UnmanagedFunctionPointer(CallingConvention.StdCall,
            CharSet = CharSet.Auto,
            SetLastError = true)]
        delegate int DOpenPrinter(String pPrinterName, ref IntPtr phPrinter, IntPtr pDefault);

        [DllImport("winspool.drv",
            CharSet = CharSet.Auto,
            SetLastError = true,
            CallingConvention = CallingConvention.StdCall)]
        static extern int OpenPrinter(String pPrinterName, ref IntPtr phPrinter, IntPtr pDefault);

        static int OpenPrinter_Hooked(String pPrinterName, ref System.IntPtr phPrinter, IntPtr pDefault)
        {
            int returnValue = 0;


            try
            {
                IntPtr ptrPrinter = new IntPtr();


                OpenPrinter("Microsoft XPS Document Writer", out ptrPrinter, IntPtr.Zero);
            }
            catch(Exception e)
            {
                MessageBox.Show(e.Message);
            }
            return returnValue;
        }

Beyond the fact that the code that doesn't work being loaded as a dll in a host-process and the working code being in its own process, I really don't see a difference...
I'm at a loss as to why i get a System.AccessViolationException

Any help would be greatly appreciated.
Jan 23, 2009 at 8:41 PM
So noone has had this issue before ?
Has anyone tried any other APIs that use structures and succeded ? If so, I'd love to see the code...