Marshalling return values from hooked method

May 21, 2013 at 6:32 AM
Edited May 21, 2013 at 6:32 AM
Hi,

I need a little help returning from a function that I have intercepted using EasyHook. The function is:
QByteArray QIODevice::read(qint64 maxSize)
... and if you want to read more about it, go to http://qt-project.org/doc/qt-4.8/qiodevice.html#read-2

Currently I have a C# delegate in the form of:
    [UnmanagedFunctionPointer(CallingConvention.ThisCall, CharSet = CharSet.Unicode)]
delegate ByteArray DDeviceRead(IntPtr Obj, Int64 maxLen);
... and the code that run when the delegate is invoked is:
    public ByteArray DeviceRead(IntPtr Device, Int64 maxSize)
    {
        // Call the native method
        Qt.QIODeviceW qtDevice = new Qt.QIODeviceW(Device);
        ByteArray retArray = qtDevice.Read(maxSize);

        return retArray;
    }
... where a QIODeviceW is simply a C++/CLI wrapper that takes the "this" pointer, and calls the function natively; and ByteArray is a struct I have declared that matches the QByteArray that is meant to be returned:
[StructLayout(LayoutKind.Sequential)]
public struct ByteArray
{
    public IntPtr d;
}
If you look at the Qt header file for the QByteArray (http://qt.gitorious.org/qt/qt/blobs/4.7/src/corelib/tools/qbytearray.h), you can see that it is basically empty except for a pointer to a "Data" struct (see lines 126 and 368 of that file).

My CLI/C++ code is called by my Qt.QIODeviceW object above:
ByteArray QIODeviceW::Read(Int64 maxSize)
{
    QByteArray result = Native->read(int(maxSize));
    ByteArray ba;
    ba.d = IntPtr(result.data_ptr());
    return ba;
}
As you can see, it calls the native method, and then tries to build my own "ByteArray" which hopefully looks something like what the target application is expecting. Trouble is, when I return my "ByteArray" struct back to C# and then back to the application, it crashes. I guess the application I am snooping on doesn't like what I tried to send back ...

Does anyone have any suggestions for what I should try next? "[return: MarshalAs()]"? Return an IntPtr? Any general information that might help?