RawDetours not working on Windows

[vars1ty]

Creating a new standard .exe project on Windows with the RawDetour example from the docs and running it in release-mode fails.

The only mitigation for this that I have found, is to use opt-level = 0 - which for obvious reasons, is not exactly the best approach.
StaticDetours seem to be unaffected, GenericDetours share the same issue.

@Hpmason

[Hpmason]

Sounds like that may be UB in the source code. I know I was testing MIRI on one of the dependencies (I think it was slice-pool, but I'd have to go back and double check) I forked and it flagged a couple things, so it may be that. I've been wanting to use MIRI on this repo, but I'd need to override all FFI with some sort of mocks to actually run it.

I'll try to look into it this week if I have time. If I can't catch it with MIRI, I may have to objdump debug vs release builds.

[vars1ty]

Alright, if you make any progress, lmk. I tried to compile and run with the GNU toolchain for x64 and that seems to work, but has its own set of issues which make opt-level = 0 a better workaround.

[vars1ty]

Well I found 1 workaround: Store the hook in a structure with other data if you need that, or leak it to a &'static.
So doesn't seem to be a direct issue with the code, but more with how Rust handles memory?

[Hpmason]

I looked at this a bit back in Sept, but got busy. Had some time and am looking into this again. I can't seem get Box::leak or mem::forget to work in release mode, do you have a code example of it working for you?

Also spent a fair bit of time trying to get MIRI to work too, but because functions aren't directly allocated by the rust allocator, it's treated as unallocated memory by MIRI (and it can't know the size of the actual function either). May be able to get around it with a custom allocator that I can "fake" allocate functions before using function pointers, but that may not fully work with MIRI either.

[workingjubilee]

Well I found 1 workaround: Store the hook in a structure with other data if you need that, or leak it to a &'static. So doesn't seem to be a direct issue with the code, but more with how Rust handles memory?

I am not so sure. The RawDetour code notes it is "vulnerable" to inlining. It is likely that the issue is inlining and dropping to a function pointer, storing that function pointer somewhere, and later operating on that is what is solving this, as that can make it become too hard to inline.