Instead of malloc, wouldn't it be safer to trigger an int80 straight to the mmap syscall? You only need to overwrite a few instructions, and put them back in place when you're done. Then you have freshly allocated memory pages for your exclusive use, and I'm guessing things get a little simpler.
WinAPI makes this especially easy too, since you can bypass the allocator and just grab pages; I believe that's how Detours does it, and I know it's how our code (which is public, somewhere) does it.
Maybe, but you still need to avoid invoking the dynamic linker while inside the dynamic linker. And you don't want the code you call inside the solib invoking malloc again either.
Comments
Instead of malloc, wouldn't it be safer to trigger an int80 straight to the mmap syscall? You only need to overwrite a few instructions, and put them back in place when you're done. Then you have freshly allocated memory pages for your exclusive use, and I'm guessing things get a little simpler.
WinAPI makes this especially easy too, since you can bypass the allocator and just grab pages; I believe that's how Detours does it, and I know it's how our code (which is public, somewhere) does it.
Maybe, but you still need to avoid invoking the dynamic linker while inside the dynamic linker. And you don't want the code you call inside the solib invoking malloc again either.