Definitely tricky. I solved it with a Python implementation by building up a big jumpbuffer so that the moment I leave Python-land I copy from temporary buffers to the right addresses and then ultimately jump at the entry point of the newly loaded binary. It's tricky and took quite some debugging to get right, but it's proven rather solid now.
Then there’s miscellaneous stuff like cloexec. Not privileged, but atomic.
The binary you exec may load code at the same address you’re using for code, unless it’s PIE. Not insurmountable, but tricky.