I did this exact same thing for porting a compiler from one language to another with Codex. I run tests at every step, and verified that bytecode output was byte-for-byte identical. I was very impressed at the results, and this is coming from someone who's always pointing out issues with AI programming.
You can run an LLM locally (and distributed compile systems, where the compiler runs in the cloud, are a thing, too) so that doesn't really produce a distinction between the two.
Likewise, many optimization techniques involve some randomness, whether it's approximating an NP-thorny subproblem, or using PGO guided by statistical sampling. People might disable those in pursuit of reproducible builds, but no one would claim that enabling those features makes GCC or LLVM no longer a compiler. So nondeterminism isn't really the distinguishing factor either.
My favorite benchmark for LLMs and agents is to have it port a medium-complexity library to another programming language. If it can do that well, it's pretty capable of doing real tasks. So far, I always have to spend a lot of time fixing errors. There are also often deep issues that aren't obvious until you start using it.
Comments on here often criticise ports as easy for LLMs to do because there's a lot of training and tests are all there, which is not as complex as real word tasks
It's worth noting that many, if not most, games on Steam don't have DRM. You can often just take the .exe files out of them and play. Sometimes you need a polyfill for Steam's client API, but that's usually it.
reply