To what degree is this possible to check statically?
It feels like at least simple breaks of the ABI rules like this can be detected somewhat statically. The author already started with a very simple and incomplete version.
In general, I wonder, are there any (many?) static analyzers for assembled binaries.
Compilers do inter-procedural register allocation and use custom calling conventions for local calls (where “local” can be quite large with LTO), while preserving ABI externally. This means that clobbering a callee-saved register without saving/restoring it in the same function is not necessarily a bug.
Curiously, I found a register clobber bug in the NaCl cryptography library today. Apparently, they used a custom assembler-preprocessor (qhasm) that avoids certain classes of bugs and aids with porting, but while the tool seems to actually model the register in some way, it does not treat it as callee-saved.
You could write an arbitrarily complex analyzer to try to find violations but I suspect that the halting problem means that you can never be sure you've found all errors.
I think that either crude heuristics (found two bugs!) or UBSan style instrumentation (finds all bugs in code executed under test) is the best set of solutions.
It feels like at least simple breaks of the ABI rules like this can be detected somewhat statically. The author already started with a very simple and incomplete version.
In general, I wonder, are there any (many?) static analyzers for assembled binaries.