So I've dealt with dozens of Fortune-100 companies implementing and using static code analysis tools. They can and will help but in general I feel that these tools are not much more than the code-equivalent of the syntax- and grammar- checker in your word processing software.
I've been doing manual code reviews for a living now (mostly security related) for roughly 3 years now and while I get assisted from time to time by code analysis tools I still find heaps of bugs not caught by any of the tools mentioned by Carmack. The biggest issue for a development shop is to properly integrate these tools and to not overwhelm developers with too much false positives.
I've had cases where a developer got a 1500 page PDF spit out by one of these static analysis tools. After spending two weeks going through everything the developer ended up with 50 pages of actual bugs; the rest were describing false positives. Then I got on-site and I still logged dozens and dozens of security-related bugs that the static analysis tools failed to find.
Edit: also consider that one even needs a SAT solver to even do proper C-style preprocessor dependency checking. A lot of these code analysis tools are being run on debug builds only and then there when the release build is being made these tools are not being run meaning they fail to catch a lot of issues. It's insanely hard to write proper code analysis tools and static source code analysis tools which do not integrate with the compilation process I wouldn't trust at all.
Nowadays with clang there are very nice possibilities for someone to write your own simple checks and integrate them into the build process. But even clang doesn't expose everything about the preprocessor that you might want to have from a static code analysis perspective.
A couple of years ago I ran across an article from Coverity about the challenges that they had getting people to use their tool. One of the most interesting was that the tool would find real bugs, but the developer wouldn't understand that the bug actually was real, and would say, "This tool is crap." Then they would not get a sale.
They had this problem particularly strongly with race conditions. There are a number of checks that they took out because, even though they were finding real bugs, developers were convincing themselves that the bugs were not real even though they really were.
It really does not help that the developers who are asked to evaluate are likely to be the same people who made the mistakes in the first place, so all kinds of defensive behavior are to be expected.
Ha, you're completely right. I've been brought in on several occasions to help my clients decide on which static code analysis tool to use after an evaluation period and I've encountered these situations too. But it helps if it's an outside consultant evaluating the product and not the developers themselves.
Through the years I've learned how to talk with developers about their bugs and how to help them get better in preventing it. It helps that I've been a developer for years too so I know where they're coming from and how they sometimes tend to get all defensive.
I'd prefer not to go too much in specifics due to NDA's and all that; but one simple example that I've encountered:
I've seen simple cases where a function returned a signed integer value modulo x. Which of course means that this function can only return a value within 0..x-1.
This function was called in another function which then did some allocations based on it and the static analyzer complained about integer overflows which were obviously not possible in this case.
Dependency chains in preprocessors is a big pain in general too.
#ifdef A
#define B
#endif
#ifdef C
#ifdef B
blabla
#endif
#endif
This will yield a dependency chain for B of (A,C) -> B resulting in blabla being compiled. With complex code bases this already becomes quickly unmanageable. As far as I know no code analyzer is able to handle this.
And I haven't even started talking about supporting #if constructs (ex: #if X > 2). You would need to implement SAT solvers for that too.
These are just other reasons why seperating the preprocessor from the rest of the language was such a bad idea; you lose all the static code analysis stuff you can do on C/C++ code because of it and preprocessor directives control a lot of things in modern and big code bases.
Yes, the preprocessor is a disaster for static analysis. I don't know anyone who attempts to analyze un-preprocessed code (we certainly don't).
Along with the problem you describe, one occasionally sees stuff like this:
#if A
void foo() {
#else
void foo(int x) {
#endif
Here there are two versions of the beginning of a function definition. There's not even any way to represent that in an ordinary AST, even if you could parse it, which you can't with any ordinary kind of parser. And it's not hard to come up with even nastier examples.
Yeah, even something standalone which attempts to implement a C-preprocessor such that my editor can hook into it and properly resolve the used code-blocks might be already useful. But it's going to be a massive pain, if not impossible, to actually implement it.
I remember the day I actually realized what a clusterfuck the whole preprocessor was when trying to figure out all this precompiled header craziness; it's only there to solve all the horrible shortcomings of the POSIX/C/C++ eco-system as designed and imposed on it by how the preprocessor works. Even Strousup at several occasions admitted that he would like to have the preprocessor removed from C++. That's not going to happen anymore and it's obviously Captain Hindsight speaking here.
> I've seen simple cases where a function returned a signed integer value modulo x. Which of course means that this function can only return a value within 0..x-1.
Actually, signed modulo can return negative numbers... See, for example, this StackOverflow question:
That may be, but I want to have all this information in my editor too so my editor can warn me about it. Take something like a developer just defining a macro like __USE_GNU to get rid of a warning about an undefined function; you just have no idea what box of Pandora you just opened up with all this preprocessor magic going on behind the scenes. Who knows? They might even be re-defining MAX_PATH or something. With code making assumptions on the value of this macro you could open up new bugs. I want all this information right there for the developer in his editor.
Also there's just no guarantees on how the software is built; in a lot of cases that's even outside a developer's reach.
But, yeah, since it's next to impossible to actually get to that state then, yes, sadly the only real option left is to analyze your finite set of builds.
I've been doing manual code reviews for a living now (mostly security related) for roughly 3 years now and while I get assisted from time to time by code analysis tools I still find heaps of bugs not caught by any of the tools mentioned by Carmack. The biggest issue for a development shop is to properly integrate these tools and to not overwhelm developers with too much false positives.
I've had cases where a developer got a 1500 page PDF spit out by one of these static analysis tools. After spending two weeks going through everything the developer ended up with 50 pages of actual bugs; the rest were describing false positives. Then I got on-site and I still logged dozens and dozens of security-related bugs that the static analysis tools failed to find.
Edit: also consider that one even needs a SAT solver to even do proper C-style preprocessor dependency checking. A lot of these code analysis tools are being run on debug builds only and then there when the release build is being made these tools are not being run meaning they fail to catch a lot of issues. It's insanely hard to write proper code analysis tools and static source code analysis tools which do not integrate with the compilation process I wouldn't trust at all.
Nowadays with clang there are very nice possibilities for someone to write your own simple checks and integrate them into the build process. But even clang doesn't expose everything about the preprocessor that you might want to have from a static code analysis perspective.