Exactly. I'd still love to see real study on whether unit testing is worth it (which our survey doesn't even begin to address, since it's very hard to measure). I think some good studies on that would be invaluable, but it's nearly impossible to control for all the variables, so I'm not holding my breath.
Failing that, most of the "evidence" one way or the other is based on asking people who test or don't test - and probably most testers would say that testing is great while non-testers would say testing wouldn't catch the kinds of bugs that matter. On the flip side, you have some testers saying it's not valuable (because their test quality is bad) and some non-testers saying they are sure tests would help, but only because they imagine tests to be a silver bullet.
In any case, our survey was just supposed to get a rough feel for how many people were doing testing (and what tools they used). What surprised me most was that even small teams (1 and 2 people) often had a test suite. This flew in the face of the anecdotal evidence we had previously gotten from other startups (almost none of ones we have talked to use tests).
If you look at page 470 of the latest Code Complete, it shows that unit tests have a very low defect detection rate,
Cleanroom software engineering has the best record of quality, but there's no buzz for that now in the ruby community, maybe because it doesnt sell books :-P
Failing that, most of the "evidence" one way or the other is based on asking people who test or don't test - and probably most testers would say that testing is great while non-testers would say testing wouldn't catch the kinds of bugs that matter. On the flip side, you have some testers saying it's not valuable (because their test quality is bad) and some non-testers saying they are sure tests would help, but only because they imagine tests to be a silver bullet.
In any case, our survey was just supposed to get a rough feel for how many people were doing testing (and what tools they used). What surprised me most was that even small teams (1 and 2 people) often had a test suite. This flew in the face of the anecdotal evidence we had previously gotten from other startups (almost none of ones we have talked to use tests).