This is very interesting actually... I just checked your profile and realized that I have your latest post bookmarked, one that I agree with very much. I still say usability testing has value, but you have to make sure you're using it in the context of the test subjects experience, and as one of many inputs in the design process.
Of course, I work in state government, so the process is a bit different than a startup. However, I still think there's a healthy amount of humility that is gained from watching customers use your design.
I am against testing on users in pseudo environments such as mockups, wireframes, designed but death screens and even most types of prototyping.
Getting someone from the outsides opinion is nothing more than that. An opinion. But it is often taken as gospel and have way to large effect on the actual development of the product/service.
And in my experience whether that is government, large and small organizations and startups. The problem persists.
Don't test something that isn't real on end users.
I agree with you that it shouldn't be taken as gospel. I disagree with you saying it doesn't have a place and shouldn't be done. As with all feedback, you have to give it the appropriate weight and factor in the likeliness that it was an off-shoot.
It sounds like your issue isn't the data, it's how some clients/people weigh that data. Is that true?
Not really. Have been doing it for 15 years worked with Fortune100 and small clients and government. The last 5 years after I was head of my own company, we have started to advice against using it.
And to great success might I add. We have turned the process around and am now advising our clients to save those money for later after the product has launched.
We have had no more problems than we used to and now we have budget to fix it where it matters instead of having to have the client find even more money.
The problem is that giving it the appropriate weight still doesn't solve the problem of transcendence between findings, decisions and design. So the problem is with the entire process rather than people or data.
Usability testing is concerned with improving usability and obviously if your usability doesn't make it useless then you are in better shape than if not.
But there is quite a range from completely useless to useful to best UI you can ever imagine.
And to most clients it really isn't before they launch that they will get the feedback that matters.
And as I have said repeatedly. Improving usability isn't in itself improving your product in any shape or form that makes it more or less successful. Yes if you make it useless then obviously it matters. But no one worth their pay does that.
So instead. Get good designers who both can do UX and the visual side and you wont need to plaster it up with in most cases useless time that could be spend improving the actual product rather than some pre-state of it.
Your experience is so different than mine that I find it quite fascinating.
I hope you continue your successes! After my own fifteen years of product engineering, I personally do not have the courage to push forward with a major design (be it code, product, etc) without some external feedback validating it.
Yeah well the point really is that there is no evidence that testing on end users before the product has launched is going to put you in better shape than not.
There are however almost infinite evidence that no product launches without a bunch of things that needs to be fixed.
But here is the trick. There are nothing that shows that there are more to be fixed if you don't do usability testing on end users.
So that way you have saved yourself both time and money for getting feedback where it matters. After product launch.
And that to me is the most important point.
And I am not saying, don't get external feedback. I am saying don't get external feedback form users on mockups, wireframes etc. They are in no way in better shape to give you proper feedback.
Of course, I work in state government, so the process is a bit different than a startup. However, I still think there's a healthy amount of humility that is gained from watching customers use your design.