In that case, why do you care about the voting? You will always get the group-think out-voting the carefully reasoned but unpopular. All you need to do is read - perhaps simply skim - all the comments. Find users whose opinion seems to be worth reading, and then seek out their comments.
Removing down-votes won't help the problem you've identified.
That's what I increasingly find myself doing. The downside is that you get less and less exposed to new material; "old" HN would usually surface e.g. tptacek's comments, but there would also be some highly-voted and insightful comments by relative nobodies who were experts in the particular field being discussed. This still happens, but reading the highest-voted comments is becoming less and less worthwhile.
The problem is that down-votes have account consequences. They aren't the mirror opposite of an up-vote. Ten upvotes and a post is just popular, ten downvotes and you might get banned.
If moderation was a flag that some people cared about but that left everything intact then people could ignore it if they wanted.
Removing down-votes won't help the problem you've identified.