If you had a Likert-type scale, this wouldn't be a problem. I would rate a post 7/10 regardless of what everyone else had rated it. But we just have ups and downs. I think we've all established a sense of what the "right" score is for a given comment. For example, a thoughtful, on-topic post probably has no upper limit for how many points it deserves, while a funny but off-topic post won't get an upvote if it's already got enough points.

Won't this lead us towards a bimodal distribution? Without the crucial knowledge of where the comment's score is, the votes will tend towards extremes: we can't collectively decide when the target comment is near the range that we find appropriate.