The points provide information about what others think about the submission, which is useful in guaging the status of the comment. It does not have to affect your opinion, unless you let it.
I can confirm this. Halfway through reading his post, I noticed that you had more points than him, stopped to read yours, then read his with skepticism. Possibly it's because you do make a legitimate point, but I'd suspect that your score disrupted the flow of my reading.