After clicking on either the up-arrow or down-arrow on comments, only that single arrow should disappear, not both. Then, you can always at least cancel out your vote if you clicked the wrong one.