Since global karma (your collected karma over time) does not have any real function in the site as far as I can see you could lose it completely and not much of value would be lost, the way to look at it now is more 'relative', if you see a comment thread with a comment directly above the comment you're looking at that you think contributes less than the comment below it vote to lower one up.

After all, the biggest function by far of the votes is to influence the sort order, so the good stuff gets read by the people that skim. A vote is like a little bit of editorial power, you get to have a say in what goes 'above the fold'.