If moderation is the problem maybe only allow downvoting? Of course that may bring to the surface a different set of problems (no solution is without drawbacks), but it removes the "numbers go up" incentives.