How about a -1 that affects ranking ONLY to people who enables the feature and follows you?

I mean, if I follow an expert in his area, it makes more sense to me to see that he doesn't endorse spammy-website.com, than to see he endorses "Chrome Beta for Android".

This used to exist - SearchWiki gave you the opportunity to either vote up a search result or to remove it entirely. Both of these actions would affect only your own results - there were vague plans to eventually use the data to improve ranking, but the spam/SEO implications were never solved.

Basically people just used it to kill experts-exchange.com. Usage was tiny otherwise, and the experts-exchange case was fixed when StackOverflow started getting traction and Panda put the final nail into experts-exchange's coffin.