Hofstadter's GEB is challenging. Francis Bacon's Popes are disturbing. Aphex Twin is irritating. None of these people hate their audience and they're all pushing boundaries.
Andy Kaufman was challenging, disturbing, and irritating in equal measure and he did hate his audience.
Also, what you personally find offputting, other people may enjoy. For example, I don't find 'Ventolin' particularly challenging to listen to.