That any particular thing is harmful can be a worthwhile discussion, one that has been going on for centuries and will continue for centuries more. The notion that nothing is harmful is idiotic, so I presume you're not claiming that. As long as we agree that harm to others exists, my point stands. If you'd like the details, I suggest you consult said centuries of dialog. But in practice, it's pretty clear that people can and do come to useful decisions on the nature of harm all the time.