that's exactly the point, the _lack of_ humans during its evolution is what it has to do with us, a mushroom may be poisonous to the species that it evolved around, while at the same time not being poisonous to humans

Hmm. I'm not sure there are many mushrooms poisonous to other mammals but not to us. We have almost exactly the same biochemistry after all.