Yeah, if you powdered pure caffeine and did a line you'd be in a bad way too I'd guess? The ld50 for caffeine is only roughly double that of cocaine. Pure nicotine has half the ld50 of cocaine. It just no one boils down tobacco juice into oil and drips it into their eyes for a bigger hit. At least not more than once. Essentially all plant based recreational substances found in their dilute natural forms absent any breeding programs to concentrate intoxicants, are probably relatively safe. It is the processing that is an issue. The alcohol buzz from eating overripe fruit is probably even innocuous.

Edit: there had to be exceptions, and I think jimsonweed is probably one now that I go looking. The recreational dose even in raw form is way too close to the lethal one.