PGP had the right idea, but the system is too hard for the average person.

With "system" I refer to building a web (or multiple!) of trust, based on parameters that you decide upon.

Given that the cryptography would necessarily be asymmetric verifying the sender on a TOFU basis seems like a trivial addition (just sign something). I doubt you can do better than TOFU though unless you tie it to an external ID system (corporate or government or etc issued hardware tokens or similar).