Don’t know how convincing it’d be in court, but Open Timestamps[1], a free service that operates by publishing Merkle tree hashes to the Bitcoin ledger and can give you independently-verifiable proofs after a while, still exists even if it doesn’t seem to be under active development. (I think Keybase tried something like that some time ago as well, they already had most of the parts in place, but then they decided to use their own something-or-other-coin and I stopped paying attention.)
A good example of using bitcoin for something that was entirely possible with regular old public key cryptography. Matthew Richardson's Stamper has been running since 1995.
In the end both are just digital signatures, and so are "entirely possible with regular old public key cryptography" as you say. You can achieve a similar effect by sending a gmail message to yourself with an sha256 of the document in the subject. The subject and date are included in the gmail DKIM signature. The cryptographic primitives used by Stamper, gmail DKIM and bitcoin are equally secure as a first approximation.
That means the security ultimately rests on the security of the key used to sign it. So do you trust Matthew Richardson to keep is gpg key secure, or Google to keep their DKIM secure, or the difficulty imposed by a proof of work where the amount of work is equal to a nation states electricity supply? I know which I'd choose out of those three, and that's the key differentiator of bitcoin. It is not the cryptographic primitives used.
Stamper signs and automatically publishes hashes of its history. It is a "block chain" in that sense: If Richardson decided to use the keys to backdate something to two years ago, he would have to fake two years of history, and risk being exposed if even one person came forward with a hash he'd signed contradicting his new fake history.
That is presumably one of the reasons that hasn't happened in the roughly 30 years the service has been ticking along.
[1] https://opentimestamps.org/