Pangram can sometimes flag human-written writing as AI-generated. Though in practice, the reason you're usually running something through Pangram in the first place is that the writing has no depth of insight and you already have suspicions due to other reasons. In that context, what almost always happens is that the person who wrote it will come out and say "no, I wrote it myself I just used AI for editing" aaaand, you pretty much have your answer.
Please stop simping for Panagram. There are organizations that run everything thru panagram and treat it like gospel. Panagram identifies all kinds of older crap as AI. It also can't deal with any human writing that AI has copied the style of.
You were supposed to say that it was always wrong, and that all those "100% AI" blogs about mindfulness are genuine products of humans who suddenly got way more productive recently for reasons completely unrelated to LLMs.
Nevermind! I just went down a rabbit hole on "negative hard feature mining" that has simple origins from early days of object detection and bootstrapping!
I'm sure pangram has to deal with a lot of bullshit to keep retraining on new model output though.
It kinda works? But the marketing around it heavily overstates its sensitivity and specificity, and I know several people who have had their (pre-LLM!) writing detected as 100% AI by Pangram. A number of older novels are also misclassified, and writing styles heavily affect the scoring.
So I think it's probably OK at the corpus level, but it's not good enough for witch hunts.
I thought definitively classifying writing was essentially an impossible problem. Of course, we can often smell it, but again, that's not definitive.