IMO by asking Jev underspecified questions like this, you're essentially using it as a random number generator (similar to the dice example). On actual NLP problems (including ones with uncertainty under human review) it does appear to be well calibrated: https://leonardgrazian.com/blog/jev-calibration/
In the dice example[0] and in this one, the expected output is a distribution e.g. "1: 16.7%, 2: 16.7%...", not a random generation. I agree that in implementation, the architecture is not designed to accurately calculate or incorporate any known uncertainties like these, but the task itself is completely reasonable and arguably trivial.
IMO the lesson here is: even in trivial cases, Jev's outputs are just ~reasonableness scores which do not correspond to actual probabilities. They should not be treated as actual probabilities without careful calibration and plenty of meta-uncertainty about how well that calibration extrapolates.
The problem is, most of the value proposition of Jev is that it gives you the probabilities without doing that, which it doesn't.
I think you hit on my core criticism of this and similar analyses: LLMs are a tool, and using them to draw from a distribution is misuse of the tool. It's not what it's designed/tuned for. Asking follow ups like "is Jev calibrated when I mis-use it?" is asking the wrong question. The right question would be, "is Jev calibrated for expected use cases?". And based on some initial exploration, I do think Jev is calibrated for common natural language questions
Jev should ideally respond in a non-random way, it should just list out the probabilities.
I suppose trying to interface with the model like this is like asking an LLM how many times the letter E appears in a word - just not the correct way to ask that given its model
Hallucinations still happen occasionally, but we bias heavily toward high-confidence findings so noise stays low. On typical projects we surface a few hundred coordination issues that are real, observable conflicts across sheets rather than speculative checks. We’re actively improving precision by learning from every false positive customers flag. We show you the drawings, specs, etc. so you can verify it yourself not just trust the AI.
We do extensive preprocessing to ensure AI receives accurate context, data, and documents for review, and we’re continuously refining this, so accuracy keeps improving every day. Right now the accuracy isn't super stable yet across projects, but we've had findings with > 90% accuracy results
