Decision models have the potential to have an even larger impact on the Real World than LLMs have to this point (which is obviously quite large). But the model itself matters less than the product experiences you build around the model, and its very likely that the incumbent labs are treating the area as something more like "oh yeah I guess we can ship that and then forget about it" rather than investing in what building business processes on decision models looks like. Unlike full language models, I don't think the primary business of Typesafe will be serving Jev at API pricing; it'll look a lot more like putting Jev at the center of a much more expensive suite of software.
There's the potential for an inverse LLM play. In contrast with LLMs, all that seems to matter is the model, and the products the labs build around the models are all really samey and boring; the same left panel list of agents, main view agent conversation, right hand extra context, and we're now in the era of everyone creating the same cutesey furry friend on top of all this tech.
> But the model itself matters less than the product experiences you build around the model
This is a very important insight. And it applies to LLMs as well. Very few people were impressed with the capabilities of GPT 3, it was mostly a techie novelty
But then when they added chat on top of gpt 3.5, all of a sudden it was a huge hit. Sure there were improvements in the model from 3 to 3.5, but the biggest impact was from the chat experience
Conversely, when they created Eliza, a basic chatbot more than 50 years ago, people even got addicted to it, despite it’s ai model being something super rudimentary and basic compared to what we have now. The model capabilities didn’t matter as much as the experience the chat created
The instruction tuning between 3 and 3.5 was a big deal. If you prompted raw GPT 3 "Tell me a story about a mouse", it was as likely to continue "2) Tell me a story about a cow. 3) Tell me a story about a rabbit" as it was to tell you a story. I tried to use GPT 3 for work and found its only utility was to get over blank page fear by writing something so bad I would fix it in anger.
It was impressive, but not that much. If you had a basic prompt prefix like:
“The following is a conversation between a human and a helpful AI assistant.
Human: [prompt]\n
AI:”
You would get consistent conversational chat, with the main limit being the model’s limited context window, and obviously frontier intelligence at the time.
I played around quite a bit with davinci-003 and conversational systems before ChatGPT. I kept using this format and API for a while, but at least during the ‘free research preview’ era, found it generally smarter than chatgpt. 3.5-turbo was probably the watershed moment where I moved away from completions on base models; to chat completions.
If you’d like to emulate this experience, spin up a base (not instruction tuned) checkpoint of Llama 1; or a more recent base model. You might underestimate how much ‘intelligence’ you can get :)
Back then the API exposed a lot of controls from sampling to logits, so you could specify a custom stop token (some rarely used Unicode); then switch to near-greedy sampling for more reliable “function calls”, etc; and then switch back to sampling for chat.
In the early days, before releasing something with the API, you had to get your use case approved by OpenAI, like the App Store review process.
I remember those days and I'm glad to see another one who's been in the industry since the very beginning :) I was quite sad when davinci-003 was taken away; still remember the green-yellow-orange-red highlights the OpenAI console would make for log probabilities of tokens and all the cool things we could do with a pretrained model.
In my courses, I teach students about that era by using gemma 4 (pretrained) and showing how you can emulate a conversation even on something that wasn't post-trained for chat.
I think the main benefit of instruction finetuning was that the model is less likely to get prompt injections and more likely to follow the specified format?
Pure davinci was easily confused by what was system prompt, user prompt and its own answers.
To be fair, there are still many jailbreaks for instruction tuned models. Surely less than before, but it could be because models got smarter (Claude often "reasons" that something is fishy even if it appears in the prompt or appears like a system prompt).
> Decision models have the potential to have an even larger impact on the Real World than LLMs have to this point (which is obviously quite large). But the model itself matters less than the product experiences you build around the model, and its very likely that the incumbent labs are treating the area as something more like "oh yeah I guess we can ship that and then forget about it" rather than investing in what building business processes on decision models looks like.
I agree that the potential is enormous, but I'm not sure why it would be Typesafe who figures out the best way to develop and package this capability. They have a bit of an advantage in the amount of time they have been thinking about this problem specifically, but I think it's a big leap to say that they will be able to dominate the category.
A major limitation of directly using "System 1 decision models" (a.k.a., logistic regression, and related uncalibrated classifiers over the output/logit space) in enterprise settings (or other high-stakes settings) is that such estimators are not reliable estimators of the predictive uncertainty in the presence of covariate shifts, and such estimators also lack a means of instance-wise data attribution (i.e., interpretability-by-exemplar), so they're not the ideal estimator for verification, routing, uncertainty over retrieval and tool-calls, etc.
For decision-making with neural networks, we instead need the older idea of estimators of the predictive uncertainty with constraints in the feature-representation space (over training/support of the estimator), as with Similarity-Distance-Magnitude estimators: https://pypi.org/project/reexpress-sdm/
If youre the one or have spoken to someone implementing "AI solutions" inside large companies recently, a decent chunk of it is soft policy enforcement with very basic context. They moved from gemini 2.5 flash lite type models to jev. Which is also why I found the price comparisons to "GPT Astra" on Twitter rather funny.
Right, example being expense pre-approvals (for low dollar value purchases like office equipment). General purpose LLM not required, and the product experience surrounding it does matter (reporting, auditing, fine tuning future responses, etc.).
It really shines when you use that decision routing in industries where they need to automate these routings reliably with very low latency, higer certainly and ,more important, at scale. Take call centers for example. If a voice company needs to automate escalation etc. an llm would be an overkill. That's where system 1 models shine.
If it was simple they would just use an LLM to write code to do it. These decisions are “fuzzy” in the sense that they are difficult to make in just code. They need the “smarts” of a language model to evaluate something, but the final answer is basically trivial (e.g. choose one of these five options, not write me 20 page of text or one shot this whole complex program).
There are a lot of those types of problems in the world, replacing what was previously spotty hardcoded decision logic that worked 70% of the time with something that now gets it right 95% of the time. Even if not perfect, the fact that you can dramatically close the gap means a lot lower cost overall, so it’s worth it.
It is. But the thing is, that is also a lot of the usage for "AI" these days. LLM based search pipelines are being deployed but not really used. Chatbots are a great hit. And then of course, in coding, finance, sales, accounts and audit, etc, it is being used to accelerate output.
These guys also love text to sql. I know 4 people each implementing text to sql for their separate companies. Or text-to-redash dashboard in some cases. Funnily enough, text to sql is one of the things where LLMs are clearly sub-human. Mostly due to lack of easy verifiability.
> These guys also love text to sql. I know 4 people each implementing text to sql for their separate companies.
Text to SQL is a really bad idea, as the agent will just generate really convincing and terrible analyses that lack all business context.
I mean, you could implement some kind of semantic layer, but that's a lot of work, for uncertain reward.
I saw a place which essentially tasked the data analysts/scientists in an area to describe the good tables, and assumptions and necessary context to make decent queries/analyses, and that definitely lead to much, much better (almost usable) results.
As I have said many times, what the world really needs is SQL2Text, rather than Text2SQL.
> I saw a place which essentially tasked the data analysts/scientists in an area to describe the good tables, and assumptions and necessary context to make decent queries/analyses, and that definitely lead to much, much better (almost usable) results.
Yep. After this has been done, the later text to sql outputs works half decent. The major issue I heard was that whenever a small assumption changes few months down the line, while humans are able to "JIT" apply it to each of their queries, these models sometimes lose it in all of the context and so it becomes hard to progressively update anything - you end up having to redo the whole workflow.
And in my experience, even the more complex dashboards requested by product or upper mgmt were stood up within the day by humans. Or the next day (but that was cos some ETL would be scheduled overnight rather than immediately). And I am not sure if speeding up this part would even matter to the guy who requested the dashboard. So I am not sure if this is a major bottleneck people should try to innovate in. If openai does the work for you then great but otherwise. Humans aided by copilot-of-old style autocomplete is probably more than enough.
Needless to say, text to sql is pointless for application queries made during API calls.
Yeah, it's not the development but the maintenance that kills you in both software and other information fields.
The whole text to SQL thing is about data teams being a bottleneck to the business (rather like software people). The goal is good in theory but honestly an Excel plugin is probably a better solution in many cases.
It's pretty high value add when you consider it's replacing human beings assessing each case. This category of thing is present all over enterprises. It was outsourced to BPO shops in India and the Philippines before, now it's going to AI.
But you're right that more complex solutions are still a bit elusive. I use a bot to plan my strength workouts based on my soreness level and what equipment I have available and it's decent, but not perfect.
The simplest example could be replacement or at the least an economical complement to the LLM step in cascading VoiceAI, almost every start of the turn a local JEV could make a decision for end of user turn detection:
- is the user finished with their thought? If so delegate to next system(LLM another JEV)
- Should I wait for the user to complete their thought for a few seconds? Set a timer and when time out expires, ask the user to continue.
Like just tuning this model can make it cheaper for voice AI providers to do semantic turn detection over the space of responses.
That was my first insight as well. i have been working on a voice agent for a while and decided to spend about 4 days messing with system one models to see where it could fit in my cascaded system. after 4 days i ripped it out of my system as i found better solutions.. i tried really hard to find a use for it but every time i found there was no need for it so i stopped trying to shoehorn it in. i also spent time with the vision side of things for this thing, and i think that where the biggest gains will be IMO. making decision based on visual ques and in pixel space only was the biggest win.
Yes, i looked in to pipecat stuff a while ago and revisit it often because its relevant for voice agents, but i never ended up using any of their stuff as my custom solutions were better. Also id like to hint that smart end of turn detection is not needed at all in a cascaded system IF you have access to the inference engine of the llm you are using in the sandwich. I dont want to reveal it all but lets just say its no longer a headache or any issue at all. If using 3d party api then yes it helps to have a smart eot system. But i try to keep everything fully local.
Yeah agreed that vision is a nice fit here. I’ve gotten some really impressive results by just throwing xgboost on 25 classification related questions.
System one models do not have an obvious use in voice ai, unless you are choosing from a list of potential utterances (eg embed -> top k -> choose from top k)
Modern stt is baking semantic EOT into the prediction. It makes sense. Conversation has many potential ends, and you have to deal with misprediction of current words that may signal completion, but mel spectrograms contain a ton of information as well.
In your version, we’d have to deal with partial transcriptions and a system that is not natively audio aware. It would also be much more expensive.
Pipecat already does this. It does it for end of turn detection and it's pretty damn good. It's just that it calls the same LLM I have to do it, potentially expensive and prone to latency and not calibrated.
I think because pretty much every step a business takes is an allocation decision: where to put money, people, time. But it will really pay off only when the models of companies themselves become probabilistic, and the decisions are explainable, at least to some degree.
> Unlike full language models, I don't think the primary business of Typesafe will be serving Jev at API pricing; it'll look a lot more like putting Jev at the center of a much more expensive suite of software.
Precisely this. Should be top comment.
Also, the model moat is understated as training data for these purposes also accrues to the winner, which due to the first mover advantage as well as the distribution advantage you speak of, is typesafe. In contrast to relatively open coding data. Openai anthropic also have that, but like you say its a different business.
At least in theory (TypeSafe has been pretty close-lipped about the details so this might just be hot air, and I think the evidence is a bit spotty) this is false, since they use a different reinforcement training method.
If the word “calibration” in probability doesn’t mean anything to you, the difference isn’t very apparent, but that doesn’t mean it doesn’t exist.
It's calibrated where it's calibrated and not where its not. Their "confidence" score is rather bunk, so it's hard to say. As with other LLMs, you have to eval it for your setting, and if you find it calibrated, then great.
Interestingly enough, at lunch this week it turns out that one of my buddies work colleagues has actually developed a relatively well calibrated version of this, so it's not impossible.
It's probably much harder for arbitrary data though (like most things).
I think you aptly describe the benefits this approach can offer.
What I haven’t seen answered is why it isn’t reproducible within very short amount of time and low effort.
Sure, the large labs are the lazy incumbents at this stage. But any other startups would be able to copy the experience. What’s the barrier of entry that I am missing?
Yes, the best way to think of a general classifier like this is like a smart switch statement. Essentially a "JEV" like thing becomes a sort of programming primitive. Once you see it, it's hard to not get excited.
But even if others surpass them and make better solutions, the fact that nobody was able to see it before typesafe is a testament of what they might be able to come up with next.
I sound like a fanboy but I swear I 'm unaffiliated with typesafe. I was building my own version of this way before they announced JEV (mine was ALE and it was mentioned here on HN for a bit), in use for VR gaming (so one can give commands to NPCs with voice and supports multiple commands in sequence in a single pass), but I missed the "killer usecase" of being a new primitive, like everyone else.
TLDR: There's a lot of value in thinking ahead and seeing the future. The clones are nice and exciting but they give me "I could have built this first, yes but you didn't" vibes. I hope they manage to keep it up and push the space forward again
> [...]the fact that nobody was able to see it before typesafe is a testament of what they might be able to come up with next.
Counterpoint: there are a lot of one-hit wonders, and they vastly outnumber the idea-factory people. This is not to minimize those people, a single idea can be very successful (see Zuckerberg), but it doesn't mean your subsequent ideas will also be great (see Zuckerberg)
You’re correct, it’s a great solution for a very narrow set of high volume classification needs that require very low latency. But that’s it.
I keep evaluating Jev for my product because of the hype, but the reality is that for my tasks, Luna is more accurate, only a little more expensive, and the latency doesn’t matter. I’d rather spend the extra $50 / month or whatever than have to shoehorn in another API and provider, and also lose the ability to change reasoning level and get reasoning summaries for eval purposes.
Did you look into self-hosted open source decision models because they are quite capable also much faster and much easier to integrate and you don't need to pay anything.
I keep hearing Jev is better for accuracy, and still in my use cases Luna beats it. Latency doesn’t matter to me. The cost for additional Luna tokens is negligible for my use case. Idk, I just can’t figure out where to use it instead of an LLM.
LLMs have higher accuracy and are more versatile, but that comes with a cost (slower and more expensive to run). On the other end you can have your own mini classifier something built on top of ModernBert or DIET or even a hash classifier (these are narrow).
Models like JEV fall in the golden middle. Somewhat cheap, somewhat fast, somewhat general.
Many people have replied saying that narrow classifiers (eg bert) are better, and they are better in many ways (essentially free and faster). But... the real world is messy, in my experience it's actually very hard to make a really good classifier that will work across a specific niche domain especially when there is very fuzzy input. And the most interesting real world scenarios are nuanced and overtime there will be edge cases discovered where it fails and that will require retraining and re-evaluation etc etc. It can becomes a full project on its own that eventually collapses into a game of whackamole (improved in some direction but regressed elsewhere).
Things like JEV (or similar models) provide ease of mind, just off load the complexity to it and move on to the next challenge type of thing.
Jev does not have the properties you’re talking about. It’s not very good at general knowledge, it frequently approaches chance level classification on business domains, and there is so much room for someone to do better.
Doing better, I suspect, will be throwing a bigger model underneath and doing something diffusion like (we’ll allocate a whole block to compute rather than just the constrained space) that doesn’t compromise on latency.
>> TLDR: There's a lot of value in thinking ahead and seeing the future. The clones are nice and exciting but they give me "I could have built this first, yes but you didn't" vibes. I hope they manage to keep it up and push the space forward again
This all sounds intelligent and likely, and yet we can come up with countless counter examples where the first mover is not the big winner, and nobody cares about who did it first. There is typically way more value in nailing the execution of a big idea someone else came up with, rather than "seeing the future".
If JEV is here to stay, we can have another split along the "one LLM rule them all" regime
name it worker or implementator or something
- mostly for agent with with spec (not directly for human, as we tend be handwave with vague intent)
- follows the instruction verbatim
- lives in sandbox by default
- basically sol6; but WITHOUT magical situational awareness, lets work in group BS
Yeah, agreed. A model or primitive on its own has no moat and frankly limited value. The paradigm behind "System One" models on the other hand is potentially huge.
Skimming the post, it seems to argue for reconstructing the very rigidity that LLMs let us escape, and that very aspect of LLMs is what made them useful and explode in popularity so much.
One may consider it "rigidity" but many application backends (including LLM powered ones) never needed the full unconstrained capabilities of an LLM. What people needed was the zero-shot capabilities over tasks as opposed to have to train a classifier, span model, etc for every task.
Constraints can be an advantage in many systems. Consider typed and untyped programming languages. Many advantages with typed languages over the latter in terms of development and efficiency.
There's the potential for an inverse LLM play. In contrast with LLMs, all that seems to matter is the model, and the products the labs build around the models are all really samey and boring; the same left panel list of agents, main view agent conversation, right hand extra context, and we're now in the era of everyone creating the same cutesey furry friend on top of all this tech.