The simplest example could be replacement or at the least an economical complement to the LLM step in cascading VoiceAI, almost every start of the turn a local JEV could make a decision for end of user turn detection:
- is the user finished with their thought? If so delegate to next system(LLM another JEV)
- Should I wait for the user to complete their thought for a few seconds? Set a timer and when time out expires, ask the user to continue.
Like just tuning this model can make it cheaper for voice AI providers to do semantic turn detection over the space of responses.
That was my first insight as well. i have been working on a voice agent for a while and decided to spend about 4 days messing with system one models to see where it could fit in my cascaded system. after 4 days i ripped it out of my system as i found better solutions.. i tried really hard to find a use for it but every time i found there was no need for it so i stopped trying to shoehorn it in. i also spent time with the vision side of things for this thing, and i think that where the biggest gains will be IMO. making decision based on visual ques and in pixel space only was the biggest win.
Yes, i looked in to pipecat stuff a while ago and revisit it often because its relevant for voice agents, but i never ended up using any of their stuff as my custom solutions were better. Also id like to hint that smart end of turn detection is not needed at all in a cascaded system IF you have access to the inference engine of the llm you are using in the sandwich. I dont want to reveal it all but lets just say its no longer a headache or any issue at all. If using 3d party api then yes it helps to have a smart eot system. But i try to keep everything fully local.
Yeah agreed that vision is a nice fit here. I’ve gotten some really impressive results by just throwing xgboost on 25 classification related questions.
System one models do not have an obvious use in voice ai, unless you are choosing from a list of potential utterances (eg embed -> top k -> choose from top k)
Modern stt is baking semantic EOT into the prediction. It makes sense. Conversation has many potential ends, and you have to deal with misprediction of current words that may signal completion, but mel spectrograms contain a ton of information as well.
In your version, we’d have to deal with partial transcriptions and a system that is not natively audio aware. It would also be much more expensive.
Pipecat already does this. It does it for end of turn detection and it's pretty damn good. It's just that it calls the same LLM I have to do it, potentially expensive and prone to latency and not calibrated.
Like just tuning this model can make it cheaper for voice AI providers to do semantic turn detection over the space of responses.