That's actually a great question. and the answer is yes and no;
While it does disable the caching mechanism for the conversation history (and not for the system prompt, who remains constant), there is a difference between a chatbot with a constant chat history (just exchange of messages) and an agent who uses a large part of the conversation as a type of "scratchpad", sometimes even holding variables value in the beginning of the chat (to be sort of 'stateful'). if these variables change, the scratchpad changes (can be even 30%-40% of the entire conversation), there is a timeout in the cache (Claude gives you 5 minutes of cache for normal caching) or any other change to the exact history - you get a recaching of the entire conversation. additionally, caching still costs money.
The main advantage of the librarian is that is an 'insurance policy' for this caching mechanism. combining it with solving the context rot issue - and you get improved performance at scale.
One architectural tradeoff we are actively working on right now is the latency of the "Select" step for shorter conversations.
Currently, the open-source version of Librarian uses a general-purpose model to read the summary index and route the relevant messages. It works great for accuracy and drastically cuts token costs, but it does introduce a latency penalty for shorter conversations because it requires an initial LLM inference step before your actual agent can respond.
To solve this, we are currently training a heavily quantized, fine-tuned model specifically optimized only for this context-selection task. The goal is to push the selection latency below 1 second so the entire pipeline feels completely transparent. (We have a waitlist up for this hosted version on the site).
If anyone here has experience fine-tuning smaller models (like Llama 3 or Mistral) strictly for high-speed classification/routing over context indexes, I'd love to hear what pitfalls we should watch out for.
Using a v3 onion address as both the cryptographic identity and the NAT traversal layer is such a clean architectural choice. No STUN/TURN servers, no hole punching, you just boot the script and Tor handles routing.
For those who use Tor regularly for things other than web browsing: how bad is the real-world latency for pushing a ~20KB Opus audio chunk over Tor these days? Are we talking a 2-3 second delay, or is it much worse?
The real world delay is about 2-3 seconds your spot on. I initially started with a full duplex version but it was absolutely terrible. Walkie talkie kinda forces the recieve, listen, response from the users so the latency isn't as much of an issue.
Is audio transmitted while it is being recorded or afterwards? Is it played before everything is received or is everything buffered? In the later case, I find it more akin an audio message on Signal or similar, than as a walkie-talkie, which is much more "dynamic".
It's not streamed. It gets recorded, compressed, (voice effects if you want), encrypted on device, then piped through, reverse process, auto played on reciever end.
Also, once it's decrypted and played back, the message gets destroyed.
Small suggestion, maybe you should send a “key down” notice when you begin recording, that generates a subtle sound on the receiving end. This would act as something like a typing indicator on a text messaging client.
This is included in 1.1.4. call interface now displays when other side is recording and optionally configure a preset chimes or record a custom notification sound.
When remote is detected as recording this sound will play if the setting is enabled.
Added link for clarity. Seems like you could get more or less realtime, udp streaming, full duplex communication . Once you have the first part of that built, then adding things like voip or video calls or what have you becomes a lot easier.
The problem is the UDP packets from the application will ultimately be in a TCP wrapper over tor. Which will add even more latency.
I know you can set up mumble over Tor but it's going to have the same latency drawbacks.
Another commenter noted the ability to configure only 1 hop instead of the standard 3. I wonder how much latency would be gained back. I want to play around with this.
Sorry if I was unclear, this stuff does get tangled, lol. I was talking about setting up a parallel veilid connection - a private route exchange between the two ends that goes straight over the veilid network, outside of tor. It's got comparable privacy and security assurance. There only a few thousand nodes/peers right now, but it's gaining steam.
You could use tor to anchor a common relay, then do key and route exchange to establish veilid, then the realtime app uses that for a very secure private route unique to the two endpoints.
From what I can tell, because of the design, veilid would be excellent, comparable to many commercial voip offerings, with 150-500ms latency . More nodes and users would quickly ramp that up. There are a ton of downstream benefits of something like this; faster file exchange for ipfs, bittorent, etc, that doesn't gum up the tor layer, a kind of implicit defense in depth, but also low latency, efficient peer to peer routing.
It'd almost be zero knowledge by construction; you could build a multi-hop escrow style key exchange.
If Musk integrated tor relays and veilid peers throughout Starlink, you could get near-native latency for wherever you connect in the world, more than enough to add timing attack noise and layer in other security features.
STUN/TUN are important because of bandwidth. With STUN the bandwidth used is only between the two connected devices, with VPN like Tor there is a bandwidth cost on all the servers where this data is passing. This is a big blocker for anyone hosting the service on a VPS with a few GB of traffic data per month.
The main advantage of the librarian is that is an 'insurance policy' for this caching mechanism. combining it with solving the context rot issue - and you get improved performance at scale.