This is fantastic. I feel the KV cache system is basically a must for any serious inference hosting, both for the provider as well as the consumer. I guess they have this but just haven’t open sourced it yet.
Gibson's quote "the future is already here - it's just not evenly distributed" continues to hold true, especially today. Amazing author and unparalleled mind's eye.
Additive manufacturing and AI have the potential for significant disruption via empowerment of the individual to solve problems without the expense of purchasing external expertise.
> “He wants to lower the barrier to entry so anyone, regardless of their background check status or legal eligibility [can] own a gun. He wants anyone to be able to design and manufacture weapons...
Empowering everyone to create more problems than solving them.
Reinforces what Karpathy tweeted [1], that AI is effectively tackling 2-5% of work and the gap between those fully exploiting AI and barely touching it is growing. The question remains whether this AI focus on the small % of workers is actually intentional (by tech/labs), or a natural limit of where AI is right now.
Karpathy is also saying the quiet part out loud: they are making progress by brute force. It's a cost driven arbitrage market.
Do I need to spend high prices on 1000 agents to solve this today, or should I wait 1000 days (with compute increasing and prices falling) and hope that I get lucky and get the right answer sooner. Is your competitor half the price but I need 1500 agents to do it in one day cause it's just a bit dumber.
The market for brute force intelligence is infinite - and at some point demand will drive the cost to single digit margins.
I wonder how much of lagging AI adoption is simply due to people not trusting the existing AI vendors, or the architecture of deployment broadly speaking? If we somehow figured out how to get frontier-class models deployed easily locally (and I don't mean the bumbling demos of today - I mean with the sort of fluidity one experiences when using frontier models from frontier labs), would adoption be way higher for structural reasons?
I wish I knew about this algorithm when I worked in databases; it would be well-suited for window functions (even over floats for most users). Now I think I've seen it described two or three different places :-)
reply