Really dumb question from a software guy. Why aren't the labs burning their frontier models into chips already? Seems like the performance gains and cost per request would be worth it. That said, I understand neither the economics nor the physical challenges to doing this.
Model SOTA moves faster than chips can be designed or produced. You'd need to commit to a particular model for years to get payoff while still burning buckets of money producing new SOTA models to keep up with the competition.
It's why everyone and their dog runs these things on GPUs. When a new model supercedes the previous one, so long as you've got the memory for it your chips aren't obsolete.
I'm looking forward to someone picking a model to be "good enough" (say, qwen 4.0 or something) and selling them as peripheral hardware
At this point, LLM's are "good enough" for all kinds of tasks. Instead of making them more capable, now the efforts are making them smaller and cheaper.
IMO this is the dream scenario! Cheaper and faster at the current level of capability gives us incredibly useful tools without the worst of the risks people fear. (Though there are certainly already great risks at the current level of capability as well.)
Ah that makes sense. I'm skeptical that AIs doing things on their own will be competitive with humans using AIs to do things. But I'm not incredibly confident in this and I think it's sensible to think the opposite. But to me, I just think about all the things I can accomplish with the aid of very good, fast, and cheap inference.
I think the sad part is that whether it's any good or not, it will be cheaper, and that will be enough even at current performance levels to cut a huge number of jobs. I think people exist in the present moment largely because legal questions about liability are not settled well enough for executives to start the great purge. I believe on some level my continued employment is simply because people in leadership positions would be incredibly unwise to not have a human to blame when things start going wrong. People are expensive insurance policies for the time being. Once there is some legal framework to absolve executives of their own culpability or they figure out a good way to insulate themselves from personal responsibility, then the real "fun" will begin.
The problem comes in to how many bullshit jobs we'll have to create to ensure that enough people can still earn a living to ensure that we don't have riots and revolution in the street.
Companies being profit seeking entities that are actively hostile to social wellbeing would gladly put all of our money in a machine money making loop and leave all but a few humans out.
Like mostly all tech, AI is destined to get cheaper with time. We're still pretty early on and are experiencing a hardware crunch, which will ease eventually.
AI doesn't have workers rights, doesn't burn out, doesn't get sick, can be instantly onboarded, etc. The second we can be fully replaced with AI, we will be. Plan accordingly. I'm pursuing FIRE and considering moving into a trade.
> The second we can be fully replaced with AI, we will be. Plan accordingly. I'm pursuing FIRE and considering moving into a trade.
Doesn't matter what plans you make, the impact will be across everyone!
Moving into a trade won't help, because the supply is doubling while the demand is lowering. Fewer people with money to spend; they'll fix their own damn toilets if the decision comes down to "buy food" or "hire plumber".
You forget that once all lose their job will all go into trade. Means there will 1000x more people to compete with into trades, which means race to the fucking bottom and lowest salaries if you can even find job. Good for real estate owners, cheap repairs. But probably won’t get any rent paid because people can’t find jobs.
Idk, has china printed houses successfully? And if there are reports of that can we trust that the information hasn’t been gov altered to make a failure seem like a success?
I’m not “leaving them out” i literally just don’t know anything about them.
And yes i am skeptical as a default on this subject. But still - photos don’t actually tell the whole story. What was the cost? How was the project schedule? Etc.
I was skeptical when they first started doing it 20 years ago.
Now, right wrong or indifferent, it’s a known entity there - for a LOT of projects. Enough of them exist now that…I’m skeptical they could find enough stupid people to keep paying for it. I know the government has a hand in a lot, but that only goes so far, for so long…
training data for physical tasks like that is much, much harder than just collecting the entire internet. It'll be resistant to AI much longer than knowledge work will be
We already have construction codes. Fitting techs with AR headsets could gather more data. Existing structures won't be the first step due to variences. The best place to implement this is modular building in controlled automated environments. Even if we enhance the current productivity, we will create a problem due labor supply driving down incomes, especially if people are coming from the other jobs affected by AI.
You can barely capture folding laundry using task data, and you're vastly underestimating how much independent decision making the trades do in their regular work. If I'm honest this whole thing reads like someone who's never been exposed to the actual work.
Out of business, but not necessarily out of existence.
If there is really nothing for humans to do, whoever has control over (not necessarily ownership of) enough robots and AI to directly maintain and grow their collection of robots and AI, has something functionally equivalent to a breeding population of the stuff. (If they can't make more of themselves and humans made the initial batch, then that's a job humans can do so the initial condition has not been reached).
None of this helps people figure out how to look after their own interests in the meantime. The rest of society may carry on as today without AI, akin to the Amish if we're lucky (rejecting further developments unless good for us) or like Pol Pot if we're unlucky (reject anything that nerds like because nerds liked AI and everything breaks down).
On the other hand, the institutions may simply fail to handle reality, like in the Great Depression.
Financial Independence, Retire Early. It's a whole subculture online, based around the idea that if you can grow your investments to ~25X your annual expenses, you can safely stop working and live off those investments.
We've had other "apocalypse" scenarios before (industrial revolution and manufacturing automation) where we successfully transitioned to a new way of living and being productive. The optimist in me says that there's nothing special about AI this time around.
In the current economic system.
There is no natural law that states that only the people lucky enough to be born rich and/or well connected to real the rewards.
Yes this is why I don't invest in hardware. TPU is better than GPU when you have a good model already. The models are already good enough to sell as an accelerator card made as a TPU. I would buy an Opus 5.5 card.
No, there still plenty of tasks that LLMs can’t do well or at all. Yes, we should absolutely make models cheaper but there’s a long way to go until they’re “good enough”. And I’m talking about the frontier closed models too, not just the open ones. So it’s not an “instead of” position we’re in. It’s both.
> Model SOTA moves faster than chips can be designed or produced.
From what I remember working in that area the hardest part is getting masks for a design. Masks were developed in the span of half an year. Masks also reusable, they can be mixed and matched and this is why fabless companies work with fabs to produce specialized masks for them, it saves time for consumer to have masks for some macroblocks prebuilt.
Given some amount of work with the fab before main pipeline set (I think a year long process), one can then spew LM-on-a-chip in six months or less and much more than 2 per year, because there can be several LMs in pipeline.
FPGA speed is a common misconception. They often have run at quite modest frequencies, and come with limited memory capacity compared to a GPU at same price range.
Speed (clocks speed) , is dependent on your design layout, and for any resonably advanced layout, it takes lots of knowledge to push the clockspeed beyond 200Mhz on 'consumer'/prosumer models. (Compare to a few GHz for GPU/CPU).
FPGA speed shine where they can pipeline massively parallel calculations through pipelines with minimal lookups.
Wrong. This is one area where FPGAs have an insanely unfair advantage compared to CPUs and GPUs. Yes the SRAM is limited but you have so many individual blocks and all of them come with dual ports and getting the maximum frequency out of block RAM is much easier than getting the maximum frequency out of programmable logic.
If you wanted the highest possible memory bandwidth while being free to look up hundreds or thousands of independent memory addresses at the same time you're better off with an FPGA.
E.g. with an Efinix Titanium Ti180 you could hypothetically have 2560 simultaneous memory requests per cycle all pointing at a different address and process those requests at 1 Ghz.
Yes you are right, I didnt express what I meant very well.
looking up static values is quick and easy. When you need to lookup results from previous stages of pipeline rather than just feeding them forward, thats where I run into trouble. But I am a relatively fresh FPGA designer, so I am sure it can be done. And I probably need to level up my boards a bit too.
They have trillions worth of obligations to their partners in terms of compute purchase agreements and equity. Burning models to chips isn't really something they can blow money on just for funsies, they need to be able to justify it. The above comments are hypotheses as to why they haven't yet.
It isn't just a matter of speed, it's also a matter of model quality. If they take 6 months to burn Fable to chips, and it takes 2 years to break even between design, custom fab, energy savings, etc, are those chips even worth running when the new models that are running on GPUs at that point are producing 10x better quality results?
Sure, your 2.5 year old models are running faster, but you can't drop prices on them without pushing the break even point further out.
If the cost difference isn't incredibly significant, will people even want to pay for the 2.5 year old model, or will they get more value for their money paying more to get better results from the newer model?
There's a lot of open ended questions that I don't have the insiders knowledge for to suggest whether or not such a capital outlay would be a worthy investment.
My guess is that state of the art stuff will stay on GPUs and models burned into chips will be for "good enough" applications that people are still teasing out. Probably highly specialized models in automated sensor units and such.
FPGAs are FPGAs by virtue of putting on the chips vast, vast arrays of wiring that can be controlled by software. Any given utilization of the FPGA will leave large fractions of the chip resources unused. If you've got a highly stereotypical use case FPGAs will have a "highly stereotypical" set of components being unused, where it would be better to use that space instead to do real work. A lot of people only see the "pro" side of the FPGA proposition without realizing they come with some very substantial "cons" that are intrinsic to the way they work.
I guess that's why they work in particular niche spaces like a synthesizer where you have a max of 8 voices and every voice goes through the same pipeline (osc / filter / env / amp) and everything is necessarily running all the time. In that sense I suppose they're very good for modelling any kind of analog circuitry?
Even then, while there are some amazing FPGA-based synths available, companies like Korg just put their code on a raspberry pi and call it a day. The same is true for emulators (SNES Mini etc. are also just raspberry pis under the hood iirc)
You don't really get an FPGA for capability. CPUs are much more capable, and they're general-purpose so they can do absolutely anything with about the same efficiency and just a little more code.
You get an FPGA for timing. They're less capable, but (in many common design architectures), they output their results once per clock, every clock, on time, every time. If you can hit a fabric clock of say 100MHz, clocking all the weird logic you can stuff in there, it gives 100 million outputs per second, never skipping a single one for any reason (short of total failure). The penalty is that making a small change to your desired "program" can be very expensive, and many things won't be realistically possible at all. Or at least won't fit into a part that you can buy. But things like audio, video, and high-frequency trading love being able to guarantee timing.
(Of course there are other ways to write your FPGA HDL, but that's one of the more common ones. And you do see DDR-style clocking, and similar, every now and then.)
> with about the same efficiency and just a little more code.
It depends. For some things, CPUs don't even come close. An XCVU13P FPGA can handle 1.2Tbps of full-duplex Ethernet @ 1 billion pps. And that part costs less than a grand at moderate qty, and with significantly less power consumption than a CPU that'd be capable of operating a dataplane at these speeds.
The point I was trying to make is that the CPU is a general-purpose creature and doesn't really care what you want it to do. If you had a CPU that could handle 1.2Tbps of Ethernet packets at 1Gpps, it could do a whole lot of other things involving 1.2Tbps of data flow too, very easily, if someone wrote the software. And more. (But you're probably not getting 2.4Tbps out of it, no matter what you do.)
An FPGA can not. There's plenty of things that those XCVU13Ps just can't do, or would do worse than a $1 microcontroller. (Setting aside for a moment implementing a CPU inside the FPGA... which does actually happen in just about every large-enough FPGA design, which is its own discussion....)
I'm not really sure what you're saying here. An FPGA is inherently more general than a CPU because it implements any digital logic including a CPU. They're just more expensive and have clock speed limitations. Saying if you had a CPU that could do 1.2Tbps is just as silly as saying if you had an FPGA that easily operated at 5GHz.
The issue is really the extreme difficulty of programming them, especially for anything sequential (rather than combinatorial). Yes, you can just put down a CPU core, but that's really slow compared to a real CPU and a headache in a lot of ways. (But still is a totally valid and used technique.)
To give a concrete example, suppose you are doing some image processing and you need to write out something like a BMP image file from the pixel data in memory. Suppose, also, that it's dynamic-ish and you can't really precompute the header and just copy it around. This is a horrifying nightmare to do in an FPGA, generally involving writing a custom state machine and populating different bits of the structure on each clock and so on and so on. (This is not an arbitrary example, sadly; I've had to do almost exactly this.) You can do it but it's just plain hard.
On a CPU you're done in five minutes, for a BMP file.
Now imagine you want to write out a JPEG or PNG. On a CPU this is hard enough that you'll probably grab a library... and then still be done in about five minutes. Maybe thirty if something's unusual or you're being extra careful. Maybe a day or two if you decide to write it yourself (don't do this).
On an FPGA? Holy hell you're in pain. You're either going to spend $$$ licensing something or you're going to be there for weeks.
That is why there are jobs that FPGAs don't do. They're not technically impossible, they're infeasibly impossible.
If it is your core business, yes, great, do them. But you'd never decide to stop making wasteful unoptimized BMP files on an FPGA "just because". It's a major chunk of work. But on a CPU, five minutes.
The current largest FPGA, the AMD Versal Premium VP1902 has 18.5 million logic cells. That's not even enough for the smallest whisper.cpp model (75M).
You'd have to order hundreds of thousands of them (or millions) to serve even a single copy of a frontier model, and at that scale inference quickly becomes starved by the speed of light.
Well, you'd use BRAM to store model weights, not fabric. But still, you only get a couple hundred MB for probably close to US $100k per chip.
It's likely that the major FPGA vendors will soon announce parts specifically architected to support LLMs and similar models. But the current generation isn't suitable for that at all.
Would BRAM even have enough bandwidth? The reason I quoted logic cells is because that's the way to get instant throughput, which is practically the only reason to use an FPGA over something like a TPU.
I think so, because memory bandwidth really comes from bus width more than clock speed. You can construct 36-bit wide BRAM arrays with bus width comparable to HBM, just by specifying multiple BRAM arrays in parallel.
But that's not an efficiency an FPGA provides in the first place.
Additionally, it's not clear how well large mask roms scale. For instance the Nintendo switch cartridges were expected to be mask roms, but instead are Macronix's XtraRom technology, which is essentially a flash cell array made denser by removing the erase functionality. So basically the die gets manufactured with all bits at the same state, a late manufacturing step either empties or fills the floating gate of the bits you want different, and then it's treated as pretty close to a mask rom. It's not even clear if the bits can be changed without a bare die, a floating probe array, and specialized hardware. Though, like flash the electrons in the floating gates will eventually tunnel and cause the data to bitrot.
So from that it appears that even at the tens of millions of chips volumes that would make sense for essentially whatever size of maskrom, the memory manufacturers tap out at 128megabit for a mask rom chip, and push you towards something flash esque.
And at the end of the day, flash without the erase functionality is pretty damn close to a mask ROM, and lets you write it near the end of manufacturing rather than at the something close to the metal 1 layer.
8 months ago, Taalas claimed https://taalas.com/the-path-to-ubiquitous-ai/#:~:text=Upcomi... that "Our second model, still based on Taalas’ first-generation silicon platform (HC1), will be a mid-sized reasoning LLM. It is expected in our labs this spring and will be integrated into our inference service shortly thereafter. Following this, a frontier LLM will be fabricated using our second-generation silicon platform (HC2). HC2 offers considerably higher density and even faster execution. Deployment is planned for winter."
Nothing was released in spring, and 2 months ago AMD announced their acquisition of Taalas. That doesn't exactly inspire confidence that their frontier LLM will arrive as promised.
> AMD announced their acquisition of Taalas. That doesn't exactly inspire confidence that their frontier LLM will arrive as promised.
Why not? AMDs chip design experience and production capacity are magnitudes larger than a small startup. Assuming that they acquired Taalas for their technology, I don't see a reason why they couldn't.
I think this company was recently acquired by AMD, so hopefully they'll start getting some this into production. I know OpenAI was working on model-on-a-chip too.
Terrafab is just going to have their entire capacity bought out. Genuinely. Demand will increase to exceed supply no matter how high supply is right now.
Most responses here are along the lines of "model capabilites move too fast to build hardware for".
I think the fact that there are plenty of 1yr+ old models on openrouter serving hundreds of billions of tokens a month shows that there's plenty of use case for models that are "good enough. Cerebras' entire business is serving older models at high speed. I would happily use an opus 4.7 at 15k tokens per second. The intelligence per second of an ASIC still makes sense even with rapidly evolving models.
"Intelligence per second" is a striking phrase! I'll be turning it around my head at a moderate IPS until I hopefully make something of it.
But you're right in sense: moderate intelligence at superhuman rates (and presuming moderate energy usage) is very compelling compared to an intelligence that takes 1000 years to return "42"
I look at it this way. A year ago is a long time in AI terms, but not that long. Those models were already decent for the tasks we're using SOTA models for today.
So imagine taking a year-old SOTA model and running it at 100 tokens per second on an edge device. That's enough to feed a screen's worth of content through it and power decent multilingual message suggestions on IM.
Imagine running it at 1000 tps. That's enough to reparse that screen mid-keystroke, and give you semantic autocomplete in text. Or fully general "the phone has a good idea of what you're attempting to do" context at all times.
There's many, many new classes of features that will open up if decent enough models can be run on edge devices at 100+ "intelligence per second".
Or vice versa... 1000tps == 1tps via solar power + a capacitor (or whatever).
I'm poking (lightly) at the locallama game, have a fancy MBP5 with gobs of ram (so I can demo/trial locally) and have been semi-waffling between whether to chase a mini or studio for local "always on" type stuff.
My outcome was "CapEx v. OpEx", and dropping another $5k for an aluminum cube buys a lot of OpEx (eg: just trickle-drip HF/OpenAI credits to a raspberry pi or VPS orchestrator rather than trying to do the inference locally), ie: $5/mo inference for 1000 months.
HOWEVER, there's definitely a role for that 1-10 TPS type "ambient inference" that I wouldn't mind sustaining on any sort of always-on / local / private compute. My main email address is still on [email protected] and their spam filtering has gone to absolute shit.
Being able to have the always-on mini (local, trusted, no private data leaves my control) poke at the IMAP/email and thresh it into SPAM/HAM/Personal/Political ... random spot check, I'm getting ~5 emails per hour, and that's completely tractable for staged low-med-high processing. (Subject + rules.py? Subject + Body + LocalSlowTPS? Subject + Body + LocalDeepTPS? Subject + Body + RemoteLLM?)
Even if you did 10000tps of "jev" that's an incredible value... not quite "Literal AI Packet Router", but as you're dancing aroud saying... "Speed is a Weapon"
Look into "OODA Loop" => """The OODA loop is a four-step decision-making model—Observe, Orient, Decide, Act—created by U.S. Air Force Colonel John Boyd to help leaders make fast and accurate choices in chaotic situations. // The main goal is to cycle through the loop faster than an opponent or changing environment. By operating inside another person's loop, you create confusion and outpace their ability to respond. While initially designed for aerial combat, it is now widely used in business, sports, and crisis management."""
It is still a bet. Is the current model still going to be good enough next year? We have no idea what will change. If the change is minor improvements than current fable on a cheap is cheaper and better than next years sonnet on GPU. However there are plenty of things people want that maybe they will deliver and suddenly I wouldn't touch today's fable when I can run next years sonnet instead.
But I think it's a good bet. I think that in two years, if I can get opus/sonnet 5.5 or the gpt-6 models for much cheaper and faster than whatever the "frontier" is at that point, that this will probably be a great trade for most of my work. I certainly don't know that for sure, that's why it's a bet, but it's what I think right now.
I wouldn't quite say that about any of the open weight models at this point. But I'm hopeful that will change in the next generation or two of those models.
Also "faster" indirectly means "more intelligent" with reasoning models.
For example, Opus 4.6 Max was somewhere between Opus 4.7 Medium and High in some benchmarks, but it was slower. If there was a way to run it 10x faster, the economics would be different.
From what I've read, not only are some labs doing it (other commenters already mentioned).
But it's complicated for other reasons, one being that the number of parameters for frontier models (especially with MoE models) are so high, and not always utilized (once again, thanks to MoE) that it would actually be incredibly cost prohibitive, if not impossible, to attempt to make giga-chips that would allow running it.
I definitely do believe that we will see more and more specialized chips over time, but putting the entire model on a chip is still a ways away.
I believe Taalas has a heavily handicapped llama 8-billion parameter model. And it still pulls >200W to run.
I can't imagine how anthropic or open ai would be able to burn a multi-trillion parameter model on a chip, we just aren't there yet.
Yes, and startups do exactly that. Check out Etched https://www.etched.com/ where they made a Transformer specific GPU (basically a form of ASIC) where they bet that transformers would be the dominant GPU architecture for running AI / LLM workload.
The bottleneck, for inference at least, is memory bandwidth. And that you can't make any faster by making it specific to your model.
So companies try to maximize the memory bandwidth they can get, balancing tradeoffs of power/area/programability of their chip. Right now they feel like the economy on power/area is not worth the decrease in programability/flexibility.
Presumably though the kernel has a pretty specific set of operations done against the weights in memory. Burning the weights into the memory with local memory cores capable of the kernel operations would be a lot more efficient than round tripping busses.
The primary constraint isn’t likely what’s possible to do, but that the kernel and weights are too variable right now and the patterns too poorly established to bake into hardware accelerators yet. Margin pressure is also not there yet.
I suspect as the marginal utility of the frontier improvement settles into diminishing returns (I suspect we are there already tbh) baking hardware models with ROM, working set, and kernel cores collocated will be the frontier space as the goal will become reducing capital spend to utility levels rather than research levels.
Once someone has a model that is sufficient for almost any practical use, making marginal inference cost effectively zero will be the competition frontier. I do shed a tear for all those lonely data centers as compute densities will almost certainly make most of them a terrible investment.
You're starting to hint at compute-in-memory as a general replacement for CPU/DIMM layouts. That could be useful for far more than LLMs, world models, or any sort of AI. It takes a bit of a different software development stack than a standard architecture though.
> Burning the weights into the memory with local memory cores capable of the kernel operations would be a lot more efficient than round tripping busses.
Sure, but now we're not talking about just burning the weights into the chip, but also designing a new architecture that has memory local to each core. A new architecture would then require a new programming model, which means new inference stack, which may mean new training stack.
I don’t think it would require a new training stack, and I’d imagine it makes more sense to distribute cores with memory. The cores can be simplified to the functions of the kernel since the inference kernel can be expressed as a reduced set of optimized functions in the pipeline rather than a general CUDA core. If the model is burned into ROM, the compute pipeline can be baked into the core.
I think one limiting factor here is that Big AI cannot focus on building solid, stable products that do a job well. It's okay if it happens incidentally, but doing it on purpose would be self-defeating.
Their stock price, be it public or estimated, is heavily pricing the notion that they are first and foremost Growth companies. Therefore their focus must remain on ever better and greater things. If they lose focus and get distracted by lesser endeavors, their valuations crumble, their ability to raise capital vanishes, and their runways collapse before they ever have a chance to reach their end goal, whatever that may be.
That means the boring job of productizing AI models into reliable systems that won't vanish in six months is left for a smaller company willing to pick up the crumbs. Unless they get acquired by Big AI before getting it done.
As a SW person (working with silicon folks though), I was always interested in the same idea. So I just vibecoded a little compiler from LLM (config.json, weights) to RTL (happy to share if you want). Then synthesized it via yosys.
I picked TinyStories-1M, synthesized via yosys against SKY130 PDK. It's GPT-Neo with 8 layers. At int8 it's ~30mm^2 per layer (including kv caches, multipliers and all the attention stuff). So it's 250 mm^2 for the full toy model.
I'm kinda surprised, I spent no time, so if you want to etch a toy model, you can do that (you can probably reduce the area significantly). If you want a mask for that is probably going to cost 100k though.
Would you pay to crystalize one of today's models in silicon so you can use it in 2028, or would you wait for another 6 months to see how models improve before pulling the trigger on that kind of commitment?
If I controlled a budget like this, I think I would put some portion of it toward paying to crystallize one of today's models in silicon, yes. Not 100%, but I do think this makes sense to invest in at this point. I would not have said so a year ago.
Naive question: would flash or some kind of write once memory be dense enough to replace a mask ROM? I'm wondering whether it is feasible to manufacture "blank" chips at the semiconductor factory that have a fixed architecture, but are model-agnostic. These chips would get the then-current weights burned in on first use. It's probably quite wasteful, but it could keep the same chip design alive for a longer time.
DRAM seems to sit in the economic sweet spot of density and bandwidth. I'm not sure if any other storage technology can provide non-volatile or one time programable storage with either the same density and bandwidth. Although I must say, it seems insanely wasteful to use RAM to store model weights, there must a better solution.
The closest anyone has gotten is Cerebras with their Wafer Scale Engine. It uses SRAM embedded with the compute. A single chip is an entire wafer, but the headline spec, how much ram, only 44GB, which is tiny for the silicon area used.
I understand traditional IC production workflows are ludicrously expensive, and glacially slow, but surely at some point the economics are going to tip in favour of mask rom.
Say you setup your foundry/packaging/ai chip facility. You come up with a new set of model weights. Run your CI/CD pipeline to produce a new mask output. The only thing you've changed are the assignments of the bits, this is extremely low risk change. The new masks should be completely interchangeable with the current process.
You produce the new masks, swap them into your foundry process, and all of a sudden your new chips have the latest version of the model. This would probably manifest itself in the form of inference providers having yearly / bi-yearly "updates" to their models as new hardware is brought online.
Tiered subscription levels would gate keep access to the latest and greatest model, cheaper subscriptions will be limited to older versions of the model, and so on, until running the hardware is no longer economically viable (no demand/running costs exceeding what the market is willing to pay).
I think Taalus is a bit reluctant to publish precisely how they do it but vaguely:
> We basically have an architecture where we are embedding the models, and we are hard coding the models and the weights into our what we call the mask ROM recall fabric, which is paired with an SRAM recall fabric. Together, they are able to store both the model as well as do all the computations of KV cache. We have adapters and customizations – we support all of that. This design allows us to be super-dense in terms of compute and in terms of storage, and we can do compute on that storage incredibly fast, which is what drives density up and cost down
Why is everyone here is talking "improving silicon" == "baking in the model weights into hardware"
what happened to improve existing hardware in the ways not obvious to humans? Some novel transinstor placement, some new approach to make logic gates faster, matrix multiplications etc - the filed is huge
It will happen once models get good enough for specific applications like robotics; there the hardware is fixed and power requirements are strict, so it makes sense to have dedicated silicon. But for use cases that doesn't require onboard, it's always better to get access to the latest and greatest, until we get to the point where improvements are marginal between model releases (we're not there yet)
Its coming, AMD snapped up the company behind https://chatjimmy.ai/ Taalas that did exactly that with an older model. As others have been saying the point is to know when to do it. At what point is there a model that cannot drastically improve that you can then do this lets say for the mid and low tier models leaving frontier to run on gpus.
We are still in the middle of the AI race. Commodity hardware is easy to use, can do everything and is fast enough.
Your optimized hardware chip might be obsolete before its back from the fab.
SOTA Frontiermodelhardwarechip is a benchmark point of a potential model slow down.
Google is doing it right now under project Frozen v2 which should be ready by 2028? which is either just a small experiment or flexible enough and thats why it takes so long for it to happen.
This question would be better answered if people were careful about distinguishing between "models" and "transformer architecture".
If you bake a given transformer architecture into silicon and then, a year later, changes in transformer architecture give a large inference performance boost, you may have to throw away all that now nearly-useless silicon that gets outperformed by humble GPUs.
Models fully deprecate in a few months. Why would you burn an algorithm that fully depreciates in value faster than a bag of potato chips. The 'inefficient' general purpose hardware is constantly renewed with every released model. Even 6 year old Ampere GPUs are still usable.
Because the iteration speed on models is so fast that by the time they have an ASIC ready for one model version, they are already significantly ahead in capability. Think of how big the jump between Opus 4.8 and 5.5 has been. They were released four months apart.
Cuz once they're in chips they can be put into robots, and once they're in robots it won't be so easy to reach the off switch, and once we can't easily reach the off switch, we're doomed.
A hybrid approach might work better for robotics. A model-in-silicon for body control and sensing, then a different model on processors for reasoning, speech and decision making.
What makes a ROM hard? I'm ignorant of the details of chip design, but I'd expect just a large array of essentially open or closed connections. Is that incorrect?
plus row and column decoders that get more and more complex with size. but consider the size of the weights. convert to mm^2. Gawk at the resulting value
Qwen 3.6-27B is 56 GB of weights. Let's round to 550Gbit. NOBODY makes huge ROMs in modern SoCs, but if we assume it is as dense as the densest TSMC N3 SRAM, this is 33.5 Mbit/mm^2
Then your weights are 16,400 mm^2 !!! The maximum reticle-size N3 die is ~800 mm^2 so you cannot make it.
But, you might say, SRAM is 6 transistors. ROM might be just one. So 1/6 of that... 2730 mm^2... which is still too big to manufacture.
GPUs are faster, but you can't make your own arch on GPUs. FPGAs offer you that possibility. Said that... There are a few beasty FPGAs used in crypto mining coming my way... I expect that OpenTPU will be able to run frontier models with those.
First opensource arch that can do modern LLMs, while maximizing the potential of its hardware;
First opensource TPU build by a recursive improvement loop...
It's upcoming second generation could run the inference of the models that are being used to improve it...
