> I think that in 10-15 years, we are going to have consumer PCs (and phones!) running models doing pretty much anything that frontier models can do right now
10-15 years? The current rate is closer to 10-15 months.
15 months ago, the top model on the Artificial Analysis index was GPT-o3. It scores 30 on the Artificial Analysis index.
Today, you can easily run Qwen 3.6 27B on a variety of consumer hardware. It scores 37 on that index.
I've run all of these models on my laptop (Strix Halo, 128 GiB of unified RAM); the bigger ones, like MiniMax M2.7 and DeepSeek V4 Flash, need to be done at fairly aggressive quants that will certainly lose some performance and not quite hit the performance of the unquantized models. But still, it's definitely the case that you can run models that are competitive with the frontier models of 10-15 months ago on consumer laptops.
Heck, just announced though the weights haven't yet been released for independent confirmation is MiniCPM5-2B, a 2 billion parameter (small enough to run on your phone) model, that according to their benchmarks has performance competitive with GPT-4o, a frontier class model from 2024.
So that's around 1 year for frontier to consumer device class, 2 years from frontier to phone.
Now, this kind of rate won't necessarily keep up; it's possible that local models will hit a performance ceiling before frontier models do. There's only so much information you can cram into a certain number of bytes, and the AI boom is causing hardware prices to skyrocket so keeping consumer hardware from advancing quite as fast as it had been.
No. We need objectively around 192 to 512gb of very fast memory to be able to run really useful models. I don't see local hardware with these specs coming in 1 to 2 years. There are a big number of initiatives currently taking place to increase ram output. But it will take another 3 years minimum to close the current supply issues. China is fast pacing forward to have its own chip baking factories with small enough nano scales to have fast chips. Will also take a few years.
> 15 months ago, the top model on the Artificial Analysis index was GPT-o3. It scores 30 on the Artificial Analysis index.
There must be something really of with those benchmarks. Yes, hallucinations gotten better, but I don't see that the big frontier models got so much better in the last 12-18 Months. They just put out bigger wall of texts and feel smarter. But they still make way too many stupid errors
12 months ago "way too many stupid errors" was constant news. Today, you rarely hear about those anymore.
Sure, the novelty of the errors has worn off a bit and thus the reporting. Nevertheless the quality has improved immensely in this regard.
Also, AI video generation is now so good and accessible that it is very, very regularly used for memes, disinformation and proper (short) movie projects. AI image generation even more so (Mitch McConnell anyone?).
Pretending progress hasn't been mindboggling is insane.
Yes, if they are made with new technology and the quality is good enough they certainly are.
AI generated video memes were NOT a thing a year ago. Yes, the AI video generation in itself was a meme (Will Smith eating spaghetti), but now there are tons of convincingly good AI video memes about things like sports events generated by average Joes. It is proof of the capability and accessibility of the technology even if the use of it is mundane.
Everybody and their dog playing snake on their Nokia 3310 was similarly mundane, but also a sign of the end of the era of Gameboys and the beginning of (normie) mobile gaming.
> 10-15 years? The current rate is closer to 10-15 months.
The leaps between models have gotten smaller and smaller. 2023-2024 models were rocketing up in quality. 2024-2025 I’d say was pretty impressive too. But 2025-2026? Very easy to feel the slowing pace of improvement. I agree 10-15 years is overly conservative but 10-15mo is far too bullish.
The speed of model releases, in my view, is actually getting faster and faster. There were nearly nine months between GPT-3.5 and GPT-4. And now in just over one month, major models already included Claude Fable 5, Claude Sonnet 5, the GPT-5.6 series, Kimi K3, GLM 5.2, Qwen 3.8 Max, Grok 4.5... and the official DeepSeek V4 release is coming soon.
Both Codex and Claude Code have it, but they work slightly differently.
Claude Code uses Haiku to read through the transcript and decide if the goal has been completed. If not, Haiku injects a prompt back to the main model to indicate what still needs to be done.
In Codex, instead it's a tool available to the main model, plus some part of the surrounding harness that will re-prompt it if the tool calls haven't yet indicated that the goal is complete.
The issue that they are trying to solve is that sometimes models will stop before they have actually fully completed whatever task they were given; attention isn't perfect, and someitmes they'll complete part of it but not the whole task. Rather than making the user come back and re-prompt to keep going, they add a way to automatically do a bit more nudging to try to get the model to finish the task.
> Claude Code uses Haiku to read through the transcript and decide if the goal has been completed.
feels kinda odd to use a less capable model to determine if the goal is fully complete. Especially if the user is expecting /goal to thoroughly complete the task. A less capable model would be more likely to misclassify `isComplete?`
I've tried doing a loop of rending the SVG and then tweaking based on that, with local models (so, not nearly as strong). It wasn't very successful; it would mostly report that the image looked great and didn't need any tweaks. Maybe I should try it again, there have been some newer models since I first tried it. And yeah, maybe worth trying with bigger models. But I have found that models aren't necessarily the best at visual reasoning and review, even with a vision loop. Their lack of visual reasoning is part of why they still have trouble with things like ARC-AGI-3.
I've found much better luck giving it an audit check-list, including some steers like: are there any visual glitches or SVG bugs, are the colours consistent, etc.
You can always ask them to draw something else, as a way to avoid any possible pelican related data contamination; given how popular the pelican test is, I'm sure there's some pelican SVG drawing in the training sets of at least some of these models by now. For instance, you could ask for an SVG drawing of a cyborg bear riding a rocket powered unicycle.
It's a silly fun little benchmark, and because Simon's been doing it for so long, you have a lot of examples over the years to compare. But you can always come up with and run your own test with other drawings.
It's really frustrating, because now when I want to write something like a "not X but Y" or "you're absolutely right," I have to stop and decide if I want to self-censor to avoid sounding like a bot.
Sometimes those constructs are actually useful, but man has their overuse really killed them!
It drives us crazy because everyone is using the same 2-3 different machines. So rather than each person having their own unique speaking style, the whole world (or, everyone that publishes direct LLM output) is now speaking in the same couple of styles.
And these machines all tend to converge on very similar styles; they have huge amounts of overlap in training data (much of it being already obnoxious internet marketing), they frequently train on each others outputs, and the RLHF process has a tendency to emphasize certain kinds of "cheap win" styles of speech.
He is saying that weaker models, as measured by a benchmark to distinguish "frontier" models, would be exempted. So an academic lab or startup that isn't yet producing frontier models would be exempted, but once it crossed some benchmark based threshold it would be subject to this kind of oversight.
Of course, right now you've got benchmaxxing going on; some companies specifically targetting benchmarks to appear stronger than they are on a wider range of tasks. Now you might see bench sandbagging, specifically looking weaker on certain benchmarks to avoid regulatory oversight.
For instance, once way I could see this going for open models is to release them undercooked; stop the RLVR process a bit early, leaving them a bit weaker on tool calls and agentic performance, but also release the RLVR environment so people can finish the process themselves.
In fact, this is fairly close to what Nvidia is already doing, the Nemotron 3 models are somewhat undercooked but they are releasing their full training pipeline, to encourage people to use these models as a base for further training, which will generally be done on Nvidia hardware.
Yeah, Kraken SDR removed some functionality due to these concerns, if I remember correctly.
Odd, because export controls don't generally apply to published material (like open source software), but maybe they were worried that because they were also selling the hardware they could have issues due to the combo being export controlled.
As someone who works daily with export-control-adjacent hardware and software, my experience is that people tend to aggressively self-censor to a far higher standard than export control regulations actually require. The perceived headache of drawing the ire of whoever it is the enforces this stuff (which as I type this comment I'm just realizing I don't know who specifically is responsible for that) is so scary that people don't want to take any risk at all of being targeted.
But there are already benchtop or handheld signal analyzer for that purpose.
This seems more like a tool for checking across entire large assemblies like an entire building, car, aircraft, etc, for unknown sources. If you have an individual discrete device that you're already testing, just using traditional instrumentation seems reasonable, but on a large, complex assembly, I can see it being useful. Also useful for things like detecting if a particular antenna is working without actually going up there to measure near it; if you have a MIMO setup with multiple antennas, this might make it easier to check if all of them are working correctly when mounted in inconvenient areas.
I think that for a single device, this probably wouldn't help much over just having a more traditional signal analyzer, either benchtop or handheld. If you know what you're testing, just using a signal analyzer around it will give you a good first pass picture of emissions, and probably be much more informative and precise than this.
This seems more useful for finding unknown or hidden RF sources, for instance looking thorugh an entire building to find unknown RF sources, or maybe a whole complex assembly like a car or aircraft.
10-15 years? The current rate is closer to 10-15 months.
15 months ago, the top model on the Artificial Analysis index was GPT-o3. It scores 30 on the Artificial Analysis index.
Today, you can easily run Qwen 3.6 27B on a variety of consumer hardware. It scores 37 on that index.
Here are a number of open weights models that you can run locally compared with the frontier class models from 7 to 15 months ago: https://artificialanalysis.ai/?models=o3%2Co3-pro%2Cclaude-4...
I've run all of these models on my laptop (Strix Halo, 128 GiB of unified RAM); the bigger ones, like MiniMax M2.7 and DeepSeek V4 Flash, need to be done at fairly aggressive quants that will certainly lose some performance and not quite hit the performance of the unquantized models. But still, it's definitely the case that you can run models that are competitive with the frontier models of 10-15 months ago on consumer laptops.
Heck, just announced though the weights haven't yet been released for independent confirmation is MiniCPM5-2B, a 2 billion parameter (small enough to run on your phone) model, that according to their benchmarks has performance competitive with GPT-4o, a frontier class model from 2024.
https://nitter.net/i/status/2079088670804767114
So that's around 1 year for frontier to consumer device class, 2 years from frontier to phone.
Now, this kind of rate won't necessarily keep up; it's possible that local models will hit a performance ceiling before frontier models do. There's only so much information you can cram into a certain number of bytes, and the AI boom is causing hardware prices to skyrocket so keeping consumer hardware from advancing quite as fast as it had been.