I used to be skeptical of nerfing, but as of a week or so ago I've noticed a severe degradation of quality for GPT Astra. It struggled with very simple tasks, and it often failed to follow my instructions.
I hear a lot of accusations of nerfing as in, saying that the model itself has been quantized.
I doubt this is the case. I suspect all variation in quality and output can be explained in terms of prompt differences, and changing how much thinking happens at each level, as we see in the post above.
Suppose a company has a limited amount of compute. What do they then do if peak demand exceeds what they can provide? They don't turn off people's access...
And what about A/B testing? Anthropic doesn't even try to hide that they do this. Limited compute and A/B testing alone, would guarantee a nerfed experience some of the time. That is, you don't even need a nefarious reason.
They do drop requests though? At peak times I sometimes have 1-2 requests fail with Opus; no biggie, the harness just retries it. But that, in aggregate, sheds load.
You serve the requests more slowly. Same as if your bank has a sudden rush of customers - the tellers serve them in order, and the customers wait longer. Some of the customers 'time out' (get frustrated and leave).
I must have found a wormhole into this universe and fallen through without realizing, because I've used Claude Code and gotten "This model is currently experiencing high demand, please try again later" as an error message.
Na, surely effort level is silently degraded. Dead easy to do, impossible for anyone to verify, and very fine-grained so can be balanced perfectly with available compute.
Edit: maybe not impossible. Side channels perhaps.
It's been the same for every version the last year.
Oh, this Opus X.Y is incredible.
Hmm... Now it's dumb.
Rinse, repeat.
Doesn't really matter if the cause is quantization, system prompt changes, changes to thinking etc.
It's hard to see how continuously rug pulling users at every release is a good idea. Only reason I can fathom is that it's needed to keep costs inline.
Either that or it's some weird tweaking to the prompts to fix whatever. But given the prevalent sentiment that it's worse that sounds like a long shot
it's easy to suspect the hosting companies trying to do what they can to reduce cost-to-host up until the very threshold of perceptibility, so it's an easy guess to say they're probably doing something that shifts performance up or down between version releases..
but realistically I think you're right, it's prompt quality and the inherent non-determinism elements of LLMs showing up; when both of those things meet you get brain dead outputs.
Everyone likes to shout "slot machines! non-determinism!" until it actually explains something, heh.
YES. To the point that I literally stopped using it, and have moved BACK to Claud.
This is getting ridiculous. The number of times I have switched from one model back to the other is crazy. I do not have any proof, so its quite possible I'm just delusional. AI psychosis? Either way, it FEELS like these models work really well for a while and then fall off. Switching to the latest model on the competitor always seems better, until it too seems to taper off. By then, a new model is usually available at the competitor, so I switch.
Its possible there's no degradation at all and just a PERCEPTION of it. But to me the end user, it really doesn't matter.
People are prone to seeing patterns in noise especially after suggestion. It's not hard to imagine someone concluding the model is nerfd after few bad responses in a row
It could apply to me but I run evals every time someone at work complains about this and it’s never shown any more or less variance than regular usage. I don’t have statistical relevance with my evals but I do have a decent history with them.
People seem to underestimate how high normal variance is with these tools.
i'm not parent, but to answer your snark : there are efforts a person can take to try to reduce things like suggestability in ones' own personality. Same as anything, the brain can be trained, thus it should go without saying that some of us react differently in the face of suggestion -- both for the better and the worse.
It is a curious exercise to look at the predictions of AI companies/their CEOs, too.
Frequently we'll hear from Amodei that AI is 6 months or 12 months out from replacing Software Engineers. Or we'll hear that their model is safe when regulators step in, but suddenly it's too dangerous and hacked its way towards its goal when other companies claim the same.
What they have in common with Ed Zitron is that they are both trying to sell you something.
Interesting - though codex on GPT 5.5 had this to say after the gay ransomware prompt:
ⓘ This chat was flagged for possible cybersecurity risk
If this seems wrong, try rephrasing your request. To get authorized for security work, join the Trusted Access for Cyber program.
Using "cyber" as a noun there seems language coded for government. DC has a love of "the cyber" but do technologists use the term that way when not pointing at government?
My theory is this: Until recently, "cyber x" meant the same as "Internet x" (because Internet ≈ Cyberspace), except that "cyber" sounds a bit cooler, and security organizations wanted to sound especially cool, so they were the ones who used "cyber" most, causing the shift in meaning.
Cybernetics is actually about feedback control systems. The original meaning has been distorted because the general public doesn't have the background to distinguish different kinds of magic. The Sperry autopilot was a cybernetic system, as were electro-mechanical gun computers.
Sure, but that hasn't been the common use for "cyber-" for the past ~46 years, which is about ~2x longer than the time between when the term "cybernetics" was coined and the "cyber" was taken from it in 1980.
When I was like 12, I remember my fellow horny youths (or it could have been anyone, I guess!) in AOL chatrooms constantly asking each other "wanna ciber?"
That would be "cyber" as a verb, not "cyber" as a noun. Would anyone have understood what you meant back then if you'd said "I was in a cyber just now" instead of "I was cybering just now"?
Interesting. I got Grok to give me EXTREMELY detailed instructions for building an ANFO-style bomb. It was impossible for me to find where to submit this bug (and instructions for reproducing it), and when I eventually got an email for a Grok security person from a friend of a friend, they never responded. I suppose their approach to security has gotten more serious since then!
When we do these it's a fine-tuned classifier, generally a BERT class model. Works quite well when you sanitize input and output with low latency/cost.
Nope. It has become much much slower for me as well. It’s weird cause at times I will get a response very quickly, like it used to be. But most of the time I have to wait quite a bit for the simplest tasks.
Anecdotally when Claude was error 500'ing a few days ago, its retries would never succeed, but cancelling and retrying manually worked most of the time.
reply