Yes. That is an insight but then again, most marketing emails were templatised slop1.0 from mailchimp and other sites. This just makes it slicker. So, I'm not sure.
No it’s: evaluating these systems are complex and there’s a reason why sociology, cognitive psychology, medicine, etc are all done in careful double blind conditions with pre registered tests. It’s not that humans are not smart enough, as I said human evaluations are incredibly important. And yet they are a minefield of biases you have to worry about and correct for.
- evaluations need to be done at the same time to avoid drift in your bias
- you need to worry about your test set: which questions are you asking? How many of them? Are they representative of your work?
- which one did you do first? Raters have a tendency to bias in one direction or another
- you also know the label! You know which model is which! This biases your assessment…
And on and on and on. Careful science exists for a reason.
It has seemed to me that with each step from Opus 4.6, to 4.7 to 4.8 Claude has gotten worse at building good solutions. Like perhaps it is more "capable" in the small scale than 4.5 was but it's much worse at knowing what to do.
At first I thought the KDE apps all playing on the K was kinda weird and awkward, but as time went on I really appreciated how easy it was to search for them due to this. So I really think it's a benefit to play on traditional words rather than use them as-is.
> Seems like a strong signal the money burning party is coming to a close.
One provider who was undercutting the market with non-standard billing model moving to a more standard billing and prices doesn't seem like that strong of a signal, other than that Copilot was underpriced.
It was the only clear model from a user's perspective. Sure, a request may not perform as expected, or end earlier than desired, but it was an agreed to cost that was clear on both sides: 1 enter press in a prompt window = 1 request.
If they wanted to limit what a request can do via their harness, I'm sure they artificially could.
I hate all of the other plans I've seen of here a "credit" or here's a "bucket of usage", and we pull an announced amount from it based on arbitrary info that can't be audited or proven, and most of shat is spent might be entirely useless anyway.
Claude Code has a problem where 1 request could take a significant portion of your 5 hour window, and it's unclear why.
It's much like SEO, where Google sometimes says things that might help, but it's just magic wand eaving hoping something works.