Really disappointed about this. I've been using Mercury 3 quite a bit and with great success. It is an odd announcement for sure. Makes me think acquisition of some sort. I'd say keep an eye out for future press releases regarding the company. This was basically a heads up to the people using the model to not rely on it long term.
I burned though my weekly fable usage last night on the $200 plan. I had $200 in promotional usage credits and was in the middle of executing a moderate sized coding plan. Ran on usage credits for about 1h 15m and burned $120 in usage credits. I was astounded to see how fast the $ usage added up. One problem was that I was using sub-agent execution so multiple agents were running simultaneously and I realized at the end that claude had "Forgotten" my directive to use cheaper models as appropriate for sub-agent tasks so I was running multiple instances of Fable at once. Still hard to imagine paying per token. $200 a month is high, $200 per night is crazy.
I am too young (most of us on here are) to have lived through the paying for time on time-share machines in the 60s/70s, but this is giving me creepy memories of paying for sprintnet/telenet and tymnet... And I guess aol, compuserv, delphi. Are we really doing this computing model again?
I think you were able to buy VS Pro perpetual licenses for one time 500-1000 at all times. Of course they had higher tiers and a subscription model which were more expensive but in reality for most people there was little to no added value.
yeah. those msdn cd's were very valuable back then. it was before the market was flooded with tech books also. aside from Russinovich's stuff, you basically needed that cd set to do any hardcore windows coding up til probably 15ish years ago.
well spotted, I couldn't place the feeling, me it's like paying per minute phone costs when downloading on a modem. You know it will cost, but you do it anyway.
The other weird thing about the subs is that if the agents aren't grinding if feels like I'm losing money.
I still remember searching really hard for shareware apps and games between 1 to 3 megs, though I am sure I used more minutes looking for them than the downloads would have taken. Back then, if we weren't limited by our dialups, we were surely limited by our very low storage capabilities. But man, code was so much better and tighter then. Haven't seen much focus on this in vibe coding, sadly.
I stopped using sub agents after they caused me to hit my limit to quickly. Had to create a separate account and pay for another max plan to unblock myself temporarily
I have a process generating trivia questions for my app and by passing in my existing dataset each time as a cached exclusion list, the output tokens it save more than pay for it.
Guardrails can and do help but they don't prevent hallucinations entirely. I generate trivia questions for my app. Questions are true/false and I tell it you must include an explanation of why the question is true for false. I also tell it to give me supporting links which validate its claim. I've had it make up explanations and invent plausible looking URLs. But I wrote a checker that pulls the content of the URL and asks a different model if the URL supports the claim. LLM's are much better at something like that than pulling a trivia question out of thin air. Reviewing diffs can be much the same. It isn't having to write the code but just evaluate if it matches what the commit comments claim it does. You are right, though. Without good comments on PR's and commits (i.e. context) the diff review is going to be a lot of assumptions made by the AI which is where hallucinations get more common.
I hear you! I read the article and thought the same thing because I've actually built a surprisingly similar pipeline myself, but it culminates with Human review. I've got a trivia app (Hedge Calibrated Trivia) whose whole content pipeline is built on a similar architecture of models evaluating models. I didn't want the same trivia that every other app has and I naively imagined that it would be easy to ask an LLM to give me questions. Turns out when you ask an LLM for trivia questions, the FIRST thing it gives you are the same questions every other app asks!
After iterating into many of the same problems OP describes I settled into a similar pipeline. I generate questions with Gemini 3.5 flash and review with Claude. The different vendor generation and validation eliminates training holes in a single model solution. This is True/False trivia so I ask for an equal mix of True/False questions and also ask for alternate phrasing of the question to the opposite variant. Each statement must provide an explanation and sources which back up the claim. I pass in my existing dataset as questions to avoid, but found the model then rewording the questions or category shopping the questions into related categories in an attempt to re-use them. That led to using Claude Sonnet to check for duplicate "themes" and similar topics. Claude also acts as a critic of the question and the explanation. Sometimes Gemini's "thoughts" leak into the explanation. The critic catches these leaks and also catches "weak" questions (basically shapes that I have identified via examples as lazy or projecting the answer in the question). Once the comparative models have stripped out anything obviously bad, my server curls the source URLs and uses Sonnet to verify that the fetched source supports the claim. Any URL that doesn't resolve throws up flags, and I found Gemini was particularly bad at inventing source URLs for sports trivia. Updated prompting directed it to prefer Wiki links over its fabricated deep links into league websites, which helped immensely. Still one of the biggest tells in review is that the URLs don't resolve or don't confirm the statement.
But review is the entire point here. I have to review them all. A question that lands in the app and is wrong reflects terribly. So while I have a trivia app on the App Store, the real complex App is my private admin app that lets me sit on the couch swiping through each and every question sorted by the model's confidence in its own quality and correctness (weakest first). No question makes it into my bank without me reading, checking all the indicators that came from the models and potentially even rewriting the question to make it better. It all happens in the admin app and exists entirely to prevent hallucinations from reaching real users. Early on I was maybe accepting 1 in 3 questions generated, but I'm probably closer to 2/3rds now. Of course not every dismissed question is wrong. The quality checks I've put in the pipeline are probably giving me 80% questions that are correct and sourced, but sometimes they just aren't what I'm looking for or don't meet my standard.
The best thing I ever did was make an app to review these questions though. Sitting on the couch, kick off a question generation, come back 20 minutes later with 50+ questions in the queue. Sort through them and pull out the 30 best when I have the time. That's the real anti-hallucinations pipeline.
Very similar experience. The review UI might actually be the most important part of the whole pipeline. Does the confidence ranking match what you end up rejecting?
It's a future evaluation for me, as I'm keeping the app with a "Data Not Collected" App Store label for now. Once I have some visibility to stats of how people answer each question and at which confidence level I expect it to reveal bad/poorly worded questions that made it through my pipeline.
Also on my to do list is an upgrade similar to my de-duplication logic. Currently I pass in my approved set of questions as a list of topics and questions to avoid. It works well especially when combined with the more deterministic comparison of generated questions vs approved question by the separate model. The upgrade would be to pass in the questions REJECTED by me. These are question which made it through ever automated gauntlet of models and code, yet still ultimately were sent to the waste bin by human review.
I'm imagining that by including the list of examples of what NOT to do, it could inform the initial Generation to be smarter. I currently have a limited list of a hand picked "bad question shapes" where I provide the question and why its bad kept statically in my prompt. But I've realized sorting through thousands of questions, I've probably built implicit data through rejection that I may not actually recognize myself which the model may pick up on.
reply