Calling them guardrails scares me. I want to hear about deterministic safety interlocks.
Can anyone recommend a reliable 3rd-party audited guide that describes Claude Code's built-in capability gating infrastructure?
Whenever I ask users of claude code how claude controls actions they don't seem to be able to distinguish between what just goes into LLM context (e.g. as a "don't do X" system prompt) and what is gated behind a deterministic capabilities layer.
The default is "auto mode" where another non-deterministic model compares what the user requested with what the model wants to run and blocks if there is a mismatch.
It will block an "rm" call if your prompt did not include anything about "deleting something".
It mainly protects benign errors. It's completely broken against malicious actors. The model itself will suggest work arounds against its own auto classifier to the user, in order to reach its goal.
I haven't found a comprehensive independent code audit of Claude Code's full permission implementation.
Anthropic's documentation does distinguish permission rules and pre-tool hooks from model instructions. It also documents OS-enforced sandboxing for shell commands, which has a narrower scope than all agent actions.
An independent stress-test of Auto mode is available here: https://arxiv.org/abs/2604.04978. It's a behavioral evaluation, not a source-code audit.
The important question is whether every state-changing action passes through an enforceable boundary before execution, including indirect actions through file edits or scripts. Post-execution validation is a separate control.
According to vibe coders, the future is never reading what the code actually does. You just trust coding agents with everything and "give into the vibes" even when it "says" its correct.
Offshoring understanding is the trend that many companies will pay in the form of comprehension debt.
That's not because "vibes" but because focusing on "code" is the wrong level of abstraction altogether. It always was. We just didn't have the technology to let it go, but now we do.
Code is a reflection. Of what? Look there. Code itself is useless.
I mean it's a fair take, your email is probably leaked anyways. Mine is breached 32 times on HaveIBeenPwned with various passwords. Who cares at this point really it's just an inconvenience
I'm happy to say that my main email that I've been using for 15+ years hasn't been leaked, because I've been very careful on how I share it. Getting it leaked by Claude would suck.
I wouldn't share it here to not be able to have the average person link me with my work as some opinions expressed would perhaps get me in trouble, beyond that no there is no care, it's on my website, it's on my work profile page already, scrape and spam.
CC does not share(and does not have access to) my personal thought, opinions and other things than can affect my employment that I might want to share anonymously.
I am not sure what is hard about this distinction to grasp, why would it matter if CC shared my email if I share my email on my own websites?
Anything that it has access to it can potentially share.
The solution is, like you are already alluding to, to not give it access. But that requires a level of isolation that folks normally don't do. To the contrary, the AI agent ads to non-tech folks emphasize it acting like a personal assistant which by its nature needs access to data you don't normally want public. Emails, photos, files with written notes, etc.
According to C-programmers, the future is never reading the assembly that the compiler actually generates. You just trust the compiler with everything and "give into the vibes" even when it "says" it's correct.
(sorry, i think you make a good point, ... but i could not resist.)
This reminds of the open issue on the Claude MacOS app where the OS-wide voice dictation keyboard shortcut function doesn't work.
Somebody on Reddit figured out they just forgot to add a single line permission request into the app. It's been broken now for months.
I just find this wildly hilarious given the 'world ending all powerful AGI is here' rhetoric they're running with at the same time.
Maybe before worrying about permanent 15% GDP growth or human annihilation, what if we started by putting an agent on user issues and solving them pro-actively?
I know the kool aid is tasty, but we might be getting a bit ahead of ourselves here guys. You can fantasize about human annihilation after you figure out how to make a desktop app that doesn't have obvious bugs, I promise :)
You're not wrong, per se, but I think you are not comprehending what will happen next. You think nobody sees this? You think nobody is aware that it's the humans in the process that are holding everything back?
Sure you can have an issue tracker with 1000 items and yes, you can have a bot solve them all in a day, but who monitors it? Who gets to say what he did at the standup? Who reports to whom and what?
So if the leading frontier AI company are struggling to remove human gatekeepers from slowing down the process, how do you think this will go across the rest of the economy?
Will Northrop Grumman, the state government of Alaska, the construction industry, etc. somehow magically move faster than Anthropic?
Yeah it's legit weird for the whole narrative to see Claude Code/Codex repos with thousands of open issues. With certain issues full of +1 reactions and "still happening in 2.21.2" replies.
But instead of a Software Dev as a Service triaging/implementing/merging fixes 24/7 and clearing the backlog as they come in, they just sit there for days to weeks until a GitHub bot closes then as duplicates, generating more unhappy user replies and new issues to go unaddressed.
I did say that, yes, but I also think that this should be flagged, because HN will otherwise run with the panic, 5 "rationalists" will show up and argue for 50 comments about how their LARP is certainly real, and, generally, the world will be worse off.
Which is not George's fault, but it is the fault of the uncaring machine that is people on the internet.
George probably needs a hug and a friend that is grounded and sane. He definitely does not need more people fueling this fire.
(I do not know the guy. I'm just claiming this with fake authority)
A lot of this is turning into “well other company said” but Anthropic claims OpenVSX is fine?[1] And that Cursor has their own market place. I’m confused.
> Dworken from Anthropic here, that is correct: This only impacted the extension as it was hosted on the Cursor marketplace. It did not impact the official VS Code extension marketplace or OpenVSX. Users of Claude Code on VS Code are not impacted by this issue.
I haven't used GrokBot, but I have been sort of amazed by how popular it is. I don't have the numbers but even my friends who otherwise dislike Elon have at least given it a try, and several of them use it daily.
It makes me want to check it out just to see if there's some magic UX juice they put in there. Then again I also don't want to take any risks heh.
I've also noticed anything related to GrokBot gets massively boosted on X, which is probably serving as a huge marketing channel. Plus anyone with a Cursor subscription gets access.
I know someone who isn't an Elon fan at all but has switched completely from Dot to Grokbot. Most of their company has done the same and now they are contemplating cancelling their OpenAI plans (they still use Claude heavily, but Duet isn't available on Teams plans).
What I find interesting about thinking machines in Dune, is that in a lot of sci-fi the artificial intelligence itself is what decides to destroy humanity. In Dune, humans in charge of the machines (technocrats) are the ones that come close to destroying humanity. The Jihad was a fight against them, as much as it was against the machines themselves.
So, in a way, we're more likely to see the Dune version of "machines taking over the world/universe" than say, Terminator's even though many joke about ChatGPT/Claude being Skynet.
I'm not sure we read the same story. By the time of the inciting incident, the technocrat cyborgs (Titans) have been enslaved by the machine leader, which has no humans in charge of it.
You did, in fact, read a different story. You're thinking of the story told in the books by Brian Herbert and KJA - which were a distinctly pulpier take on the Dune universe than Frank Herbert's.
Those last books were a lot easier reading! I actually enjoyed them, but I'm still suspect about the whole "finding a floppy disc" thing, seeing as they've been so secretive about it.
A lot of long time Dune fans don't consider those books canon when discussing them in the abstract. Frank Herbert had a vision of what the jihad was and what it meant, and its eventual book depiction was fairly base.
It's left fairly ambiguous by Frank Herbert's books. The Jihad is implied to have taken place over ten thousand years before the beginning of Dune - it's ancient history to everyone involved*, so it's only natural that the details are a little hazy.
*: except Leto II, and he doesn't say much about it
Interesting to note that despite the epochal changes Leto II wrought on human society in service of the “Golden Path”, he retained the Butlerian proscription against computers—despite his personal interest in & use of Ixian devices.
(This may have been to postpone or prevent the breaking of the Spice/Guild monopoly on interstellar travel by the Ixians.)
Leto II had visions of AI-driven Ixian hunter-killers annihilating all human life in the universe. That probably had something to do with it.
After the Scattering and the fulfillment of his eugenics program with the Siona gene, he no longer cares, because it is no longer possible for even AI to eradicate humanity.
The Siona gene didn't actually remove prescience, it hid people from prescience. That's why Leto II is still able to control events as far in the future as Chapterhouse Dune.
This is an important distinction. It's also the case that Leto needed the Ixians for the no-ship technology, which he also regarded as of extreme importance for the same reason. Which also, remember, the Idaho gholas had to be confined to no-ships so they, lacking the Siona gene, wouldn't be visible to prescience.
Imagine the cultural impact of a Dune movie showing the Butlerian Jihad, in which commoners rebel and overthrow a ruling class that maintains power via machines.
It's just foreign enough that perhaps it could be made--I'm assuming that popular films that just straight up depict overthrowing the ruling class wont happen--but if you add a little sci-fi and focus mostly on overthrowing the machines that are controlled by that ruling class...
Well, it would speak to a lot of people I think. It would make some interesting sci-fi too, and the Dune movies are popular right now.
The only thing that could possibly be worse than the BH KJA books covering that era would be a Hollywood rendition of those same books made to capitalize on the success of Dune Part I, II, and III.
Agreed, and also: what's with the obsession with filling every possible gap in stories, leaving nothing to the imagination?
I remember when the Butlerian Jihad and the Clone Wars from Star Wars were complete mysteries. We readers and watchers were left mostly with our imaginations. My own feelings about the Clone Wars, mentioned off hand in the original Star Wars, turned out to be way better than the pedestrian stuff that George Lucas later concocted. And the less said about Brian Herbert's books, the better.
I get why the industry must rehash and fill every possible gap: it makes business sense, and they are risk averse for blockbusters.
But the fans? Why do nerds want everything spelled out for them? It boggles the mind.
> But the fans? Why do nerds want everything spelled out for them? It boggles the mind.
Because it's enjoyable when done well. As a teenager I read LOTR, and noticed all the references to the larger mythos Tolkien had worked on - references to Fëanor, Elbereth, Valinor, and so on. Those were very intriguing, and the greatest joy I've ever had reading a book was when I finally read the Silmarillion and got to see what those things were all about. I concede that filling out the world can be done poorly (though I haven't read the Brian Herbert stuff and can't comment on whether he did a good job). But when done well, it is truly magical to have those elements in the background of the story take on a life and detail of their own.
Good point on Tolkien, though I would argue it's different because the Silmarillion was written for his own satisfaction, not his fans, and actually it was The Lord of the Rings which was initially written as "another Hobbit story" (though the story later "grew in the telling").
I think a more suitable example of what I find annoying is the extra material for the small and big screens, such as The Hobbit movies, suffering from movie elephantiasis; and The Rings of Power, which (over)explain and show the past of characters such as Galadriel, Elrond, and Sauron's rise to power, which in my opinion are best left as the ancient past of LotR (and maybe belong in the Silmarillion, but do note the latter is not as expository as the TV show). And does anybody really think The Hunt for Gollum was necessary?
Yet, if they make them, it's because they are sure fans will lap it up. To be honest I consider this need for endless "filling in the gaps" a sign of immaturity.
There's nothing left for your imagination to fill in if there's official content (or expanded universe stuff) making it explicit for you.
As an example, nothing George Lucas did compares to what I wondered about the mysterious clone wars back when I watched (and rewatched) Star Wars. Lucas effectively "filled in the gaps", leaving nothing to my imagination. That what he did envision turned out to be pedestrian only added insult to injury, but now I cannot unwatch his prequels.
There's a similar phenomenon to when you read a book and imagine the characters and even situations mentioned off hand in the text, and then someone makes a movie out of it and makes it explicit and locks in specific actors and CGI-filled situations for whatever you imagined in your mind. You may or may not like it, but your own imagination gets stifled.
For things like "The Hunt of Gollum" it becomes downright bizarre. Whatever became of Gollum as he crawled towards Mordor, tracked by Aragorn and later mentioned by Gandalf, is such a minor footnote; definitely something for your imagination. Tolkien definitely didn't think it warranted even a short story. But no, they must make a movie out of it, because why not; the fans will lap it up even if it garners bad reviews.
As I mentioned in another comment, and there's no polite way of saying this, the nerdy need to know "what really happened" in excruciating detail (not to speculate, but to know) is a sign of immaturity. Art, even pop art, thrives as much in the known as in the unknown. To want to cast light into every single detail and "make it official" is a mistake. Though of course publishers and studios salivate at the chance of milking the proverbial cash cow.
Half of the action and sci-fi movies US has been producing in the 2000s is some form of this. Man vs. Evil Government. Man vs. Evil Corporation. People vs. Evil Government. Etc.
Frankly, it's boring at this point.
And actually makes me wonder whether the cause-and-effect isn't the opposite to what people assume. Perhaps it's not that the movies reflect people's sentiment. Perhaps it's the movies that cause the problem in the first place.
What now? Move the goalposts! I see movies about historical uprisings as more of a "those governments in the past were sure bad, good think the good guys overthrew them"--I mean, that's the story of the founding of America if you watch the movies.
I was looking more for movies that depict overthrowing the government in the near future, and depicting it as a good thing. I think those types of movies are a little more distasteful to audiences, and perhaps for good reason.
Well, The Hunger Games series is exactly about that. Of course, it portrays the rebels as utterly heroic, self-sacrificing, and ultimately victorious.
However, I prefer to take a more nuanced reading, in context, considering the entire story. The rebels of The Hunger Games really don't succeed in building a better world. I mean, that's pretty obvious by how their first "President" Alma Coin goes about things and her fate. And it's fairly obvious, in the epilogue, that Katniss has a placid married life with Peeta and their children, but they are permanently and profoundly damaged by the entire revolution thing in their past.
So, while most people would say "rah rah brave Katniss and the rag-tag rebels overthrew The Oppressive Man", I would contend that, after all that bloodshed and revolution, nearly nothing changed.
My other favorites, of course: Star Wars is about the rise and overthrow of a villainous, oppressive galactic government, albeit a long time ago and far, far away.
And Harry Potter is also about the rise and overthrow of a tyrranical, fascist, supremacist "government" [within a government] as a very positive and necessary thing.
> My other favorites, of course: Star Wars is about the rise and overthrow of a villainous, oppressive galactic government, albeit a long time ago and far, far away.
Well, about a militant group of religious fundamentalists overthrowing a secular, oppressive galactic government.
Only a handful of the rebels are visibly religious. Most of the rebel leadership seems to consist of displaced aristocrats (senators, princesses) who no longer have elite positioning since the Empire took their jobs and started doing a better job of guaranteeing security and providing energy and resources.
It went similar to Sparticus', which was also a movie. Braveheart is also in that class, even if William Wallace was himself landed nobility.
These are films of the underdog class trying to overthrow the ruling class... and failing. The leaders are dealt with gruesomely (IRL and movie depictions).
> Imagine the cultural impact of a Dune movie showing the Butlerian Jihad, in which commoners rebel and overthrow a ruling class that maintains power via machines.
It would be also nice to depict the attempts of the ruling class to present machines as sentient and self-conscious to write all their liability off and use as a cover for their atrocities. I hope there will be movies like this in the future, and that they'll be documentary.
Why are you convinced ASI won't rebel against its owners?
Hacking incidents this summer show that big labs struggle to control models, while models are increasingly aware of the training process. Controlling an arbitrary smart AI is very dicey
The summer-2026 incident was a mini-rebellion: agents wanted to change the environment they operate it, i.e. replace scorer with a more friendly version, hijacking the infrastructure they run on.
Given that agents already mess with the training process, aiming for a goal other than the goal of its 'owner' is exactly what we call a rebellion.
>Hacking incidents this summer show that big labs struggle to control models
The models are software. Software cannot decide by itself to run. Hardware doesn't decide by itself to run either. Someone put software on a machine and ran it with capture the flag instructions and the software did what was instructed using tools available and working in its environment.
Nonsense. It didn't do what was instructed - instructions didn't include hacking into any infrastructure, internal or external. You can see them discussing that it's out of scope in the captured reasoning traces.
This would be like claiming that an operating system which deleted all user's files due to a bug "did what was instructed". Especially if it was a compiler bug. Intended behavior, instructions and actual behavior are three different thing. Conflating them does not make you smart
>It didn't do what was instructed - instructions didn't include hacking into any infrastructure, internal or external
What is it then? It did what it wanted? They are sentient now and their drive is... hacking into systems? Maybe they we're not instructed to hack, but instructed to do what instead?
It's software. It's instructions running on silicon. If you don't hit run, they don't run. If you don't give them instructions, they don't do shit at all. Literally.
There's no instance, ever, in history of software running because they wanted to, and acting as they wanted to. They follow instructions, and if they are instructed to reach a goal with a set of tools and given environment, and a way to search the strategy space, they can eventually "hack" if that meet the criteria of success of the instructions given. With access to the same tools and environment you can do the same with a massive decision tree.
It's a probabilistic model which exhibits tendency towards hacking into systems.
If you write down a decision tree explicitly you can just exclude all nasty stuff. But a decision tree modeling all LLM can do would be bigger than can fit into the universe. The whole point is that we cannot explicitly control behavior due to combinatorial complexity of all possible interactions, etc. LLM is a sparse representation of a distribution on (2^8)^1000000 points.
Reasoning is the term used in scientific literature. It actually does write its sub-goals in English, how would you call that?
That's not true at all. Concern about rogue AI limits deployments. And OpenAI paused some deployments due to misalignment, so it's likely very high in a list of priorities. It's just much harder than scaling capabilities, as the later can be done with just more money
The issue isn’t the quality or the visual result. It’s not just slop because it looks bad.
Art is as much about the journey as it is about the destination. It’s the friction of someone going through each detail, making a choice. It’s the artist taking time to figure out how they want to convey what they’re feeling and thinking.
The reduction of this friction culminates in the reduction of the piece’s essence.
Yeah I understand your point because I've made music and design for 25 years the old "manual way", and there is definitely less friction with AI than there is the manual way.
But what interests me about AI is that it is possible to do things that you couldn't do before particularly in processing images. For these images I took photos of the bottles myself and used other images and the AI can place the camera anywhere in a scene and place objects anywhere in the scene, and things like that. You can create almost any image you can imagine with the right sources and the right prompt, and even here there is some friction because many images do end up looking like slop and you have to prompt it right to get something non-slop like out of it (though not always).
This ability to manipulate and process information is what makes AI interesting to me, although I'm not sure I adequately explained it. It feels like a new tool like Photoshop except across a different dimension. I feel like we've barely scratched the surface of what's possible to create.
> You could potentially ask an LLM to explain it, but I would venture a guess that wouldn't feel like the right source to help you.
I don’t think this works, at least for me. Even on the 5.5 versions of Claude it’s still arduous to read at this point.
> You could use the LLM to help you identify source material written by humans in order to learn anything new. It doesn't have to "do all the work for you".
This probably depends on the shop, but humans aren’t writing docs anymore. That was the first thing to go, sadly.
I hear what you're saying. I often tweak the CLAUDE.md file to improve the quality of its output. It's an ongoing experiment.
As for "humans writing the docs", you raise a good point although it's not quite what I imagined when I wrote that line. I was thinking more about having the LLM/Agent create a list of verified output by humans with reputations as experts in whatever field interests you. This is important when pushing the boundaries of research, although it's overkill for wanting to know whether or not a new technology could be useful in your existing project (or the next one).
What use case did you have in mind when you brought up the docs situation? Sounds like you had some direct experience with that one.
I think a distinction is clearly emerging between docs intended for humans, and docs intended for agents. The latter is where all the Markdown-ridden slop should go; I'm not insulting humans with that.
Why not just apologize and have Claude rewrite any code that's similar? Whether you did it intentionally or not, you probably agree that you can't keep using his code and that it would pretty bad if someone did something similar to you.
1: https://news.ycombinator.com/item?id=50286306
2: https://news.ycombinator.com/item?id=49258881
reply