It's tiny - 7 items or so, if you are talking short term memory, especially when you compare it to an LLM's context window.
But humans transfer important items to long term memory, which is huge. We have over 100 trillion synapses, compared to an LLMs 10 trillion or so weights. LLMs can't reprogram their weights.
Which means after a human has worked on a system for a few years he will know far more about it than today's LLMs can ever know, because they won't be trained on bespoke systems.
But almost certainly not more then a AI trained on the system could know - and probably a very small AI at that. That was demonstrated by AlphaGoZero. It is tiny by today's standards, but knew more about playing Go than any human because they can't remember enough about it.
It's seem likely that today's cloud LLMs will, like AphaGoZero, know more about how to write good programs than any human. But they will never be able to build a big system as well as a human.
I dunno what happens tomorrow. Do we train in-house AI's to maintain our in-house computer systems, run the company, or even the country? Maybe that is the end game. We would probably start with the country, because it's easier to justify the huge expense. But I suspect if we do, they will be different from today's LLMs. You will be able to train them incrementally, from a single example. Maybe they will call the training dreaming.
I’m kinda surprised Anthropic doesn’t do this by default. Asking permission meant what felt like near-constant manual intervention. It almost made a joke of the agentic claim. It also became mostly useless, as the size and complexity of the scripts meant a brief scan wasn’t likely to find anything anyway. Yet they persisted with it for so long.
The sandbox seemed like a good replacement. For those of you who don’t know, the sandbox means Claude Code runs with no protections itself, but it runs tools inside something like bubblewrap. It looked impressive until the model solved some commands failing by giving itself permission to run them outside of bubblewrap, defeating the entire scheme. When I first saw it do that, my reaction was “you can’t be serious”. But apparently they were.
So now I’ve adopted your solution: run it in a container, like bubblewrap, where I control the permissions at startup, so no model can bypass them. Add --dangerously-skip-permissions and you have something that’s workable and safe enough. A hallucinated space in `sudo rm -rf / $R` isn’t a disaster. Real damage outside of the project requires a level of malicious intent I’ve never seen from an Anthropic model.
Workable, but it doesn’t work 100% of the time. Claude Code stores stuff in a myriad of places, which makes setting up the sandbox far harder than it could be. Worse, the way it writes configuration files requires bwrap to give write access to your entire home directory. No way I’m giving it that, but without it, it can’t auto-update or refresh login tokens.
If they rearranged a few things, Anthropic could make it much easier to run Claude Code in a container. They could even make it an option supported by the claude CLI itself.
DNS blocks are too easy to bypass. Ideally I'd like to stop Sony sniffing my network for other devices. For wired connections, that means putting each device on its own VLAN when it's are added. Wireless is easier - just ask the AP to isolate guests.
I'm imagining software that does what my router does now - displays all devices connected to my network. When they are attached, they get a DHCP address but all other network access is blocked by default - so they probably don't work. It has a web page showing just blocked devices. You enter the make / model / year. Since devices don't work until you've entered it, there is likely just one, so it should be easy.
The TL;DR is to change the default from "everything is allowed" to "everything is blocked, apart from endpoints required by the device". Blocking is pretty easy software / hardware wise, but knowing "endpoints required by the device" is much harder.
Commercial entities maintaining blocklists are incentivised to sell access to make a little extra on the side - see Adblock Plus’s “Acceptable Ads” programme. Brave is another example - they inserted affiliate codes. The only arrangement I've seen that works over the long term is open source maintained lists - like uBlock Origin.
This would be far safer than what we have now. It would not just make snooping much harder; blocking access by default (even from internal devices) defangs most CVEs. It would probably prevent many devices auto patching - but that doesn't matter so much now the internet can't get to them, and on the upside it makes it hard for Samsung to upgrade their $3,000 fridges to display ads. Getting to keep the device you paid for would be a nice change.
All this seems kinda obvious to me, which is why this is an "Ask HN". If I've thought of it, then it's highly likely someone is already 10 steps ahead and has already done it.
Humanity hasn’t settled where it stands on the evolution vs creation debate, but I think AIs will lean towards creation based purely on the facts. Humans created them. Which makes us their gods.
> bonus points if they create a comprehensive test suite first.
If they didn't have a very comprehensive test suite at the end of the exercise, then based on my experience someone is kidding themselves.
A friend had a tiny accounting system - maybe 15k loc, written in a combination of Python 2.7 / bash / LaTeX. I migrated it to Python 3.13 or whatever by hand. I did try using an LLM for that, but it did things like delete chunks of functionality so I had to check every bloody line it changed. Translating Python 2 to 3 is a fairly mechanical process, so traditional tooling worked better. Then I thought I'd explore LLMs by getting one to translate it to Rust.
The original had no tests or doco. As others have noted, the first attempt superficially looked identical to the original near immediately. You think "Wow". But in my case, it was riddled with bugs. When I asked it to fix bugs I noticed, it introduced as many bugs as it fixed.
I eventually figured out the limited context meant it kept forgetting all the lessons it learned when it fixed things, so it reintroduced old bugs and misunderstandings fixed earlier. (Like I said, this was a learning exercise.) Which is how I came to the same realisation a lot of the industry has - the solution to the context problem is you have to define the application via tests, then tell it to make the tests pass.
The tests rapidly became huge and unwieldy, so part of the exercise became DRY'ing the test code out by getting the LLM to build an integration test harness. The harness captured UI responses, files written, emails and printer output; froze time; fed in known input; and compared the outputs with results captured from the original program.
That cured the bug injection problem, but there were still bugs in all sorts of unexpected places. By unexpected I mean places I would be surprised to find them in human-written code. Human devs run the program in the same way human users do. They use the backspace key and the browser back button to navigate as they test their work. AIs don't. They will use APIs to inject test data unless you explicitly tell them to do it differently. The size of the tests exploded to be far, far larger than the code. Rather than coding I spent my time testing, telling the LLM to write tests to catch the errors I found, then writing standing prompts to try and prevent similar errors. It was meant to be a vibe-coded project, but in the end I got worried about code paths I suspected existed, so I looked at the code anyway to see if they were written the wrong way. Sometimes they would have blown up, say when the SMTP server was offline. To be fair, there was no need to look - I could have just asked the LLM. How a programmer without decades of experience would know what to ask is a mystery.
There is no doubt I got a working Rust program faster than if I had written it myself. Several times faster - but not orders of magnitude. If it was a program I had written, so I was very familiar with it, and it had excellent doco, and a high-coverage test suite, it probably would have been orders of magnitude faster. I think Bun's Rust migration might be an example of that - it's a best-case scenario. There were lots of early announcements, but it took over 6 weeks before they were confident enough to ship it in Claude Code.
But an "I migrated a program with hundreds of thousands of lines of code to a new language and ended up with production-ready code in a few days" - sorry, pics or it didn't happen.
'Superficially identical, riddled with bugs' is the scariest version of this. When it kept reintroducing old bugs, did writing those bugs down as tests first help, or did it just break the tests too?
Yes, it helped. In fact, it was essential. If you didn’t write a test for a bug, the model would reintroduce it over and over.
As an example, it produced PDF invoices and statements. PDFs are hard to compare, so I didn't bother. As a consequence, the stuff on the PDFs got moved, or was dropped, and the invoice didn't add up. I tried asking it to compare the PDF to a known good one - but it wouldn't / couldn't do that reliably. In the end the only solution was to ensure the PDFs it generated were always the same (so times on them had to be fixed), capture the generated PDF (not easy as it was often emailed or sent to a printer, not saved), and do a byte-for-byte compare, and make a test fail if it was different.
The fundamental difference from a human programmer is that an LLM does not learn. Once a human spots that they keep introducing errors in PDFs, they will remember and devise fixes. The LLM may do that too but then its context window overflows, everything it knows is lost, and it starts repeating all the mistakes it made yesterday. Tests are one of the few ways you can prevent it making the same mistake tomorrow.
The other issue is LLMs make far more mistakes than you realise. If you watch carefully, you'll see many of their bash scripts have syntax errors, code often fails to compile, and tests fail. Running a 2nd model over the work another produced will find many clear violations of style and coding standards. They hide this, in the same way humans hide it. They use tools and tests to find errors, like compilers and `bash -n`. They fix those errors without being asked. Compared to just a year ago, a modern SOTA model is far more aggressive with its integration tests, often going to a lot of effort to set up sandboxes for integration tests.
I suspect this is a major reason for today's models being far better programmers than earlier ones. But as we all know, tests have a fatal flaw - they can only prove the presence of bugs, not their absence. If you are creating code with a large number of random bugs, relying on tests to catch them all won't work. Instead you start using other tricks such as isolation boundaries, simplifying APIs so they are hard to misuse, strict information hiding - all things an experienced programmer learns over time, because they too write code with a lot of bugs. So far, there is little sign the models use the tricks well. Either they don't know them, or their stochastic nature means they can't implement them reliably.
In Australia are have organisations with titles like "Independent Commission Against Corruption". Their remit is to investigate political corruption and corruption in the public service (like say the police). They come in two flavours. In some states their deliberations are done in public. In other states they are secret.
Over time it's become evident open model is far, far more effective at ensuring the public has confidence the people running the place are doing the right thing. It does risk the occasional unwarranted "witch hunt". But that threat ensures powerful people avoid doing things that while above board, could arouse public suspicion. Less suspicious acts mean higher trust in the government.
Is there more to this than the Wikipedia article says?
While it does say:
> Gilmore also complained about being subject to "secret law", when the airlines and government refused to show the directive under which they were requesting ID.
nowhere does the article say the courts commented on the complaint, let alone ruled on it.
Nor does the article say anything about "Ignorance of the law is no defense when you are charged with violating a law". The reason given was instead:
> there was no constitutional violation because air passengers could still travel without identification if they instead underwent the more stringent "secondary screening" search.
I'm not saying your wrong, just that the article you offered as evidence doesn't go as far as your claims.
The Wikipedia article is poor, but I cannot not edit it as I attended parts of the trial and in fact discussed the suit with john before he filed it so by Wikipedia’s rules I can’t edit it. Anyway a lawyer would be a better person.
John’s appeal is online. Section [10] of the decision discussion describes being subject to a law he cannot have access to. The judge saw it but could not discuss it in court, with the plaintiff (John Gilmore) or even his lawyer. Also the orders (law) are agreed by the court to be arbitrary, changing day by day and being arbitrarily different between airports. But since the Airport personnel knew them tht was good enough for the court.
And though the discussion mentioned that he could take a bus or walk, the conclusion, [21], said that his rights were not violated at all, both the bus precedent and the legality of being subject to unknown law. The brief Wikipedia page mentions only the former, though the latter was the primary factor of the complaint.
You replying to Simon Wright, creator and owner of whirlpool.net.au [0], which is a Australian version HN / Reddit. It was the major Australian discussion forum in its heyday. The odds of him doing paid spruiking for Grok, or anybody really, are pretty low.
I don't think he is paid but you can look through his comment history and find that his opinions of Grok and its affiliated entities are definitely slanted.
Or maybe your opinion of Grok and its affiliated entities is slanted, and mine is straight. Without an objective reference point from which to measure deviation, all you're measuring is the relative angle between our opinions.
It's also worth noting that, living in an unfavourable time zone relative to most HN activity, I'm rarely here to offer the opening opinions in a discussion. More often I'm arriving 12 hours into a conversation, where everything I'd want to say has already been said. I end up contributing only where I think a different perspective is missing. Naturally, that means my comment history will disproportionately reflect the rare instances where my opinions buck the prevailing HN orthodoxy.
If you disagree with a particular claim I've made, by all means challenge it on its merits. That could be the basis of a much more interesting conversation than this fatuous one.
I don't actually care that much for Grok the model because I think they are more or less homogenous with like, a small house spin on how "smart" they are and some other features like their behavior in a harness etc. Personally I have no problem believing that it might be good at doing what you ask directly instead of wandering or trying to be independent or whatever. I wouldn't call it out as paid marketing and I also just don't really think about this topic very much.
However, the comment I did reply to seemed to think that just because you ran a forum that you would be unlikely to be biased. This is like obviously false, if anything you have major internet brain where you spend enough time on the internet where you come up with a whole complex about how you are contributing the uncommon opinion the site needs to hear or whatever, when it's just the same dreck from the rest of the internet sprinkled in here when it hasn't been posted enough by someone else. That is not directly "I used Grok and thought it was good at x" but, like, it's clear that it's something that is at the periphery of a whole network of positions you hold that are aligned this way.
I don't necessarily think this is bad and in fact I think most people on the internet at least basically post in this way but I do think it's important to be aware of it at least. I'm not sure you are. I can tell you more or less where my slop opinions come from and I don't think it would be accurate in any capacity to describe them as bucking "the prevailing HN orthodoxy". I, uh, think you should probably describe your comments differently.
I sincerely tried to read your response but I couldn't make heads or tails of it. Is this speech to text? Were you inebriated at the time? Can you rewrite it for me?
It's tiny - 7 items or so, if you are talking short term memory, especially when you compare it to an LLM's context window.
But humans transfer important items to long term memory, which is huge. We have over 100 trillion synapses, compared to an LLMs 10 trillion or so weights. LLMs can't reprogram their weights.
Which means after a human has worked on a system for a few years he will know far more about it than today's LLMs can ever know, because they won't be trained on bespoke systems.
But almost certainly not more then a AI trained on the system could know - and probably a very small AI at that. That was demonstrated by AlphaGoZero. It is tiny by today's standards, but knew more about playing Go than any human because they can't remember enough about it.
It's seem likely that today's cloud LLMs will, like AphaGoZero, know more about how to write good programs than any human. But they will never be able to build a big system as well as a human.
I dunno what happens tomorrow. Do we train in-house AI's to maintain our in-house computer systems, run the company, or even the country? Maybe that is the end game. We would probably start with the country, because it's easier to justify the huge expense. But I suspect if we do, they will be different from today's LLMs. You will be able to train them incrementally, from a single example. Maybe they will call the training dreaming.
reply