Not trying to throw shade at curl, although I don’t think this feature can realistically be done by a drive-by contributor.
No, the article listed some apps that were not vulnerable, and the first question to me is “so what is vulnerable?” And the answer is “every curl | sh installer on the internet since April 2025” (among assuredly many other less prolific apps).
> Does a system need the knowledge of the full works of Shakespeare to be able to output Javascript?
The same can be said about software engineers. The tricky thing is, it's hard to separate a subset of knowledge from the whole, for both human beings and LLMs
Then stop experimenting until they know how to do it safely. Just think about it, if a bio lab doesn't know how to contain the virus they are experimenting with, would you just shrug and say ok?
If the progress is dangerous, then yes, we need to take careful steps. These hacks are not that big of a deal until something major is broken and human lives are in danger
The brazen disregard for safety is funny in this context, if a little frighteningly indicative about the toxic attitudes that permeate Silicon Valley at the moment.
Anyway, even if we froze training right now LLMs in their current state would have the ability to continue to provide tremendous utility. We’re not asking frontier model providers to delete their previous generations of LLMs (and harnesses), we’re trying to hold them accountable for the crimes they’re committing while training the next generation of LLMs (and harnesses).
All this whining about human safety just results in a complete lack of progress that could’ve probably saved hundreds of millions of lives. Instead all the short-term thinking and lack of appetite for risk is killing us. Everyone’s just thinking of the next 10 years of their life, there don’t seem to be old men that plant trees that they won’t sit in the shade of anymore.
This is some serious touch-grass territory you're treading in. Might I introduce you to the Bottomless Pit of Suffering[1] thought experiment of utilitarianism? Because it's incredibly relevant to the arguments you're presenting.
Disclaimer: I don't generally hold Silicon Valley rationalists in high regard, but this particular piece of writing has always stuck with me quite deeply, as someone who used to be superficially interested in utilitarianism.
Would disagree. It’s not Scott’s conclusion I’m referencing, it’s his thought experiment: “What if there was a bottomless pit of suffering that no matter how much you threw into it, there would be no meaningful progress into reducing that suffering?”
It’s a thought experiment that gets to the core of why fundamentalist utilitarianism (like you’ll often find in SV rationalist communities, especially AI fanatics) is an unbalanced approach to morals and ethics.
Utilitarians can declare that they believe that there is a proverbial universal panacea, and then work backwards to justify within their moral framework literally any action that they take that could be construed as work towards that panacea.
But what if no matter how much effort they put into achieving that panacea, they never make meaningful progress? And all the while they’ve massively increased the amount of suffering that people who they are surrounded by?
As an absurd example, imagine a utilitarian who’s convinced that if we sacrifice infants to the Gods that we’ll get to an AI singularity faster.
That’s an interesting thought experiment and the first time I have come across it. The only problem with it is there is no end to the suffering in that scenario. There is an end state to real utilitarianism and it’s achievable and makes your thought experiment not grounded in reality. There is a state of human development that the majority of people would agree approaches a utopia, which is a tangible and achievable goal.
> There is a state of human development that the majority of people would agree approaches a utopia, which is a tangible and achievable goal.
The Star Trek vs. The Expanse meme is relevant here. There’s no guarantee that there is a continued reduction of suffering in the future, especially due to technological progress. There may be technology that reduces the suffering only for a few select elites while the rest of humanity lives in destitution, unable to meaningfully revolt.
Sanjay has been Google's most tenured IC since forever and is basically as immune to politics as you can be in a megacorp.
He is also rich beyond dreams of avarice and can do basically whatever he wants, but apparently he decided to go hack some more with his buddy Jeff. There's a famous New Yorker story about them: https://www.newyorker.com/magazine/2018/12/10/the-friendship...
Is it surprising? Let's face it - distillation is common practice and every lab does it either intentionally or unintentionally (your training corpus always includes AI generated content in the post ChatGPT era).
> No. Nobody has added any such functionality (yet). Feel free to work on that.
If someone thinks a feature can be useful, especially in the era of agentic coding, instead of asking, do it.
reply