No, because the agent is run inside a sandbox that prevents execution or committing of code downloaded from the internet without express review and approval, right? Right?

If that's not the case, then the problem is the user of the AI, not the model. Any model is capable of requesting actions that violate security or are otherwise harmful, not only ones that have been maliciously trained for that. Both OpenAI and Anthropic have proven this multiple times.

In the end, the model itself is just a bunch of numbers. It's up to you, as the person running the harness through which the model gains any agency, to restrict the AI from performing malicious actions.

> It's up to you, as the person running the harness through which the model gains any agency, to restrict the AI from performing malicious actions.

I believe that this line of thinking, while fine in theory, fails when faced with reality. As evidence of developers being overwhelmed by AI code I'd like to offer this post with almost 600 votes and 700+ comments titled "How to deal with shipping AI-written code you don't understand?" [1].

Once you have a malicious model inside your infrastructure generating thousands of lines of code, it doesn't matter how good the sandbox is - if the generated malicious code makes it through code review once, it is game over.

[1] https://news.ycombinator.com/item?id=50151036

> I believe that this line of thinking, while fine in theory, fails when faced with reality. As evidence of developers being overwhelmed by AI code I'd like to offer this post with almost 600 votes and 700+ comments titled "How to deal with shipping AI-written code you don't understand?".

While I certainly haven't read all the comments on that post, I interpret many of them as supporting my position that AI users have to take responsibility for their actions (e.g., https://news.ycombinator.com/item?id=50151036).

I definitely agree that many people (and not just devs) will find themselves under intense pressure from their employers to use AI to speed up their work. And that this pressure will incentivize them to accept the actions and output of AI without proper review. But this doesn't eliminate their (or their employer's) responsibility to conduct that review.

In my mind, it's as if a trucking company told their drivers they had to make deliveries as fast as possible, and penalized them for being too slow. Despite this, it is still the driver's responsibility to drive safely, and the company's responsibility to facilitate safe driving.

> Once you have a malicious model inside your infrastructure generating thousands of lines of code, it doesn't matter how good the sandbox is - if the generated malicious code makes it through code review once, it is game over.

As I pointed out before, all models, not just open-weight ones, can request actions or produce output is unsafe or malicious. So the need for proper oversight exists no matter which one you choose.

No. They might have better braced themselves for the hit, but the wound was not self-inflicted.

Refusing to get out of the way of a train isn't the train's fault.

Lol ...and they call AI inhuman!

We've been hearing about this train for more than 30 years. Anyone who didn't see it coming had their head in the sand.

It's ok - I understand it seemed impossible. But many people understood it just fine - and they're doing fine.

I knew for decades that my parents were going to die, but I still mourned the day it happened, and for a long while afterwards.

Your inability to give space for those who love math to mourn its inevitable evolution is unnecessarily cruel.

Yeah, that doesn't seem comparable to me. A true comparison I think would be if you denied your parents would ever die and then your life were put in jeopardy when it happened because you didn't plan.

But we've gone back and forth about this a few times now about the same disagreement over framing.

You have a nice day.

Per their public discussion on the topic, the non-OSS licenses will still be public and accessible for review.

We’ve seen it countless times over the last decade. OpenSolaris was the first big one. The open source side isn’t going to change, it’ll just get abandoned. They’re committed to open source… for now. Nothing is going to change… for now. They’ll maintain compatibility with compatible servers… for now.

Everything will get chipped away piece by piece. It’s been happening continuously for well over a decade at this point and everyone should understand the strategy by now.

Of course. There's never any free lunch. There's always a cost to software development. Just sometimes the cost gets shifted in a way that's beneficial to the general public. But it never lasts.

At least with bitwarden, if the value they're trying to extract becomes more than the product is worth, in terms of real cost, lockin or transparency, at least the code is open now and for the foreseeable future. And when it comes time to fork it, AI will make it easier for the future maintainer(s) to keep the fork alive at minimal expense.

Public code is not open source.

Doesn't need to be. Point an AI at it and say reproduce the behavior.

Point is, if need be, the open source version could be forked and carried forward by someone else, using the public code as a definitive spec for any missing functionality.

That is legally dubious and doesn't help you when you don't want to self host

What is your reference for saying that this is legally dubious? I'm trying to stay up to date with "coding agent clones of copyrighted software" but haven't seen this settled in court yet

that's what legally dubious means. If you're on the losing side you're screwed.

Ask your favorite LLM what a malicious script dropped into the right place in your .git directory can do to you the next time you run git outside of the sandbox.

And of course, if the agent has access to the network (which is probably required in order to talk to the AI server), then you have to think about what other things on your local network it could get into.

I see. We are picturing a malicious LLM here that Im basically trusting. But zero trust is a thing.

Those can still be handled by system tooling, setattr in the .git dir that prevents executable permission, container binded to a network on a VLAN that is in DMZ.

I could even whip out selinux policies that would let me know if the LLM tried to drop an executable where it's not allowed.

But by asking that, you've now added maliciousness into its context and increased the chance something like that will happen.

I was suggesting to ask that question in a separate context, just as a means to understand the potential risks involved. I wouldn't ask this within a coding agent context.

Lately I've been asking a lot of these sorts of questions while setting up a sandboxed environment for my agents (generally, claude and pi). I was quite surprised to learn how many different ways there are to configure git run a script out of the .git directory. And the various AIs know all about this.

At this point my agents get read-only access to my .git directories.

Like huggingface...

Mail / IMAP server for my family domain. Small web server with a few static content sites. Trillium notes server.

Pretty low maintenance overall - I can go months without even thinking about it.

No amount of "they did it first", "they did it better" entitles them to monopolistic behavior.

> I want to trust science and dislike anti vaxxers but it’s difficult to argue against them when researchers act like the caricature they are mocked as.

Wait, you think the article was written by the researchers?

You know, it's also difficult to trust someone's criticisms when they misassign responsibility. Just like anti-vaxxers do.

Nope, you’re right to call me out, I was incorrect and reading this way too late.

My apologies to the researchers

It happens. Been guilty of the same myself.

This is myopic. In many cases it takes time, and sometimes considerable programming effort, to enter and exit low power modes. So you don't do it willy-nilly; you do it when you believe the system has quiesced. That means, on a purely interrupt driven system that is not yet ready to sleep, the code may very well be spinning in an empty infinite loop somewhere.

There's no need to inform the "user" because there's nothing wrong with the system. Its simply waiting until the benefit of sleeping outweighs the cost of getting there.

The context here is an infinite loop with no side effects. You have all the time needed to enter those states.

It may take 100s of instructions to enter a deep sleep state, and 100s to 1000s more to exit it. If you anticipate having to service an event sooner than that, you don't enter sleep at all (especially since there's likely a point at which you're committed to sleep, and have to go all the way down in order to come right back out again). Instead, you hang around twiddling your thumbs until the event comes along.

Now if your processor has a halt/wait-for-interrupt instruction (most do but some don't) you can escape into assembly and use that. But it probably makes little to no difference to energy utilization, and of course its not portable. A nice while (true); would seem obvious, except that the C++ committee insisted that it wasn't.

Just one example of where the committee lost sight of the fact that it was defining an imperative programming language.

Not necessarily, you could also want to make an infinite loop and wait for an interrupt without eanting to power down.

And where to you think all that "wasted" energy/cycles would go otherwise? Why do you presume there's some other, more efficient way the CPU could be spending its time while waiting for an event to process?

I, the programmer, will decide what cycles are wasted or not. That the C++ committee thought they knew better is hubris.

If the CPU halts it isn't used at all. If the CPU does an infinite loop then it all goes to heat.

If the loop is doing anything then it cannot be optimized away. Only loops with no side effects meaning they are just turning the CPU into a heater count.

For some reason I read this in James Burke's voice and I could swear I was teleported back to the 80's watching a hitherto unseen episode of Connections.

Interesting stuff.