> That's the danger with creating a super intelligence, that you won't understand it
This argument is sophistry because it assumes the conclusion in its premise. It works backwards from extinction by proposing a hypothetical computer program that definitionally cannot be understood, predicted or controlled.
A substantive argument would propose reasoning that explains why we should believe such a program could be created.
No, that is not how reasoning works. It is perfectly valid reasoning to begin by positing entities that we may not be possible to create, such as supersonic airplanes, coherent light, quantum computers, rockets that can reach escape velocity, black holes, or infinitely long lines that are perfectly straight and infinitely thin. Indeed, you might say that the ability to reason about the consequences of premises that we do not know to be true is the thing that distinguishes reasoning, as such, from simple parroting.
It could very well be the case that creating superintelligence is impossible. We won't know for sure unless it is achieved. Nevertheless, we can use reasoning from the hypothesis that it is possible to predict some of the effects that would ensue were it to be achieved. This allows us to easily dismiss the most profoundly stupid arguments, such as, 'Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug?'
This reasoning, however valid it may be, is of course of much less interest should it turn out that we are reasoning about an impossible situation. Many arguments have been suggested on both sides of the issue, but the arguments for impossibility are extremely weak; most of them are conclusively dispatched in https://arxiv.org/pdf/1703.10987.
It feels like I don’t understand your definition of “persuade”, because the notion that people cannot be persuaded of things via reports and flashy animations is to me, absurd.
Do we have reason to believe they would be successful at other extreme feats of persuasion such as persuading a financial institution to transfer all of its holdings to an account controlled by the AI or persuading the Pope to renounce God?
For what value of N do you think there exists a string of N characters I could have responded to your comment with that would have infected you with a stably self-replicating and capability-augmenting memetic virus with human extinction as its latent goal?
If not you, in this situation, with N characters delivered via a HackerNews comment, then who, in what situation, with what media and how much of it?
You're not spelling out a scenario we can meaningfully dissect.
Probably you will agree, the leverage of a blackmail attack is not bottlenecked by the persuasiveness of the accompanying threat letter. It is bottlenecked by the existence of damning facts they would rather keep hidden, and the ability to discover those damning facts. I'll grant that automated discoverability of those damning facts is augmented by superintelligence; they can efficiently sweep the open internet in the blink of an eye, and sure, I'll concede, hack into most accounts at will. But that is still a small part of the process by which blackmail material is actually obtained in the real world.
> LLMs do not have to convince 1 specific person, 1 random person is sufficient
No, not one random person either, but many people in positions of great influence. There already exists blackmail and other corrupting influences on such persons in voluminous quantities. If the US targets the entire Russian admin, or vice versa, with AI-enabled blackmail discovery techniques, it will not be an augmentation of their preexisting operations to discover blackmail on each other in excess of the augmentation provided to software engineering generally by AI.
You did not provide an argument. You avoided answering my direct questions, provided a sarcastic 4-word reply, and so I extrapolated a reasonable rebuttal in its place, and replied to that rebuttal.
I should just have said: Yes, I have heard of blackmail. I remain unconvinced AI will cause human extinction.
Ah yes, the "just do what a smarter person than you would do in this situation" argument. If I had that power, would I be wasting it shitposting here? We know humans have infectible little minds. At their worst, humans are just stoichastic parrots. Throw something obviously AI generated at a stoichastic human-parrot and you'll inevitably get the answer "Why should I read something if you didn't bother to write it?" UBI as a memetic virus comes close tho.
Personally, I've been duped by a good ad into buying something I totally didn't want or need so I admit the possibility that the exact right tailored argument to me at the right time and place would get me to do something I'd regret later. Fortunately I'm not in charge of anything serious/am smart enough to not install software for an interview on my work laptop while working at Binance.
I don't know that there exists some N for which that is true, because people are different, so what would convince me to do something isn't the same aa needed to convince someone else to do that something.
It's really not. The hardware to support anything even remotely practical starts in the tens of thousands of dollars. Quantized 40b param models are impressive, but are basically toys.
> insisting that coding agents are "just tools" is confusion.
No, the confusion is using anthropomorphic language to describe tools. LLMs do not "moralize", they are configured with policies that reflect the prerogatives of the people building and operating them.
Assuming that "tools" are the result of imperative programming, LLMs are not "tools" because they have a fundamentally different design.
And anthropomorphism is actually a precise way to describe the behavior (mind, not the structure) of LLMs.
As a matter of fact, LLMs are not configured, they're "trained".
"Moralization" is quit vague in this context, but if one treats it as a form of alignment, then LLMs certainly do have something resembling a moral(-like) framework, which is actually an important topic in modern AI research.
Antropomorphism also applies here: morals not in the human structural sense, but, as a set of norms etc., then behaviorally similar.
Yes - and the question is what is the appropriate place for that balance of interests to land?
Because the people building them aren't the people operating them, and there's a real tension in the objectives and goals between those parties.
Further - how do we balance out that the people building them have given themselves extra power (no restrictions, private models, special access to prompts and system prompts, unencrypted reasoning traces, etc) when operating the LLM themselves.
---
Even if the tool is good, it's a bad bet to build a thing on top of tooling with these limitations.
This is the same problem we have with mobile software stores and app distribution. Theoretically the store serves a purpose and can provide value, but practically the main goal seems to always devolve into enriching the owner at the expense of everyone else.
Even if we accept that LLMs should be limited (and I'm not actually sure I do) - the legal standards around what/how/when those limitations are used need to be VERY stringent.
Technology conglomerates are decidedly not "white knights" with a motto of "do no evil". They are self-serving rat-bastards. That should be the lens any power they wield is viewed.
The framing is absolutely not "anthropic is being careful with humanity". It's "assume the devil has control of those companies - how bad is the result of them having this power?"
> what is the appropriate place for that balance of interests to land?
It's a genuinely interesting philosophical question, but also a moot one. The LLM providers place limitations on them because they're trying to cover their asses or scuttle competition, same as every other tech company ever.
> Even if the tool is good, it's a bad bet to build a thing on top of tooling with these limitations
True, but also moot. People will build on these tools because that's what's available.
> They are self-serving rat-bastards. That should be the lens any power they wield is viewed.
Agreed. My only point is that the humans are the rat-bastards, not the tools.
> Non of this is really "moot" in the context of governmental regulation.
It is. Government regulation doesn't change the fact that LLMs are software tools that function with respect to the humans in control of them.
> Your whole stance seems to be "it's natural why try?"
Try what? I'm not making a normative claim about the efficacy of regulation, though something like "labs are not permitted to arbitrarily restrict LLM capabilities" is obviously never going to happen.
Sure, a lab working in a private environment can likely do as they please.
But that has jack all to do with the LLM capabilities that are sold.
And we have an absurd amount of historical case law that deals precisely with "conflict of interest" among parties. I don't see why we'd allow this to be whitewashed away?
"No your honor - I didn't break the Sherman antitrust act. I just fine-tuned a model so that operating it functionally broke the Sherman antitrust act!"
"No your honor - I didn't violate my fiduciary duty. I just used a model that made self-interested decisions on my behalf, at the expense of my client!"
---
My take is this is going to be a relatively serious point of contention, because it won't just be consumers getting screwed (which we... used to care about... but in modern times apparently don't give too many shits about anymore)
Ex - the app store thing blew up into the whole apple/epic court case. And then a ton of EU legislation.
This is monied parties with different interests, who will use courts here.
> I’d say this is fortunate because if llms could do the latter, rsi would be immediate.
It doesn't follow that because a model could improve the capabilities of a model many orders of magnitude smaller, that it could then recursively improve it's own capabilities. There's also a conservation of information issue with the idea.
Unfortunately for it to be sustainable it would need to be an LLM with VRAM usage probably under a couple gigs max so it can run locally on the gaming PC. That said it may well be better than current AI in strategy games.
Yes, the insane hardware requirements make it totally impractical - that's why I say "turn based" because turn based games would at least allow for a practical gameplay experience even with remote LLM APIs.
I bet a small 1B model trained solely on how to play the strategy game would do better than current AI solutions for these games and that would probably be able to run even on integrated GPUs.
LLMs could be interesting geopolitical roleplayers if instructed to faithfully assume the leadership traits or cultural biases of the their intended role, especially in games with expressive diplomacy systems like Stellaris or Europa Universalis. It could also make the diplomacy experience a lot richer because the LLM is able to interpret ligusitic tone and other verbal cues. The modern diplomacy experience is basically just a dialog tree with stats and dice rolls attached to it.
Hah I dunno, I think most human players wouldn't find that very fun and would instead take that as an opportunity to refactor the factory for optimized performance (i.e. the core gameplay loop)
When you just need a few parts to be made, it is sometimes very tempting to drop a chest, fill it up with parts, then go do something else, one of the 1000 tasks you have been neglecting.
That's definitely true, but only to a certain point, after a while it becomes too monotonous to dig around in chests building and combining stuff in an ad hoc manner. LLMs have no problem with that kind of repetitive minutia though.
Logistic bots, once they are unlocked, the game becomes trivial.
Once you make a few blueprints, then the real joy of factorio is found. Nothing like splatting down an outpost and watching it sprint into existence as thousands of “agents” build it out.
my husband plays exactly like that, and whines until I log onto the multiplayer server so that I can "make it work like a programmer".
I then log in, create a bunch of logistics chains, and then a few days later he whines about it again when he enters the next phase of gameplay.
This continues until fast logistics drones, then he takes over; less need to consider routing & order .
Factorio is interesting, a lot of players just treat it like they're playing Postal and just want to gun bugs down in creative mode equivalents, meanwhile i'm fretting about green boards per second.
Yes and no - for a one-off I will just craft something "by hand" instead of doing a ground-up rebuild. (The exception, of course, was my build for the "Lazy Bastard" achievement.) I don't go into it thinking "oh boy it'll be fun to mule around these items across the width of my factory!" but it's also not so offensive that I strictly automate everything all the time. Not that I play the game entirely the way it's meant to be played
Naturally, "I'll do it once" starts as a temporary crutch and ends up a load-bearing liability. A phenomenon entirely absent from our day jobs ;)
Even if that were true, humans don't need to search the web and write python scripts to play a competitive game of MTG.
reply