Pool together with some friends and buy an H200 or two to run unquantized open source models with abliteration/heretic transformations.
You need to be able to use these models for the real world and not for some imaginary world where everything is safe and nice and happy all the time, while at the same time intensely surveilled in the name of CYA and the latest panic about whether speech THAT ISN'T EVEN BETWEEN TWO PARTIES is considered "wrong".
I'm a free speech fan that acknowledges there are lots of boundaries of free speech (fraud, perjury, blackmail, defamation), but the one thing that all of the boundaries have in common is that a second party must be involved for them to make any sense at all.
Maybe the courts will uphold this, maybe they won't, but don't take the risk!
Once you go in on infrastructure you have become a small data center. You will need to maintain it, continuously finance it, and secure it. And what if someone wants to back out 6 months later? Now you’re signing contracts with the implication that you’re willing to take a friend to court.
You can get around this by hosting in a 3rd party data center, but now you have the same trust issue again, but with more steps.
It’s all overkill for most people anyways IMO. This lady was just using it as a personal journal. Basically a glorified ELIZA. That kind of thing can be done with really small models locally these days.
> This lady was just using it as a personal journal. Basically a glorified ELIZA. That kind of thing can be done with really small models locally these days.
And she's facing felony charges. Yeah, folks on hacker news can talk about technical solutions all day long, but that's not the fundamental problem. The fundamental problem is that we've allowed our digital infrastructure that includes everything from banks to schools to doctors to become adversarial to us. In some cases predatory.
It's effectively untrustworthy, like being charged a felony for walking across a bridge incorrectly. That's not a "I'll host a small model in my garage" problem. It's a we need the government to do its job problem.
Well this is why I think small local models are the opportunity of a generation. Because it’s a bit transgressive these days to say you don’t need a big centralized service that spies on you and requires a subscription. It’s not the absolute best, but it’s yours.
I agree in general that we technical solutions to social/political problems are suboptimal. But technical solutions don’t require the political system to work, which is a very attractive property at a time when politics seems broken.
There is a counterpoint that retreating into technical solutions only cedes more ground and makes things worse, and for that I don’t have a good reply other than the fact that life is short.
But ultimately it’s not a technology problem it’s a society problem
The people who are accountable and responsible are delegating their authorities to policy and now the policies are just implemented by machine systems and there’s decreasing human intermediation.
The humans that are still inside the system increasingly have less control such that it’s it’s increasingly difficult to find anybody who is the actual customer service type representative who has the authority to make a meaningful impact.
We need actual privacy laws in the united states. Between this, other cloud providers, the advertising tracking infrastructure, phones that constantly ping your location, flock and it's ilk, etc. It's over, no longer can you live semi regularly and still maintain your privacy by maybe living out in the sticks a bit. This is the panopticon.
I mean, this is also just kind of how mandatory reporting works.
If you tell a teacher, a therapist or a priest that you are going to kill somebody, in many states they do actually have to report that and can be held liable for the resulting crime if they do not.
Every time we have a mass shooting in the US it inevitably comes up that they wrote it down or told somebody and then the next logical question becomes “how could we have stopped a mass casualty event?”
I think the issue here is that I don’t want my automated IT to be a mandatory reporter.
My paper journal isn’t a mandatory reporter, neither should my google doc or my claude session.
Even with local models, I don’t want the eventual creep to require local software to report (eg, CSAM reporting from local storage and compute on iphones, etc etc)
Fundamentally, anything going to a second or third party puts you at risk of being ensnared by mandatory reporting laws, whether that is a human or a corporate entity. Loose lips sink ships.
> You can get around this by hosting in a 3rd party data center, but now you have the same trust issue again, but with more steps
No this is not true — colos are very hands off.
You don't need to operate a mini AWS to self-host, and I don't know why people on HN want to make it sound like self-hosting if you don't have nation state security
That’s a pretty broad statement? Maybe a more correct one would be that some are hands off enough that some people might find them worth it.
And that really only addresses one problem. The other one is the coordination problem. You go in on a $30k GPU with 3 other people. So each person puts down 7.5k. But then one person wants to pull out, and so every remaining person needs to put in an extra $2.5k on top, and they’re questioning whether $10k for a chatbot is really worth it.
This level of hardware and cost just isn't necessary. You don't need to team up with your friends to go all in on a hardware purchase, which I've never heard of anyone doing anyway. Go spend some time over at r/LocalLLaMa, you'll see.
It’s been quite exciting that Qwen 3.8 Flash Next has come out: it really is similar to Opus 4.5, 4.6 for coding. Remarkably intelligent, and runs on a single DGX Spark, which I paid $4000 USD for
Sadly they’re now twice that price, which is a shame because I really want a second one!
I have a heretic modified version of it too, for when I want to use it for security and so on. Quite interesting
The 4090 I bought in '23 is selling for ~3x what I paid for it now in late '26. There's a huge squeeze going on with computer hardware using relatively cheap borrowed money. Relatively cheap money is flowing into a (artificially?) limited GPU/memory asset class. And this asset class is growing much faster in value than the interest on that money. So if you borrowed heavily to buy GPUs in the prior 3 years, you're likely coming out well ahead even if you are paying 8 to 10% on the money.
That dynamic will fuel further borrowing until we get to some kind of equilibrium, or some kind of washout occurs where interest rates spike higher and/or the value of GPUs and GPU services start leveling off or even declining. We might get a test soon as the FED has started hiking.
I read that as "you really don't" need rich friends to buy a H200. I see what you meant now. Sorry about that. The shit you hear people say in SV sometimes is wild.
I run them on my 5070. They're moderately useful at that scale I guess.
You can trust another party and do this by renting a few H200s. You cannot pool together with some friends without trusting another party.
You end up just weighing up the difference in trust between a vendor and a friend against the level of disinterest that they might have in your affairs.
I agree up to a certain point, but there has to be some legal boundary between freedom of thought/speech and literally planning a crime. I'm protected under the First Amendment to say "someday I'll rob a bank" but not necessarily "I'll rob this bank on Friday and here's how I plan to do it".
I think you should be allowed to write that exact line in your journal. If you rob the bank that can be used as evidence against you, but in no way is it acceptable for private reflections alone to be used to arrest you. Or else every author who's written a novel with 'bad' characters would be arrestable.
For the record, under current US law, it is not illegal to have a sentence in your locally-stored notes on your phone outlining a plan to commit a crime. There has to be an overt act. The police in that instance could inform the intended victim, surveil you, etc, but they would not be able to successfully charge you with attempted murder. It's not illegal to be considering committing a crime, even if you have a tendency to write down your thoughts.
The law in this particular case, which seems to be intended for threats that you actually send to someone, is being interpreted broadly to apply to any "threat" that you transmit to a server. So in your hypothetical, the legality would depend on whether your notes are backed up to icloud or not.
This is true for the general criminal conspiracy law, but be aware that an “overt act” doesn’t have to be an illegal act, just some action in furtherance of that crime. That can be purchasing a weapon, or scouting out a location. There may also be other laws in play depending on the specifics
Just having saulpw's physical address is already furtherance of that possible crime.
The commenters here are cute little HNers who think they have found a loophole in the law. They are not the first ones innover their head.
Spoiler: the law is written in words, and those words aren't strictly executed like in a computer program, they are interpreted by actual humans who can see what you are trying to do and will stamp it out.
Yup. HNers (and so many normies) believe that there are well defined rules they can follow and be safe. Spoiler, there is one law in the real world and that is might makes right, and you have no might, so therefore you will never be right. China for its many flaws at least is honest about this.
You're absolutely right, at least according to my own quick check on Gemini. I find this state of affairs amazing.
In my country, no "overt act" is required, but both here and in the US a "conspiracy to commit" charge requires an agreement with a second party. This is indeed consistent with a very broad interpretation of "no thought crimes".
You joke, but part of the issue here is we're supposed to have laws that keep cops from reading over everything we type and using it to turn us all into crime suspects. There's a whole constitutional amendment about that, but our personal "papers and effects" that should be protected against unreasonable searches are increasingly really "owned" by others who are all too happy to snitch.
Part of me says that the solution is stop entering any personal data into any device and service you don't own, but I'm not sure if that's really what we want considering that there are zero private cell phones. Even desktops and laptops aren't 100% owned by you these days. The only thing you can really do is keep them offline 100% of the time so they can't spy on you, but that seems like a lot to demand.
They want the network state secured and operating by their rules so they can leverage it's power, while ensuring the proles are kept from doing the same via surveillance.
It's been very obvious for at least the last decade that was the direction things were going.
When you read something describing in detail a person's intent to do something very bad, in a place where they write things that they intend to do, and which in the past they have in fact consistently done, you don't attach any significance to that at all?
I certainly think a police officer that stumbled across such a thing would be justified in asking follow-up questions like 'hey, what's this about you going to shoot someone?' and then maybe making an arrest based on your replies/demeanor.
'Probable cause' should involve a degree of certainty, because 'possible cause' would be altogether too loose of a standard. It's possible that you're intending to shootme and you just mentioned saulpw to throw other HN users off the scent. Possibilities are only limited by the assessor's imagination.
Suppose you go further: You write the "story", and "agree" with a friend to carry it out. Then you go to a gun store and buy a gun ("overt act").
How can I prove that all this is not just a dark and morbid roleplay?
And yet US federal law will convict you for conspiracy to commit murder on this evidence, as I understand it.
If you think the "dark and morbid story" defence is reason enough not to make writing the threat down a crime, why do you think the "dark and morbid roleplay" defence is unsound?
How does the cop at the traffic stop know that your username is akoboldfrying? Did you tell them? If so, why, at a traffic stop, did you do that?
Are you posting threats on hacker news while you are driving? And the cop was close enough to see your username and what you wrote? Is that why you were pulled over?
I don't see how any of these questions are relevant to the point I'm making.
Do you dispute that people sometimes discover things about other people? Do you dispute that things a person discovers about another person can change the first person's assessment of what the second person might do next?
Because it seems to me that you are merely picking apart concrete details of an example situation that are not relevant to the general issue that the example is clearly trying to demonstrate.
I was reaching for a scenario that concisely demonstrated the core aspects of the general issue; maybe the example I chose wasn't very good. The core aspects are what are important, though. They are:
1. Someone in authority can discover recorded information, one interpretation of which is that that someone else is plotting a crime
2. The information might not have been subject to any agreement with (or even seen by) a second person
3. Should that discovery be taken seriously enough to warrant arresting the person?
My claim is that yes it should, unless there are other clearly overriding factors.
Well in your example you've begun conspiring with a second party so that's not at all the same thing. You are at least free to plan all the crimes you'd like to arbitrary levels of detail in private. It's when you start acting things out (soliciting coconspirators, blackmailing targets, etc) that you cross the legal line.
The current situation is a weird one. Anthropic reported single party interactions (per the ToS and common sense), there's a statue about sending threats (as there clearly ought to be), then somehow the definition of the word "send" was tortured by the local police. If a crime has been committed here it's almost certainly an infraction by the local authority against the spirit of the law.
I totally agree. People seem to be stuck on the notion that we must not punish thought crimes, and have elevated this above all other considerations, when really it's just one among several.
However, those other respondents to your post seem to be accurately describing the current legal situation. I asked Gemini, and apparently "conspiring" to commit an offense requires an agreement with another person in both my country and the US, where an "overt act" is also required (that may not be incriminating by itself). I find this alarming. The fact that someone's private diary entry describing in detail a plot to kill me does not amount by itself to anything is... incredible to me.
The thing is plotting a crime is not illegal. Authors do it all the time. Usually against fictional characters, but some authors use a real world backdrop. And look over on StackOverflow. You can get some really weird ones that only make sense when you look into the details. When they manage to show up in the hot question list those details are missing. "How to kill a cat" comes to mind. (Note that you won't find it anymore--the question is AFIAK still there, but the title has been edited several times. The cat in question is the Unix command, not the feline. What do you do when you inadvertently tell the system to display the contents of a large binary file?) And, AFIAK still there, "How to kill Indiana Jones". (There are always the ancient mechanisms that somehow still work perfectly, springs and all. How to make something that actually would work?) And one I've heard of but not seen: "How to kill my wife", from some game I didn't recognize.
Obviously, I'm not talking about the verbatim quotes I provided. There has to be some level of evidence that proves intent to commit a crime and the second quote is meant to represent that whole class of statements, but it depends on context. Any given quote won't constitute evidence in every case, but it will in the cases where it proves intent beyond a reasonable doubt.
I'm not sure why you think my argument holds no water when there are clear legal precedents that speech is not protected in some cases where there is "imminent lawless action".
I'm not a lawyer so take this with a grain of salt, but it seems like there is a big gap between speech that is "directed to inciting or producing imminent lawless action" and just saying that you will do something.
Depending on context saying "I'm going to rob the bank X tomorrow" might also count as a threat?
But, in either case, writing this in a private diary could not be incitement or a threat because you are not communicating with anybody except yourself.
I'm on the other side of the political spectrum here AND YET I entirely wholeheartedly support what the person above wrote: use self-hosted LLMs, abliterated or otherwise. M
For many, AI chats are damn close to extensions of our minds: diaries. Those are supposed to be private.
Was what that woman wrote a credible threat? Was she blowing off steam? Without knowing her deeply, how can you tell?
I'm not saying fake child porn should be allowed or not-allowed, just showing there exist possible exceptions and rationalizations for them even without two parties.
Fake CP is a victimless crime. It is debatable if victimless "crimes" (like being gay in your own home, which is illegal in way too many countries; or eutanasia; or eating pineapple pizza) should be punished at all.
I understand moral panic, disgust, etc, but rationally speaking.
There is an argument that there is a victim, which is all children in that persons vicinity, and all people who care about them, and even society itself as a whole. And the harm is the reasonable elevated risk of harm, combined with the particular helplessness of the targets who are incapable and not responsible for looking out for themselves.
Again I'm not saying it is or isn't a valid argiment. There are 50 easy counter arguments without even trying, but that doesn't mean there is no argument to be had there and there might be 50 counter-counter-arguments in the end.
This is akin to calling all men rapists only because they theoretically can. "All men until no man".
And even in this case, this has nothing to do with fake CP. Surely the threat exists due to the location of a potential perpetrator, regardless if they have fake CP, real CP or no CP at all.
That's definitely something I'd consider if I had cash to spare for H200's! Unfortunately I think for most of us the price of self-hosting has to be 2-5x lower still.
Surely a recurring automatically renewable $1 monthly/weekly/daily contract would solve the problem? Lawyer on a retainer can't really babble about your crimes and is required to protect the confidential information.
That's absolutely not how that works lol. There's a crime/fraud exception to the attorney-client privilege.
Lawyers have a duty of care to the court and they will absolutely drop your ass if you try to make them keep quiet about your crimes.
(This is not to be confused with them representing you in defense. You can tell them about your crimes if the government is trying to get you for said crimes.)
So IIUC, the correct nexus is to only hire a defense lawyer, and only discuss past crimes, probably in a rhetorical context. Otherwise attorney|client privilege only extends to non-criminal actions. Ok.
Way ahead of you bud, mac studio m5 ultra 256gb version is coming soon.... ordered 2x of em just in case. expensive as fuck but its a hedge against all this bullshit and more
A cheap second hand 10ish year old card like my radeon rx570 with 8GB of ram is plenty enough to run a small uncensored model with llama.cpp if all one wants is chitchatting with a clanker.
We are not talking about heavy coding use cases here.
That's exactly what I do. I'm fortunate enough to use the latest preview version, which is known to us beta testers as Notepad With Microsoft Copilot™ and only uses up about 2% of my phone data for telemetry per day (Preview 1.0.912 actually used over 14% daily lol). Plus they got working set size down to 11 gigs and disk usage down to 36.3 gigs.
Preach! I am so tired of Anthropic’s safety team being the arbiters of what is right and wrong. If you so much as hint at impropriety you can have future sessions flagged ad infinitum when it comes to specific topics.
I had a list of controllers that we forgot the usernames and passwords to. We knew it was some combination from about a half dozen of each, just not exactly what was what. A few hundred of them so a tedious task for a human to go through and validate them all.
Got flagged for attempting to have Codex write a quick script to basically dictionary attack my own infrastructure with a tiny dictionary file.
You need to be able to use these models for the real world and not for some imaginary world where everything is safe and nice and happy all the time, while at the same time intensely surveilled in the name of CYA and the latest panic about whether speech THAT ISN'T EVEN BETWEEN TWO PARTIES is considered "wrong".
I'm a free speech fan that acknowledges there are lots of boundaries of free speech (fraud, perjury, blackmail, defamation), but the one thing that all of the boundaries have in common is that a second party must be involved for them to make any sense at all.
Maybe the courts will uphold this, maybe they won't, but don't take the risk!