Also, the page you link to in the original post is in German; it might be nice to have an English page for the mostly English speaking audience here on HN
And I had been looking at it on mobile, which might have been why I was having an issue with the site; now that I'm looking at the site on a desktop browser it looks much better.
Data center power usage has been fairly flat for the last decade (until 2022 or so). While new capacity has been coming online, efficiency improvements have been keeping up, keeping total usage mostly flat.
The AI boom has completely changed that. Data center power usage is rocketing upwards now. It is estimated it will be more than 10% of all electric power usage in the US by 2030.
It's a completely different order of magnitude than the pre AI-boom data center usage.
The first chart in your link doesn't show "flat" usage until 2022? It is clearly rising at an increasing rate, and it more than doubles over 2014-2022.
It might help to look at global power usage, not just the US, see the first figure here:
I think you're referring to Figure ES-1 in that paper, but that's kind of a summary of different estimates.
Figure 1.1 is the chart I was referring to, which are the data points from the original sources that it uses.
Between 2010 and 2020, it shows a very slow linear growth. Yes, there is growth, but it's quite slow and mostly linear.
Then the slope increases sharply. And the estimates after that point follow the new, sharper growth.
Sorry, when I wrote my original comment I didn't have the paper in front of me, I linked it afterwards. But you can see that distinct change in rate at around 2020.
ES-1 is the most important figure, though? As you say, it is a summary, and the authors consider it their best estimate, hence they put it first, and in the executive summary.
Figure 1.1 does show a single source from 2018 (Shehabi et al) that estimates almost flat growth up to 2017, that's true, but the same graph shows other sources with overlap on the same time frame as well, and their estimates differ (though they don't span enough years to really tell one way or another).
I still wouldn't say that your assertion that data center energy use was fairly flat until 2022 is true. Even in Figure 1.2, for global data center usage, tracks more in line with the estimates in the executive summary. It just seems like the run-of-the-mill exponential increase with the same rate since at least 2014, a good amount of time before genAI was used heavily.
Basing off Yahoo historical price data, Bitcoin prices first started being tracked in late 2014. So my guess would be the increase from then to 2022 could have largely been attributed to crypto mining.
The energy impact of crypto is rather exaggerated. Most estimates on this front are aiming to demonstrate as a high value as possible, and so should be taken as higher upper bound, and yet even that upper bound is 'only' around 200TWh a year. Annual energy consumption is in the 24,000TWh range with growth averaging around 2% or so per year.
So if you looked at a graph of energy consumption, you wouldn't even notice crypto. In fact even LLM stuff will just look like a blip unless it scales up substantially more than its currently trending. We use vastly more more energy than most appreciate. And this is only electrical energy consumption. All energy consumption is something like 185,000 TWh. [1]
This is where the debate gets interesting, but I think both sides are cherrypicking data a bit. The energy consumption trend depends a lot on what baseline you're measuring from and which metrics you prioritize.
Yes, data center efficiency improved dramatically between 2010-2020, but the absolute scale kept growing. So you're technically both right: efficiency gains kept/unit costs down while total infrastructure expanded. The 2022+ inflection is real though, and its not just about AI training. Inference at scale is the quiet energy hog nobody talks about enough.
What bugs me about this whole thread is that it's turning into "AI bad" vs "AI defenders," when the real question should be: which AI use cases actually justify this resource spike? Running an LLM to summarize a Slack thread probably doesn't. Using it to accelerate drug discovery or materials science probably does. But we're deploying this stuff everywhere without any kind of cost/benefit filter, and that's the part that feels reckless.
I periodically ask them questions about topics that are subtle or tricky, and somewhat niche, that I know a lot about, and find that they frequently provide extremely bad answers. There have been improvements on some topics, but there's one benchmark question that I have that just about every model I've tried has completely gotten wrong.
Tried it on LMArena recently, got a comparison between Gemini 2.5 flash and a codenamed model that people believe was a preview of Gemini 3 flash. Gemini 2.5 flash got it completely wrong. Gemini 3 flash actually gave a reasonable answer; not quite up to the best human description, but it's the first model I've found that actually seems to mostly correctly answer the question.
So, it's just one data point, but at least for my one fairly niche benchmark problem, Gemini 3 Flash has successfully answered a question that none of the others I've tried have (I haven't actually tried Gemini 3 Pro, but I'd compared various Claude and ChatGPT models, and a few different open weights models).
So, guess I need to put together some more benchmark problems, to get a better sample than one, but it's at least now passing a "I can find the answer to this in the top 3 hits in a Google search for a niche topic" test better than any of the other models.
Still a lot of things I'm skeptical about in all the LLM hype, but at least they are making some progress in being able to accurately answer a wider range of questions.
I don't think tricky niche knowledge is the sweet spot for genai and it likely won't be for some time. Instead, it's a great replacement for rote tasks where a less than perfect performance is good enough. Transcription, ocr, boilerplate code generation, etc.
The thing is, I see people use it for tricky niche knowledge all the time; using it as an alternative to doing a Google search.
So I want to have a general idea of how good it is at this.
I found something that was niche, but not super niche; I could easily find a good, human written answer in the top couple of results of a Google search.
But until now, all LLM answers I've gotten for it have been complete hallucinated gibberish.
Anyhow, this is a single data point, I need to expand my set of benchmark questions a bit now, but this is the first time that I've actually seen progress on this particular personal benchmark.
That’s riding hype machine and throwing baby with bath water.
Get an API and try to use it for classification of text or classification of images. Having an excel file with somewhat random looking 10k entries you want to classify or filter down to 10 important for you, use LLM.
Get it to make audio transcription. You can now just talk and it will make note for you on level that was not possible earlier without training on someone voice it can do anyone’s voice.
Fixing up text is of course also big.
Data classification is easy for LLM. Data transformation is a bit harder but still great. Creating new data is hard so like answering questions where it has to generate stuff from thin air it will hallucinate like a mad man.
The ones that LLMs are good in are used in background by people creating actual useful software on top of LLMs but those problems are not seen by general public who sees chat box.
Or more likely Google couldn't give a rat's arse whether those AI summaries are good or not (except to the degree that people don't flee it), and what it cares is that they keep users with Google itself, instead of clicking of to other sources.
After all it's the same search engine team that didn't care about its search results - it's main draw - activey going shit for over a decade.
Those summaries would be far more expensive to generate than the searches themselves so they're probably caching the top 100k most common or something, maybe even pre-caching it.
I also use niche questions a lot but mostly to check how much the models tend to hallucinate. E.g. I start asking about rank badges in Star Trek which they usually get right and then I ask about specific (non existing) rank badges shaped like strawberries or something like that. Or I ask about smaller German cities and what's famous about them.
I know without the ability to search it's very unlikely the model actually has accurate "memories" about these things, I just hope one day they will acutally know that their "memory" is bad or non-existing and they will tell me so instead of hallucinating something.
I'm waiting for properly adjusted specific LLMs. A LLM trained on so much trustworth generic data that it is able to understand/comprehend me and different lanugages but always talks to a fact database in the background.
I don't need an LLM to have a trillion parameters if i just need it to be a great user interface.
Someone is probably working on this somewere or will but lets see.
Basically making sense of unstructured data is super cool. I can get 20 people to write an answer the way they feel like it and model can convert it to structured data - something I would have to spend time on, or I would have to make form with mandatory fields that annoy audience.
I am already building useful tools with the help of models. Asking tricky or trivia questions is fun and games. There are much more interesting ways to use AI.
Well, I used Grok to find information I forgot about like product names, films, books and various articles on different subjects. Google search didn't help but putting the LLM at work did the trick.
So I think LLMs can be good for finding niche info.
So this is an interesting benchmark, because if the answer is actually in the top 3 google results, then my python script that runs a google search, scrapes the top n results and shoves them into a crappy LLM would pass your benchmark too!
Which also implies that (for most tasks), most of the weights in a LLM are unnecessary, since they are spent on memorizing the long tail of Common Crawl... but maybe memorizing infinite trivia is not a bug but actually required for the generalization to work? (Humans don't have far transfer though... do transformers have it?)
I've tried doing this query with search enabled in LLMs before, which is supposed to effectively do that, and even with that they didn't give very good answers. It's a very physical kind of thing, and its easy to conflate with other similar descriptions, so they would frequently just conflate various different things and give some horrible mash-up answer that wasn't about the specific thing I'd asked about.
So it's a difficult question for LLMs to answer even when given perfect context?
Kinda sounds like you're testing two things at the same time then, right? The knowledge of the thing (was it in the training data and was it memorized?) and the understanding of the thing (can they explain it properly even if you give them the answer in context).
Counter point about general knowledge that is documented/discussed in different spots on the internet.
Today I had to resolve performance problems for some sql server statement. Been doing it years, know the regular pitfalls, sometimes have to find "right" words to explain to customer why X is bad and such.
I described the issue to GPT5.2, gave the query, the execution plan and asked for help.
It was spot on, high quality responses and actionable items and explanations on why this or that is bad, how to improve it and why particularly sql may have generated such a query plan. I could instantly validate the response given my experience in the field. I even answered with some parts of chatgpt on how well it explained. However I did mention that to customer and I did tell them I approve the answer.
Asked high quality question and receive a high quality answer. And I am happy that I found out about an sql server flag where I can influence particular decision. But the suggestion was not limited to that, there were multiple points given that would help.
Even the most magical wonderful auto-hammer is gonna be bad at driving in screws. And, in this analogy I can't fault you because there are people trying to sell this hammer as a screwdriver. My opinion is that it's important to not lose sight of the places where it is useful because of the places where it isn't.
The problem with publicly disclosing these is that if lots of people adopt them they will become targeted to be in the model and will no longer be a good benchmark.
Obviously, the fact that I've done Google searches and tested the models on these means that their systems may have picked up on them; I'm sure that Google uses its huge dataset of Google searches and search index as inputs to its training, so Google has an advantage here. But, well, that might be why Googles new models are so much better, they're actually taking advantage of some of this massive dataset they've had for years.
This thought process is pretty baffling to me, and this is at least the second time I've encountered it on HN.
What's the value of a secret benchmark to anyone but the secret holder? Does your niche benchmark even influence which model you use for unrelated queries? If LLM authors care enough about your niche (they don't) and fake the response somehow, you will learn on the very next query that something is amiss. Now that query is your secret benchmark.
Even for niche topics it's rare that I need to provide more than 1 correction or knowledge update.
I have a bunch of private benchmarks I run against new models I'm evaluating.
The reason I don't disclose isn't generally that I think an individual person is going to read my post and update the model to include it. Instead it is because if I write "I ask the question X and expect Y" then that data ends up in the train corpus of new LLMs.
However, one set of my benchmarks is a more generalized type of test (think a parlor-game type thing) that actually works quite well. That set is the kind of thing that could be learnt via reinforcement learning very well, and just mentioning it could be enough for a training company or data provider company to try it. You can generate thousands of verifiable tests - potentially with verifiable reasoning traces - quite easily.
Ok, but then your "post" isn't scientific by definition since it cannot be verified. "Post" is in quotes because I don't know what you're trying to but you're implying some sort of public discourse.
1. The purpose of the benchmark is to choose what models I use for my own system(s). This is extremely common practice in AI - I think every company I've worked with doing LLM work in the last 2 years has done this in some form.
> To me it's in the same spirit as claiming to have defeated alpha zero but refusing to share the game.
This is an odd way of looking at it. There is no "winning" at benchmarks, it's simply that it is a better and more repeatable evaluation than the old "vibe test" that people did in 2024.
I see the potential value of private evaluations. They aren't scientific but you can certainly beat a "vibe test".
I don't understand the value of a public post discussing their results beyond maybe entertainment. We have to trust you implicitly and have no way to validate your claims.
> There is no "winning" at benchmarks, it's simply that it is a better and more repeatable evaluation than the old "vibe test" that people did in 2024.
Then you must not be working in an environment where a better benchmark yields a competitive advantage.
> I don't understand the value of a public post discussing their results beyond maybe entertainment. We have to trust you implicitly and have no way to validate your claims.
In principle, we have ways: if nl's reports consistently predict how public benchmarks will turn out later, they can build up a reputation. Of course, that requires that we follow nl around for a while.
My question was "What's the value of a secret benchmark to anyone but the secret holder?"
The root of this whole discussion was a post about how Gemini 3 outperformed other models on some presumably informal question benchmark (a"vibe test"?). When asked for the benchmark, the response from the op and and someone else was that secrecy was needed to protect the benchmark from contamination. I'm skeptical of the need in the op's cases and I'm skeptical of the effectiveness of the secrecy in general. In a case where secrecy has actual value, why even discuss the benchmark publicly at all?
The point is that it's a litmus test for how well the models do with niche knowledge _in general_. The point isn't really to know how well the model works for that specific niche.
Ideally of course you would use a few of them and aggregate the results.
I actually think "concealing the question" is not only a good idea, but a rather general and powerful idea that should be much more widely deployed (but often won't be, for what I consider "emotional reasons").
Example: You are probably already aware that almost any metric that you try to use to measure code quality can be easily gamed. One possible strategy is to choose a weighted mixture of metrics and conceal the weights. The weights can even change over time. Is it perfect? No. But it's at least correlated with code quality -- and it's not trivially gameable, which puts it above most individual public metrics.
It's hard to have any certainty around concealment unless you are only testing local LLMs. As a matter of principle I assume the input and output of any query I run in a remote LLM is permanently public information (same with search queries).
Will someone (or some system) see my query and think "we ought to improve this"? I have no idea since I don't work on these systems. In some instances involving random sampling... probably yes!
This is the second reason I find the idea of publicly discussing secret benchmarks silly.
I learned in another thread there is some work being done to avoid contamination of training data during evaluation of remote models using trusted execution environments (https://arxiv.org/pdf/2403.00393). It requires participation of the model owner.
Don't the models typically train on their input too? I.e. submitting the question also carries a risk/chance of it getting picked up?
I guess they get such a large input of queries that they can only realistically check and therefore use a small fraction? Though maybe they've come up with some clever trick to make use of it anyway?
Yeah, probably asking on LMArena makes this an invalid benchmark going forward, especially since I think Google is particular active in testing models on LMArena (as evidenced by the fact that I got their preview for this question).
I'll need to find a new one, or actually put together a set of questions to use instead of just a single benchmark.
Heres my old benchmark question and my new variant:
"When was the last time England beat Scotland at rugby union"
new variant
"Without using search when was the last time England beat Scotland at rugby union"
It is amazing how bad ChatGPT is at this question and has been for years now across multiple models. It's not that it gets it wrong - no shade, I've told it not to search the web so this is _hard_ for it - but how badly it reports the answer. Starting from the small stuff - it almost always reports the wrong year, wrong location and wrong score - that's the boring facts stuff that I would expect it to stumble on. It often creates details of matches that didn't exist, cool standard hallucinations. But even within the text it generates itself it cannot keep it consistent with how reality works. It often reports draws as wins for England. It frequently states the team that it just said scored most points lost the match, etc.
It is my ur example for when people challenge my assertion LLMs are stochastic parrots or fancy Markov chains on steroids.
I also have my own tricky benchmark that up til now only Deepseek has been able to answer. Gemini 3 Pro was the second. Every other LLM fail horribly. This is the main reason I started looking at G3pro more seriously.
This is an experimental homebuilt aircraft, so the regulations on manufacture and replacement parts are much less stringent than commercially sold aircraft.
Also, the part it was replacing was a fiberglass part in an epoxy resin, with a glass transition temperature of 84°C. So the 105°C glass transition temperature of the replacement part should have been better than the original.
However, the original had an aluminum tube supporting the inlet, which provided extra structural support beyond fiberglass epoxy resin. And upon testing, the actual glass transition temperature of the 3D printed part was 52.8°C for one sample and 54.0°C for another, so much lower than expected.
Now, because the regulations are much less strict for experimental homebuilt aircraft, there may not be the traceability to figure out where in the chain the issue came up. Was it a bad batch of filament? Did the person making the part use the wrong kind of filament? Who should have tested the glass transition temperature of a coupon of the same material as the replacement part? Did the 3d printed material glass transition temperature change over time, possibly due to something like fuel or exhaust fumes?
The recommendation from this report is to disallow 3d printed replacements for this part, but it should be possible to do with the right material and proper testing and analysis (as well as leaving in the aluminum tube for additional support), as this is an air intake and it should be possible to find a 3d printed material that can withstand the kind of temperatures an air intake is subjected to, given that the original part is a fiberglass with epoxy resin.
All of this sounds like something you could just do yourself after putting a piece down for a day or two and coming back to it with fresh eyes. What benefit is there of cooking the oceans with a bullshit generator?
Like, sure, it's possible to do this with an LLM, but it's also possible to do it without, at roughly similar levels of effort, without contributing to all of the negative externalities of the LLM/genAI ecosystem.
Because the complaints about the power and water usage of AI are mostly motivated reasoning. I don't like AI, therefore I'm going to find a reason not to like it. I
Listen, if it's Greta Thunberg pointing out that AI datacenters use a lot of resources, yeah, I'm willing to listen. But when the voices saying "but what about all the water/electricity is wasting" is coming from individuals I know personally haven't previously given a shit about the planet or conservation or recycling and have made fun of me for reusing things instead of throwing stuff into the garbage, I'm sorry, but those complaints from those individuals fall on deaf ears. Not saying you are, just a theme I've noticed with people in my life.
So all uses of water, land, and power are the same?
They have no grading in terms of importance and priority, especially in a world contending with climate change, lack of arable land, lack of drinkable water, and so on? AI usage of these resources is on par with every other use?
That's one of the big problems with any kind of AI alignment or safety work.
Safety for whom? Alignment to whose needs?
And a lot of time, that's contextual. You don't necessarily want to be effortlessly crafting novel exploits for a ransomware attacker, but you do want to be able to create a PoC exploit when deciding the severity of a CVE.
Or one valid use of an LLM is to craft examples of various kinds of abuse for training some smaller simpler model as a classifier.
So yeah, in trying to create a general purpose tool and then applying some notion of alignment or safety, you are automatically limiting some use cases that are valid for certain people.
> That's one of the big problems with any kind of AI alignment or safety work.
That's why I found this announcement interesting, with regard to its discussion of alignment. Alignment as you're talking about here centers around ethics and a moral framework and is so named because a lot of the early LLM folks were big into "artificial general intelligence" and the fear that the AI will take over the world or whatever.
But fundamentally, and at a technical level, the "alignment" step is just additional training on top of the pre-training of the gigantic corpus of text. The pre-training kind of teaches it the world model and English, and "alignment" turns it into a question and answer bot that can "think" and use tools.
In other words, there's plenty of non-controversial "alignment" improvements that can be made, and indeed the highlight of this announcement is that it's now less susceptible to prompt injection (which, yes, is alignment!). Other improvements could be how well it uses tools, follows instructions, etc.
Please, when commenting on the title of a story on HN: include the title that you are commenting on.
The admins regularly change the title based on complaints, which can be really confusing when the top, heavily commented thread is based on the original title.
According to the Wayback machine, the title was "OpenAI ends legal and medical advice on ChatGPT", while now when I write this the title is "ChatGPT terms disallow its use in providing legal and medical advice to others."
Yes, the mod stopped posting publicly around the time she went to prison. That seems to have been the catalyst for the conspiracy theory.
But it was actually a couple days apart; he stopped posting before she went to prison. And he actually posted to some private subs, and was involved in some DMs, after he stopped posting publicly and after she went to prison.
There's really very little evidence other than a vague coincidence of when he left Reddit and when she went to prison, and the name.
And, like, if she were posting anonymously, why would she use that name?
It's basically just completely incoherent. Like many conspiracy theories, they take a lot of other random data points, and if you sift through and cherry pick enough data points you can find others that taken out of context look like coincidences. But that's just because you're cherry picking between two large distributions of data.
>It's basically just completely incoherent. Like many conspiracy theories,
These are absurdly strong claims. This isn't an incoherent theory... it's inconclusive, sure. Unprovable? Probably (difficult to imagine what would have to change to find out with certainty one way or the other).
There's no need to respond to counter-evidence that can't be independently verified. The people claiming that (Reddit corporate, basically) have an interest in distancing themselves from her.
It's incoherent. It simply makes no sense at all that Ghislaine Maxwell, a wealthy socialite, would post a bunch of stuff on Reddit for years to accumulate meaningless internet points, under something based on her own name, while somehow also trying to be anonymous.
This is a simply incoherent theory. There's no sense in it. You don't post "anonymously" under your own name.
It’s really not that hard to believe, rich people get bored and use the internet too you know. They’re just not that special and I find this narrative about how it couldn’t possibly be her really weird and deferential. We only need 33bits to deanonymise people on the internet, count the bits.
>It simply makes no sense at all that Ghislaine Maxwell, a wealthy socialite, would post a bunch of stuff on Reddit for years to accumulate meaningless internet points, under something based on her own name, while somehow also trying to be anonymous.
When do normal people ever make sense? And she's not exactly normal, she pimped teenage girls to her boyfriend (? whatever he was ?). All usernames leak something about the person who chose them. They're trying to be clever, come up with the perfect joke or pop culture reference. Except for people who aren't that clever, and then it was the third grade class's pet hamster's name with their birthday tacked onto the end. When you say "but she wouldn't do a username based on her own identity"... you're seriously overthinking this. Don't you feel a little silly pretending that she was some supervillain genius who wouldn't ever make such a classic blunder?
>But it really doesn't take this much detail to realize that this conspiracy theory is incoherent
Incoherent in that dozens or hundreds of people all contributed to it in an unorganized fashion over a period of days/weeks on reddit? Sure, 100%. Incoherent as in it makes no sense and doesn't have a shred of plausibility? 0%
>The entirety of the "evidence" is standard conspiracy stuff,
This is just blatant lying at this point. Standard conspiracy stuff is that the Illuminati, working with the Rosicrucians and enslaved sasquatches are blackmailing the CIA to use their mind control satellites on the Spanish royalty in an attempt to foment nuclear war between Gibraltar and Cameroon.
The "this convicted felon fucked around on reddit when she wasn't busy being a degenerate pervert sex monster" isn't standard conspiracy stuff. It's run-of-the-mill no shit sherlock territory. And you're insulting for claiming otherwise.
Also, the page you link to in the original post is in German; it might be nice to have an English page for the mostly English speaking audience here on HN