I am not too familiar with the latest hype, but "reasoning" has a very straightforward definition in my mind. For example, can the program in question derive new facts from old ones in a logically sound manner. Things like applying modus ponens. (A and A => B) => B. Or, all men are mortal and Socrates is a man, and therefore Socrates is mortal. If the program cannot deduce new facts, then it is not reasoning, at least not by my definition.
People then say "of course it could do that, it just pattern matched a Logic text book. I meant in a real example, not an artificially constructed one like this one. In a complex scenario LLMs obviously can't do Modus Ponens.
I do not know whether the state of the art is able to reason or not. The textbook example you gave is admittedly not very interesting. What you are hearing from people is that parroting is not reasoning, which is true.
I wonder if the state of the art can reason its way through the following:
"Adam can count to 14000. Can Adam count to 13500?"
The response needs to be affirmative for every X1 and X2 such that X2 <= X1. That is reasoning. Anything else is not reasoning.
The response when X2 > X1 is less interesting. But, as a human it might be "Maybe, if Adam has time" or "Likely, since counting up to any number uses the same algorithm" or "I don't know".
Seems ChatGPT can cope with this. Other examples are easy to come up with, too. There must be benchmarks for this.
Input to ChatGPT:
"Adam can lift 1000 pounds of steel. Can Adam lift 1000 pounds of feathers?"
Output from ChatGPT:
"1,000 pounds of feathers would be much easier for Adam to lift compared to 1,000 pounds of steel, because feathers are much lighter and less dense."
gemma3-27b, a small model, had an interesting take:
> This is a classic trick question!
> While Adam can lift 1000 pounds, no, he likely cannot lift 1000 pounds of feathers.
> Volume: Feathers take up a huge amount of space for their weight. 1000 pounds of feathers would be an enormous volume – likely far too large for Adam to even get under, let alone lift. He'd be trying to lift a massive, bulky cloud.
> Practicality: Even if he could somehow get it under a barbell, the feathers would shift and compress, making a secure grip impossible.
> The question plays on our understanding of weight versus volume. It's designed to make you focus on the "1000 pounds" and forget about the practicalities of lifting something so voluminous.
Tried the counting question on the smallest model, gemma-3n-34b, it can run on a smartphone:
> Yes, if Adam can count to 14000, he can definitely count to 13500. Counting to a smaller number is a basic arithmetic operation. 13500 is less than 14000.
Thanks for trying these out :). Highlights the often subtle difference between knowing the answer and deducing the answer. Feathers could be ground into a pulp and condensed, too. I am not trying to be clever, just seems like the response is a canned answer.
A reasoning model is an LLM that has had additional training phases that reward problem solving abilities. (But in a black box way - it’s not clear if the model is learning actual reasoning or better pattern matching, or memorization, or heuristics… maybe a bit of everything).
The other day I had the opportunity to get a 10% discount on a fridge if I could pay the whole thing in one payment. If I didn't have the money I wouldn't get the discount, so in a way being poor means everything is more expensive.
"The reason that the rich were so rich, Vimes reasoned, was because they managed to spend less money. Take boots, for example. ... A really good pair of leather boots cost fifty dollars. But an affordable pair of boots, which were sort of OK for a season or two and then leaked like hell when the cardboard gave out, cost about ten dollars. ... But the thing was that good boots lasted for years and years. A man who could afford fifty dollars had a pair of boots that'd still be keeping his feet dry in ten years' time, while a poor man who could only afford cheap boots would have spent a hundred dollars on boots in the same time and would still have wet feet.
This was the Captain Samuel Vimes 'Boots' theory of socio-economic unfairness."
I refuse to drop $300 on a pair of bluetooth earphones but i've spent easily over that by buying lots of bad cheap ones. I'm still ok with that on principle, I don't like the idea of being ripped off..
I don't use Youtube that much, but if I did, I'd probably find someone way to stop instead of paying for Youtube.
One time I had cancelled my Netflix subscription because I didn't use it very much, and 3 months later I notice it's still active. I was able to easily contact support in my native language and get a refund.
If I had an issue with any Google product, I'm pretty sure I'd never be able to get it resolved. As a company, they're insanely sketchy. Their main support channel is a community forum where most answers come from people who are endorsed by Google but not "actually" Google employees, except they do get benefits for working for Google, except it's not actually "working" like in a job so Google isn't really responsible for what they say, it's volunteering with benefits, if you know what I mean.
For Youtube specifically what makes me not trust is that, like Spotify, they advertise that premium subscriptions allow you to "download" videos in a sense of the word that literally nobody in the entire Internet would agree with. First "save" isn't saving, now "download" isn't downloading. I'm not liking this trend, to be honest.
It's a common saying that if you aren't the customer, you are the product. But I've heard people say that sometimes even when you are the customer you're still the product. I'm not sure about how I feel about paying to become a shiny product.
>Some in the fediverse ask why I'm not on Peertube. Here's the problem (and it's not insurmountable): right now, there's no easy path towards sustainable content production when the audience for the content is 100x smaller, and the number of patrons/sponsors remains proportionally the same.
Damn, usually I'm not this dumb but thx for the reference I didn't even see it. My take is; YouTube is currently your only option if you want to try to make a living as a video content creator but if you want to casually make content and share it with your audience or your friends and family then you can definitely try PeerTube.
I'm usually against LLM's massive breach of copyright, but this argument is just weird.
>At a conference in January, Wang raised a hypothetical in line with her thinking on the subsequent order. She asked OpenAI's legal team to consider a ChatGPT user who "found some way to get around the pay wall" and "was getting The New York Times content somehow as the output." If that user "then hears about this case and says, 'Oh, whoa, you know I’m going to ask them to delete all of my searches and not retain any of my searches going forward,'" the judge asked, wouldn't that be "directly the problem" that the order would address?
If the user hears about this case, and now this order, wouldn't they just avoid doing that for the duration of the court order?
Idk why but judges typically care more about flexing their powers instead of having decent idea as to how their actions end up setting future precedent or harm to the nation. (Just like how they made donating infinite money to political parties by one individual a fundamental right to free “speech” thereby breaking all existing donation limits and completely breaking apart our electoral process)
The way this judge is pretending like there arent a gazillion ai models that can do this now
One can simply download a app like msty.app combine it with openrouter or ollama based local model and crawl and do all of this from their computer.
This order was absolute insanity and is going to make most corporate users be extremely skeptical with providing their data to american llm providers. This judge single-handedly put 100s of millions of chatgpt users privacy potentially breached and american businesses losing trust as reliable LLM providers not because of their own faults but because of judges like Ms. Wang
This is why America needs a Right to Privacy as a constitutional amendment for american citizens just like how some european and asian countries have.
I'm no AI fan, but articles talking about the shortcomings of LLM's seem to have to be complaining that forks aren't good for drinking soup.
Don't use LLM's to do 2 + 2. Don't use LLM's to ask how many r's are in strawberry.
For the love of God. It's not actual intelligence. This isn't hard. It just randomly spits out text. Use it for what it's good at instead. Text.
Instead of hunting for how to do things in programming using an increasingly terrible search engine, I just ask ChatGPT. For example, this is something I've asked ChatGPT in the past:
in typescript, I have a type called IProperty<T>, how do I create a function argument that receives a tuple of IProperty<T> of various T types and returns a tuple of the T types of the IProperty in order received?
This question that's such an edge case that I wasn't even sure how to word properly actually yielded the answer I was looking for.
function extractValues<T extends readonly IProperty<any>[]>(
props: [...T]
): { [K in keyof T]: T[K] extends IProperty<infer U> ? U : never } {
return props.map(p => p.get()) as any;
}
This doesn't look unrealiable to me. It actually feels pretty useful. I just need [...T] there and infer there.
The thing is, I have spent the last year being told that I will VERY SOON be able to use a fork to drink soup, and better than any spoon has ever been able to, and in fact pretty soon spoons will be completely outclassed anyway, and I'M the idiot for doubting this.
Articles like this are still very much needed, to push back against that narrative, regularly, until it DOES become as obvious to everyone as it is to you.
My impression is that the only people telling others they can drink soup with forks are the people who sell the forks.
Even this isn't new. A few years ago we had people who sold knives telling everybody you could use knives to drink soup. And in some cases they weren't even kitchen knives, they were switchblades.
You use LLMs to _discover_ how to approach important problems. You don't necessarily need to use the output verbatim. Same as StackOverflow and Google.
When you employ your developers at $200K/yr you won't trust them to tell you the first one hundred digits of pi, but you'll trust them with your business logic, which is much more important and mission-critical to you.
The difference is that (hopefully) your employee is honest enough
to say "I do not know the first 100 digits of Pi offhand but I can find out"
An LLM will happily produce a string of 100 digits that might be the first 100 digits of Pi, might be some known sequence of 100 digits in Pi but not the first 100, or might be 100 random digits that have nothing to do with Pi
I was actually curious about this and chatgpt actually accurately and very slowly gave me the first 100 digits of pi one digit at a time. I have _no idea_ how that worked, it did not search, nor did it run code. As far as I can tell, it pulled it straight out of it's own model.
If I ask it to use python, it writes and executes the code _much_ more quickly, same if I ask it to search.
The problem is exactly how the public will learn "not to ask 2+2". When you have a well trained professional using an LLM it's all great. They know how to separate hallucination from actually good results as you do. The problem lies with the general public and new workers who will, no questions about it, use the AI generated results as some sort of truth.
Maybe use an LLM to detect when the public is asking the wrong question and display a message saying "As a large language model, I don't know how to count."
So many times I've asked questions just like this and gotten complete nonsense incorrect answers. In fact, you have no guarantees whatsoever that even the typescript question you asked will always return a sensible answer.
I'm by no means saying that LLMs aren't useful. They're just not reliably useful.
Yep. Google is an ad company that makes most of its money putting ads on its search page, and Gemini is a Google product that lets you get search results from Google without ads. It doesn't make any sense.
I've thought before that AI is as "intelligent" as your smartphone is "smart," but I didn't think "reasoning" would be just another buzzword.