We can expect the model landscape to consolidate some day. Progress will become slower, innovations will become smaller. Not tomorrow, not next year, but the time will come.
And then it'll increasingly make sense to build such a chip into laptops, smartphones, wearables. Not for high-end tasks, but to drive the everyday bread-and-butter tasks.
The world continues to evolve, in a way that requires flexibility - not more constraints. I just fail to see a future where we want less general purpose computers, and more hard-wired ones? Would be interesting to be proven wrong though!
TPU usb-c dongle is less than $100 (widely used for detecting people in home assistant / frigate nvr camera feeds). If one-off $100 purchase can replace (and improve 10x by speed) anthropic subscription even for 12 months - I don't see why not.
Neither the blog nor Taalas' original post specify what speed to expect when using the SRAM in conjunction with the baked-in weights? To be taken seriously, that is really necessary to explain in detail, than a passing mention.
No, it did not mostly refer to to-do lists. At least that's not what I understood.
It was an example, yes. But he also referred to Luhmann’s Zettelkasten, a huge collection of notes, systematically interconnected, see https://en.wikipedia.org/wiki/Niklas_Luhmann#Note-taking_sys.... Looks like the author followed a similar approach in Obsidian, a tool to store markdown notes in an interconnected manner.
He wrote: "Roam Research turned bidirectional links into a cult. Obsidian let the cult go off-grid. The lore deepened. You weren’t taking notes. You were building a lattice of meaning. ... A quote would spark an insight, I’d clip it, tag it, link it - and move on."
But it didn't work out: "the insight was never lived. It was stored. ... In trying to remember everything, I outsourced the act of reflection. I didn’t revisit ideas. I didn’t interrogate them. I filed them away and trusted the structure. ... The more my system grew, the more I deferred the work of thought to some future self who would sort, tag, distill, and extract the gold.
That self never arrived."
Hence, what he mostly wrote down were thoughts, ideas, quotes, hoping that insights and value will come from this huge collection. Turns out, it wasn't so easy for him. I believe he'll need more focus, more curation, a more targeted and heartfelt approach.
Well, it sounds like it did work out back then for Niklas Luhmann. He said that all his famous publications wouldn't have been possible without his structured note taking approach. His 90,000 cards were digitized and are now available online, see https://niklas-luhmann-archiv.de/bestand/zettelkasten/zettel...
That's one thing where I love Golang. I just tell Aider to `/run go doc github.com/some/package`, and it includes the full signatures in the chat history.
It's true: often enough AI struggles to use libraries, and doesn't remember the usage correctly. Simply adding the go doc fixed that often.
The author would be excited to learn that CouchDB solves this problem since 20 years.
The use case the article describes is exactly the idea behind CouchDB: a database that is at the same time the server, and that's made to be synced with the client.
You can even put your frontend code into it and it will happily serve it (aka CouchApp).
Even if I don’t care much about the „digital garden“ concept, I was much surprised to visit a page called „blogscroll“, click on a link, and then not find a blog.
> When we talk about understanding a simple axiomatic system, understanding means exactly that the entirety of the axioms are modeled and applied correctly 100% of the time.
Yes, but then, when we talk about understanding in LLMs, we talk about existence, but not necessarily about determinism.
Remember that, while chess engines are (I guess?) deterministic systems, LLMs are randomized systems. You give the same context and the same prompt multiple times, and each and every time you get a different response.
To me, this, together with the fact that you have an at least 1-in-10 chance of getting a good move (even for strange scenarios), means that understanding _does exist_ inside the LLM. And the problem following from this is, how to force the LLM to reliably choose the right „paths of thought“ (sorry for that metaphor).
For me that discussion is always hard to grasp. When a human would learn coding autodidacticly by reading source code, and later they would write new code — then they could only do so because they read licensed code. No one would ask for the license, right?
Because humans aren't computers and the similarities between the two, other than the overuse of the word "learning" in the computer's case, are nonexistant?
Don't know if they are, and don't really care either and I don't care to anthropomorphize circuitry to the extent that AI proponents tend to, especially.
Humans and Computers are 2 wholly separate entities, and there's 0 reason for us to conflate the two. I don't care if another human looks at my code and straight up copies/pastes it, I care very much if an entity backed by a megacorp like Micro$oft does the same, en-masse, and sells it for profit, however.
Okay, so the scale at which they sale their service is a good argument that this is different from a human learning.
However, on the other hand we also have the scale at which they learn, which kind of makes every individual source line of code they learn from pretty unimportant. Learning at this scale is statistical process, and in most cases individual source snippets diminish in the aggregation of millions of others.
Or to put it the other way round, the actual value lies in the effort of collecting the samples, training the models, creating the software required for the whole process, putting everything into a good product and selling it. Again, in my mind, the importance of every individual source repo is too small at this scale to care about their license.
The idea that individual source snippets at this scale diminish in aggregation, is undercut by the fact that OpenAI and MSFT are both selling enterprise-flavoured versions of GPT, and the one thing they promise is that enterprise data will not be used to further train GPT.
That is a fear for companies because the individual source snippets and the knowledge "learned" from them is seen as a competitive advantage of which the sources are an integral part - and I think this is a fair point from their side. However then the exact same argument should apply in favour of paying the artists, writers, coders etc whose work has been used to train these models.
So it sounds like they are trying to have their cake and eat it too.
Hmm. You sure this is the same thing? I would say it’s more about confidentiality than about value.
Because what companies want to hide are usually secrets, that are available to (nearly) no one outside of the company. It’s about preventing accidental disclosure.
What AIs are trained on, on the other hand, is publicly available data.
To be clear: what could leak accidentally would have value of course. But here it’s about the single important fact that gets public although it shouldn’t, vs. the billions of pieces from which the trained AI emerges.
It's really not different in scale. Imagine for a moment how much storage space it would take to store the sensory data that any two year old has experienced. That would absolutely dwarf the text-based world the largest of LLMs have experienced.
But that also exists in the AI world. It’s called „fine tuning“: a LLM trained on a big general dataset can learn special knowledge with little effort.
I’d guess it’s exactly the same with humans: a human that received good general education can quickly learn specific things like C.
Humans have experienced an amount of data that absolutely dwarfs the amount of data even the largest of LLMs have seen. And they've got billions of years of evolution to build on to boot
The process of evolution "from scratch", i.e. from single-celled organisms took billions of years.
This is all relevant because humans aren't born as random chemical soup. We come with pre-trained weights from billions of years of evolution, and fine-tune that with enormous amounts of sensory data for years. Only after that incredibly complex and time-consuming process does a person have the ability to learn from a few examples.
An LLM can generalize from a few examples on a new language that you invent yourself and isn't in the training set. Go ahead and try it.
There is the element of the unknown with LLMs etc.
There is a legal difference between learning from something and truly making your own version and simply copying.
It's vague of course - take plagiarism in a university science essay - the student has no original data and very likely no original thought - but still there is a difference between simply copying a textbook and writing it in your own words.
Bottom line - how do we know the output of the LLM isn't a verbatim copy of something with the license stripped off?
The way I see it is that with AI you have really painted your own Caravaggio, but instead of an electrochemical circuit of a human brain you've employed a virtual network.
> but instead of an electrochemical circuit of a human brain you've employed a virtual network.
technically it is still a tool you are using, differently from doing it on your own, with your hands, using your own brain cells, that you trained over the decades, instead of using a virtual electronic brain pre-trained in hours/days by someone else on who knows what.
Okay if it’s about looking at one painting and fake that. However, if you train your model on billions of paintings and create arbitrary new ones from that, it’s just a statistical analysis on what paintings in general are made of.
The importance of the individual painting diminishes at this scale.
I'd add to this, the damage an LLM could do is much less than a human could do in terms of individual production. A person can paint so many forgeries... A machine can create many, many more. The dilusion of value from a person learning is far different than machine learning. The value extracted and diluted is night and day in terms of scale.
Not to say what will/won't happen. In practice, what I've seen doesn't scare me much in terms of what LLMs produce vs. what a person has to clean up after it's produced.
They are not the same because an LLM is a construct. It is not a living entity with agency, motive, and all the things the law was intended for.
We will see new law as this tech develops.
For an analogy, many people call infringement theft and they are wrong to do so.
They will focus on the someone getting something without having followed the right process part while ignoring the equally important someone else being denied the use of, or loss of property part.
The former is an element in common between theft and infringement. And it is compelling!
But, the real meat in theft is all about people losing property! And that is not common at all.
This AI thing is similar. The common elements are super compelling.
But it just won't be about that in the end. It will be all about the details unique to AI code.
Using the word "construct" isn't adding anything to the conversation. If we bioengineer a sentient human, would you feel OK torturing it because it's "just a construct"? If that's unethical to you, how about half meat and half silicon? How much silicon is too much silicon and makes torture OK?
> Most people will [privilege meat]
"A person is smart. People are dumb, panicky dangerous animals, and you know it". I agree that humans are likely to pass bad laws, because we are mostly just dumb, panicky dangerous animals in the end. That's different than asking an internet commentor why they're being so confident in their opinions though.
Full stop. We've not done that yet. When we do, we can revisit the law / discussion.
We can remedy "construct" this way:
Your engineered human would be a being. Being a being is one primary difference between us and these LLM things we are toying with right now.
And yes, beings are absolutely going to value themselves over non beings. It makes perfect sense to do so.
These LLM entities are not beings. That's fundamental. And it's why an extremely large number of other beings are going to find your comment laughable. I did!
You are attempting to simplify things too much to be meaningful.
Define "being". If it's so fundamental, it should be pretty easy, no?
And I'd like if this were simple. Unfortunately there's too many people throwing around over-simplifications like "They are not the same because an LLM is a construct" or "These LLM entities are not beings". If you'll excuse the comparison, it's like arguing with theists that can't reason about their ideological foundations, but can provide specious soundbites in spades.
A being is a living thing with a will to survive, need for food, and a corporeal existence, in other words, is born, lives for a time, then dies.
Secondly, beings are unique. Each one has a state that ends when they do and begins when they do. So far, we are unable to copy this state. Maybe we will one day, but that day, should there ever be one, is far away. We will live our lives never seeing this come to pass.
Finally, beings have agency. They do not require prompting.
Also twice now you've said the equivalent of "it hasn't happened yet so no need to think about the implications". Respectfully, I think you need to ponder your arguments a bit more carefully. Cheers.
They've got a few fantastic attributes, lots of different beings do. You know the little water bear things are tough as nails! You can freeze them for for a century wake them up and they'll crawl around like nothing happened.
Naked mole rats don't get any form of cancer. All kinds of things the beans present in the world that doesn't affect the definition at all.
You didn't gain any ground with that.
And I will point out, it is you who has the burden in this whole conversation. I am clearly in the majority if you want things with what I've said. And I will absolutely privilege meets face over silicon any day, for the reasons I've given.
You, on the other hand, have a hell of a sales job ahead of you. Good luck maybe this little exchange helped a bit take care
> Or do they magically become "beings" when they die?
quoting from your link
although in practice individuals can still die. In nature, most Turritopsis dohrnii are likely to succumb to predation or disease in the medusa stage without reverting to the polyp form
This sentence does not apply to an LLM.
Also, you can copy an LLM state and training data and you will have an equivalent LLM, you can't copy the state of a living being.
Mostly because a big chunk of the state is experience, like for example you take that jellyfish, cut one of its tentacles and it will be scarred for life (immortal or not). That can't be copied and most likely never will.
Regarding the copying of a being state, I'm not really sure that's ever even going to be possible.
So for the sake of argument I'll just amend that and say we can't copy their state. Each being is unique and that's it. They aren't something we copy.
And yes that means all of us that thinks somehow they're going to get downloaded into a computer? I'll say it right here and now that's not going to fucking happen.
Sounds like the perspective of someone who never gave that tool a real chance. To me it’s primarily just another aid like IDE inspections, IDE autocompletions etc.
I use GitHub Autopilot mainly for smaller chunks of code, another kind of autocompletion, which basically safes me from typing, Googling or looking into the docs, and therefore keeps me in the flow.
And then it'll increasingly make sense to build such a chip into laptops, smartphones, wearables. Not for high-end tasks, but to drive the everyday bread-and-butter tasks.