I asked ChatGPT to estimate memory requirements of a brain simulation, with neurons from a paper linked in OP post ("Single cortical neurons as deep artificial neural networks"). This paper uses neuron models that are supposed to simulate real ones. The answer was 8-50 PB, with 10^7 weights shared between neurons, or 3.4 EB if each neuron has its own 10^7 weights. So it's not out of the realm of possibility.
On the other hand it shows how utterly tiny and primitive state-of-the-art LLMs are compared to real-world brains. By the way our brain only runs on on a max clock speed of around 100-1000 Hz.
Yet LLMs can achive miraculous results with so little. It really begs the question, where is the limit? How smart would a artificial human brain be if it would clock in gigahertz and had the somehow orders of magnitudes more efficient reasoning and detail recall of LLMs?
The greater the clock speed, the greater the energy consumption, but I guess compute would scale linearly with frequency.
However LLMs are not really tiny compared to real brains, brains FLOP/s have been estimated to be around 1e15 to 1e21, while a single H100 SXM can already do 1e15 BF16 FLOP/s.
The brain also has about 1e15 synapses, each holding the equivalent of about 4.7 bits of information, which would be about 590 TB worth of parameters, equivalent to 294T BF16 parameters (would need 7.4k H100s to hold those, at 80GB per H100).
So in my opinion, it's more of a memory bandwidth problem than a computing power problem, as H100s have comparatively very little memory bandwidth per FLOP than the brain, which holds information directly inside the computing units (synapses).
And I also think that detail recall is a double-edged sword. Memorizing details can certainly be useful, but it can also compete with pattern generalization. If you can perfectly remember the training set (overfit) you don't need to learn to generalize.
I completely agree. I'm betting on physical learning systems via reservoir computing personally. Will be going back to school to study nanotechnology soon to pursue this formally in addition to some experiments Ill be funding and running on my own dime as well. Have you ever seen steel ball bearings submerged in castor oil? If you apply an electrical current to that what do you think will happen? The most fascinating thing occurs, the ball bearings self organize into a network that can even display properties of hebbian learning if set up correctly! I suspect experiments such as these will eventually even raise the bigger question of, what is life actually? Another way to think about the potential of this area of technology is to think of a biological synapse, its about 1 micron in size right? Well, how many "neural atoms" can we fit in that space? 1000x1000x1000!! Its marvelous how much space there is when you shrink things down. Ill even go so far as to speculate that to create genuine superintelligence, youll need this technology Ill be working on to mature and form into some.new form of stochastic processor/computer
To be honest, I haven't looked at any statistics but I imagine a tiny few of those looking for gold found any and got rich, the most either didn't find anything, died of illness or exposure or got robbed. I just like the quote as a comparison. Updated the original comment to reflect I haven't checked if it's correct.
