Probably the same way everyone does, by pulling it out of thin air as a guess. When nobody even knows what theoretical breakthroughs are necessary, you'll always end up with a scattershot all over the place, even amongst experts. Try asking working mathematicians how long until the Riemann hypothesis is resolved one way or another, or look at what people were saying about Fermat's Last Theorem up until it was solved.
What we do know is that current techniques won't get us close to AGI, so something new is needed (or perhaps like backprop, something old will work once we have enough compute power). Personally I'm bullish on AGI because I have strikingly low faith in the ability of evolution to operate very effectively as a tool for algorithm discovery, so I suspect that once we've hit the compute threshold we'll find that many different algorithms can do the trick, and 40 years is probably not out of the question for us to hit that point (or 10, or 100), depending who you talk to about what the compute threshold might be.
I'd caution against putting too much weight in what experts say, though, since with a tiny few set of exceptions anyone working on "AI" today is actually just working on narrow AI, which is, as someone put it, just glorified linear regression. Those tools will almost certainly be part of the solution, but only in the sense that the classical theory of Diophantine equations was part of Weil's proof of Fermat's Last Theorem - they are not the core of the theoretical approach.
Evolution is a slow algorithm, but it had access to an absurd amount of compute (all neuronal organic matter on Earth) and environment simulation (all of physical reality on Earth) when discovering us; so the discovery of the algorithms/architectures/principles in our heads shouldn't be viewed as trivial.
The massive compute/time advantage evolution has makes me bearish about AGI. We really need to fix our compute capabilities before we can start overruning evolution. The math dictates it'll happen, but exponentially slowly if we don't innovate in compute.
There's more to the story, too: advances on top of CRISP may give us better tools to self-improve the species, accelerating evolution.
Personally, I'm bearish about AGI because I believe we will eventually realize that the brain is a glorified linear regression too, with a custom wiring to help learn language and vision.
>What we do know is that current techniques won't get us close to AGI, so something new is needed (or perhaps like backprop, something old will work once we have enough compute power).
With backprop we didn't just need bigger machines, we needed better algorithms, palliatives for the exploding-gradient problem that made values exceed our numerical representations, and then hardware specifically designed for doing the matrix-ops involved.
If I saw something capable of speeding up probabilistic program inference the way GPUs sped up backprop, I'd start saying we should expect to see powerful AI applications quite soon.
Better algorithms were invented because of bigger machines. Once computers got fast enough, researchers could experiment around with different algorithms on realistic sized models and datasets. Without waiting 2 years for the experiment to finish training.
Probabilistic programming isn't going to help general AI much. Things like dropout seem to work well enough, and for the most part AI is severely underfitting rather than overfitting. Our models are far to simple and small to really learn language and do complicated reasoning. Making them bayesian doesn't fix that.
>Probabilistic programming isn't going to help general AI much.
Excuse me while I laugh.[1,2,3,4]
>Things like dropout seem to work well enough, and for the most part AI is severely underfitting rather than overfitting.
For the most part, neural networks can't reason at all. They just induce deterministic functions over high-dimensional Euclidean spaces.
>Our models are far to simple and small to really learn language and do complicated reasoning.
They're also not compositional (new concepts as functions of old concepts), productive (able to draw an unbounded number of inferences from each representation), or unbounded in size of representation (unboundedly many concepts). Neural networks don't even represent causal structure, let alone model how an intervention will affect outcomes!
It is, however, really nice to hear an AI booster admit just how incredibly limited connectionist models actually are.
>Making them bayesian doesn't fix that.
No, changing to a causal, compositional representation that allows for productive and nonparametric (unboundedly large) learning does that. The Bayesian part just makes it extra nice by letting us "put information in" anywhere in the model (at any variable) by conditioning.
What we do know is that current techniques won't get us close to AGI, so something new is needed (or perhaps like backprop, something old will work once we have enough compute power). Personally I'm bullish on AGI because I have strikingly low faith in the ability of evolution to operate very effectively as a tool for algorithm discovery, so I suspect that once we've hit the compute threshold we'll find that many different algorithms can do the trick, and 40 years is probably not out of the question for us to hit that point (or 10, or 100), depending who you talk to about what the compute threshold might be.
I'd caution against putting too much weight in what experts say, though, since with a tiny few set of exceptions anyone working on "AI" today is actually just working on narrow AI, which is, as someone put it, just glorified linear regression. Those tools will almost certainly be part of the solution, but only in the sense that the classical theory of Diophantine equations was part of Weil's proof of Fermat's Last Theorem - they are not the core of the theoretical approach.