My naïve understanding of deep learning is that it works by finding patterns in the answers, instead of actually solving problems.
If I take a multiple-choice exam and always answer "C", then I have a good chance at getting more than 25%.
For image recognition, I think the classifier is doing the real work (trying to actually answer the question), and the deep learning is just seeing if the answer matches the pattern of expected answers.
Somehow, this actually works. I think that it's because true randomness is hard to find.
The problem that I've found is that it's really difficult to teach deep learning. I'm making a Chinese-English teaching tool ( http://pingtype.github.io ) and sourcing my translations from Google Translate. I find a lot of mistakes in my dictionary that obviously came from Google's model getting the word spacing wrong. I can fix it in my own dictionary immediately. If I submit the correction to Google, it just changes some weightings, and hundreds of people will have to submit the same correction before their deep learning will finally catch on that it needs to change something.
Your naive understanding is supported by at least one deep learning authority:
> I haven’t found a way to properly articulate this yet but somehow everything we do in deep learning is memorization (interpolation, pattern recognition, etc) instead of thinking (extrapolation, induction, etc). I haven’t seen a single compelling example of a neural network that I would say “thinks”, in a very abstract and hard-to-define feeling of what properties that would have and what that would look like.
> All the while I'm thinking: this thinking process this person goes through as he analyzes this data: THAT is what Machine Learning SHOULD do
-- Andrej Karpathy
Deep learning for image recognition works because our visual world is made up of structured hierarchical features: Dark/Light, Texture, Edge, Part of Object, Object, Scene. Deep learning layers create increasingly higher-level features in a computationally feasible way.
I personally prefer 'generic hashing/parsing'; deep learning excels at the automatic creation of a mapping of unstructured information to structured information, after a sufficient period of training.
Hmm... but isn't that what our brains do as well? Unstructured intensities of light bouncing off our retinas which becomes a structured recognized object.
It definitely seems to be part of what our brain does. The visual cortex is an apt comparison since that's where a lot of the structural inspiration for modern ANNs comes from. But, there does seem to be a little more than that too; it's not clear whether all the brain does is reducible to a hash function (reducible in any useful sense, at least; a very very very big, very very very sparse hash function, perhaps).
Our brain can understand that a cartoon-picture of a cat is a cat. Also, our brain can understand that a picture of a cat taken from a hugely different angle than seen before is a cat. Deep learning cannot do those kind of tricks.
There's quite probably some of that. A quote from J.S. Mill on the distinction between science and technology strikes me as useful:
"One of the strongest reasons for drawing the line of separation clearly and broadly between science and art is the following:—That the principle of classification in science most conveniently follows the classification of causes, while arts must necessarily be classified according to the classification of the effects, the production of which is their appropriate end."
Essays on some unsettled Questions of Political Economy
What are your thoughts on newer recurrent architectures like the DNC (or its predecessor, the neural Turing machine)? While the demonstrated results with DNCs so far are pretty limited, it seems that they embody a push towards allowing a neural network to actually "think" over multiple steps: storing complex information, formulating a plan, and acting on that plan.
Yes. I think these architectures are very exciting and a step in the "right" direction. Eventually we will want to move from rote memorization and pattern matching to more challenging aspects of intelligence.
As much as I dislike calling on the neural net / biological net metaphor, I do think that computer science has made some headway in how "useful codes", in the sense of semantically-meaningful interpolation, can be derived from natural scene stimuli, and therefore the onus that "we do something different" is to some extent now on the neuroscientists to think about and try to prove that "reasoning" in the human sense is anything other than an algebra of latent codes, i.e., linear or non-linear combinations of codified summaries of sensory input.
Geoff Hinton refers to thought vectors performing reasoning by analogy using algebra [1] in his Royal Society Lecture.
The other widely reported vector algebras in a semantic space were discovered by Mikolov et al when producing ~300 dimensional vectors for a billion word Wikipedia corpus.
If one performs vector algebra and ~= is near by cosine distance then using Mikolov's Vectors[3].
King - Man + Woman ~= Queen
France - Paris + Gernmany ~= Berlin
Surprisingly this works for other modalities, Chintala, Radford & Metz found a latent semantic space in images, that adds vectors for glasses or smiles to peoples faces. [4] With a generative model new images can be created as outlined in this blog post by Soumith [5]
Karpathy shows trained nets can be assembled like lego across modalities, slice off the classifier to reveal the rich semantic 'thought vector' layer of an Imagenet trained Alexnet, plug in a RNN sentence generator using word2vec and ( some over simplification ... ) you get a convincing image captioner [6].
The thought vectors are akin to high level representations of the world and can cross modalities . Text to Images using thought Vectors ( from hnnews discussion [7] )
So the vectors of though are in some way a an AI mentalese or encoding of a symbolic representation of the world derived from the data and can ( again drastic over simplification ) transfer modalities and even between previously unlinked languages [8]
[2] The paper Geoff Hinton is reffering to : Sequence to Sequence Learning with Neural Networks by Ilya Sutskever, Oriol Vinyals, Quoc V. Le https://arxiv.org/abs/1409.3215
[3] Efficient Estimation of Word Representations in Vector Space by Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean
https://arxiv.org/abs/1301.3781
[4] Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Alec Radford, Luke Metz, Soumith Chintala https://arxiv.org/abs/1511.06434
Don't just down vote this guys, that doesn't help.
No, your naive understanding is not correct.
'Deep learning', and by that I mean a neural network, works at a super high level by generalising some input (say X) into some related output (lets say, Y).
It's not just random choice; it's like defining a programming function:
foo(x) -> y { ... }
Where the '...' is implemented by a set of statistical weights and training data and so on.
...but ultimately, you raise a valid point.
Training models is incredibly time consuming, and incorporating 'corrections' as new training data is extremely non trivial.
It's one of the issues with deep learning.
I would even go as far as to say, 'You need to regularly update your training data with new examples and counter examples' as a 'when not to use deep learning'.
> I can fix it in my own dictionary immediately. If I submit the correction to Google, it just changes some weightings, and hundreds of people will have to submit the same correction before their deep learning will finally catch on that it needs to change something.
This isn't a problem with deep learning, but a problem with verifying a data source. You are trusting yourself to submit a proper correction, but Google doesn't know you from John.
To use another example, imagine that someone says 他的 is "they're", another person says it's "their", and still another person says it's "there". Your approach would just accept all three in a row, wouldn't it?
Google's approach presumably trusts that the crowd + what it sees on the web is correct and thus attempts to verify that a submission is typical language use before actually putting it into play.
I just tried your tool, and I must say it generated some impressive nonsense.
Putting in 这是什么东西 ("What is this thing") in the Chinese box yielded the translation "this is Why tender east west", which is a _character for character_ translation of the phrase. That's not especially meaningful or useful given that Chinese words are polysyllabic - I hope this is a temporary bug.
By contrast, Google Translate gives me the much more sensible "What is this" as a translation.
> My naïve understanding of deep learning is that it works by finding patterns in the answers, instead of actually solving problems.
I think this is quite profound and inspiring.
Although perhaps it is only one half of it. Concretely, deep learning finds patterns, the best patterns are derived from the highest bandwidth signal, often this is the input.
Geoff Hinton has argued that the task, solving the problem, is a low bandwidth signal.
Hinton aphorises [approximately] 'If you want to learn computer vision first learn to do computer graphics, i.e. a generative model.' - this is about the bandwidth of the data signal.
Hinton: [1] "Each image has much more information in it than a typical label... Each image puts a lot of constraint on the identity function. Whereas if I give you an image and a label and I try and get the right answer I don't get much constraint on the mapping from image to label. The bits of constraint on that mapping imposed by training example are just the number of bits to say what the answer is which is not very many."
Your brain also works finding patterns. The only difference is that your brain works for a lot of domains and can translate information of one domain to other. We have not find a good way to do this in deep learning yet. But once we find it, it will know that answer C all the time probably is not correct.
Humans seem to solve problems with a combination of learned stock knowledge, induction, and constrained improvisation.
Constrained improvisation is the most interesting part of that process, and the one we know least about.
It's one thing building a system that asymptotically improves over millions of trials, and then sending out a press release claiming your system is as smart as a human.
It's another building a system that learns a domain as efficiently as a human.
Compare the relatively small number of games played/analysed by a Go master on their way to master status, compared with the number of simulated games played/analysed by AlphaGo.
ML is still a rather naive form of constrained brute forcing. It's a long way short of efficient learning.
> ML is still a rather naive form of constrained brute forcing. It's a long way short of efficient learning.
Uh... Bayesian can do the above at in term of expert domain. They call it elicitation in the Bayesian world.
I think you're overall generalizing all learning techniques. And also it's not like we actually really know how the human brain learn. Psychology is a field with huge uncertainties and you can see that in their research papers with correlation values. So the concept of learning may be out dated and/or we are still learning about what makes us learn.
You're describing a technique for finding patterns. There are many possible techniques and machines can (probably) use any of them. The constraint would be if the technique involved kinetic or quantum mechanics.
I have the same question about Google translate. I'm not convinced that its errors can simply be chalked up to neural networks. As you said, they ought to be constantly trying to improve their word segmentation and datasets.
I'm not convinced that Google Translate is as major of a focus at Google as other things.
Perhaps I'm wrong, and it's just a hard problem, but the translations I've seen haven't improved as much over the years as I expected given progress in other areas of AI.
They may be more focused on adding new languages than improving existing ones. Don't know.
Japanese translation is pretty good. Chinese is bad. I guess Chinese is harder because nobody really uses any phonetic writing for reading beyond grade schools. And then there is traditional/simplified, and, I imagine, other differences between how different regions use characters. Baidu's translate is better than Facebook/Google in some cases.
I wonder if translation just isn't as sexy as image recognition and self driving cars, therefore research dollars aren't as focused on it.
Machine translation has a lot of research funding, and has had it for decades.
I'm not sure about USA, but it's been a major focus of very large EU grant programs due to the obvious multilinguality of EU and the explicit goal to move towards a single European market by reducing barriers in trade, including language barrier.
It also has many commercial use cases and thus has always had quite a lot of people and teams working on it compared to other fields of ML or NLP.
The problem is that it's hard. Every 0.1% of progress has historically required a lot of work.
Now that the general public understands a bit more about NN's , AI and machine learning, maybe there will be a renewed push to invest in even better machine translation.
More research into theory, more / better data from professional translators, and more efficient implementations from engineers.
If I take a multiple-choice exam and always answer "C", then I have a good chance at getting more than 25%.
For image recognition, I think the classifier is doing the real work (trying to actually answer the question), and the deep learning is just seeing if the answer matches the pattern of expected answers.
Somehow, this actually works. I think that it's because true randomness is hard to find.
The problem that I've found is that it's really difficult to teach deep learning. I'm making a Chinese-English teaching tool ( http://pingtype.github.io ) and sourcing my translations from Google Translate. I find a lot of mistakes in my dictionary that obviously came from Google's model getting the word spacing wrong. I can fix it in my own dictionary immediately. If I submit the correction to Google, it just changes some weightings, and hundreds of people will have to submit the same correction before their deep learning will finally catch on that it needs to change something.