Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I'm just now getting into neural networks and machine learning. I notice that there's a lot of research going into RNNs, CNNs, and deep learning.

So far, I've only really delved into NNs with one hidden layer, and I read that problems really only require one hidden layer. For what kinds of problems are multiple layers required, or CNNs, or RNNs? It seems that there's a lot of "cool factor" there, but I don't know what they bring to the table that's new.

Where I read about one-layer being "good enough" for most problems: http://stats.stackexchange.com/questions/181/how-to-choose-t...



Usually you need multiple layers when you train NN directly from raw features (like pixels or words) for perception-like problems. In that case you want to learn complicated multi-stage transformations to get useful high-level features.

If you work with already extracted features (like a classical ML pipeline where you can use SVMs or logistic regression as well), you may go with one-layer network. But deep learning really shines with perception-like problem.

Don't be fooled by the approximation theorem (you can approximate any function with only one hidden layer). It is beautiful in theory, but in practice it is like saying that you can write any program in brainfuck because it is turing-complete.


I found an analogy with electronic circuits useful. You can implement any bitwise function using a single layer, a lookup table. The problem with that is it requires as many logic gates as there are inputs, exponentially many. As a result, it's more efficient to design deeper circuits that compose multiple logic gates. The tradeoff is that it is slower to evaluate, since the signal has to pass through more logic gates.


That answer is from 2010, something to keep in mind. The field is evolving so rapidly you should try to find something more recent. And preferably from an expert in the field.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: