Summary of the NatureDeepReview paper

by Yann LeCun1,2, Yoshua Bengio3 & Geoffrey Hinton4,5

Reading Machine Learning research papers is a great way to keep up with recent developments in the field. It also helps you understand the underlying fundamentals behind any model. When working on a problem it is great practice to first check for any existing literature in order to avoid reinventing the wheel. Here is a quick summary of the NatureDeepReview paper.

Images are normally represented in the form of an array of pixel values.

Learned features in the first layer typically represent presence or absence of edges. The second layer detects motifs. The third layer may assemble motifs into larger combinations and the subsequent layers detect objects as a combination of these parts.

Suppose a problem of classifying Samoyeds from wolves. Features such as background, pose, lighting and surrounding objects are insensitive. At the pixel level, images of two Samoyeds in different poses and in different environments may be very different from each other, whereas two images of a Samoyed and a wolf in the same position and on similar backgrounds may be very similar to each other. A linear classifier, or any other ‘shallow’ classifier operating on raw pixels could not possibly distinguish the latter two hence shallow classifiers require a good feature extractor that solves the selectivity–invariance dilemma. The key advantage of deep learning over linear classifies is that it learns the good features automatically.

For smaller datasets, unsupervised pretraining helps to prevent overfitting.

Convolutional Neural Networks are designed to process data that come in the form of multiple arrays. The 4 key ideas behind ConvNets that take advantage of the properties of natural signals are local connections, shared weights, pooling and use of many layers. The architecture of a typical ConvNet is structured as a series of stages. The first few stages are composed of two types of layers. Convolutional layers and pooling layers. The role of the convolutional layer is to detect local conjunctions of features from the previous layer. The role of the pooling layer is to merge semantically similar features into one. There has been numerous applications of ConvNets going back to 1980s, starting with time-delay neural networks for speech recognition and document reading, Optical Character Recognition and handwriting recognition systems.