Showing posts with label Deep Learning. Show all posts
Showing posts with label Deep Learning. Show all posts

Wednesday, January 14, 2015

Deep Learning Reference

Introduction

Deep learning is a set of algorithms in machine learning that attempt to model high-level abstractions in data by using architectures composed of multiple non-linear transformations.

Deep learning is part of a broader family of machine learning methods based on learning representations. An observation (e.g., an image) can be represented in many ways (e.g., a vector of pixels), but some representations make it easier to learn tasks of interest (e.g., is this the image of a human face?) from examples, and research in this area attempts to define what makes better representations and how to create models to learn these representations.

Various deep learning architectures such as Deep Neural Networks, Convolutional Deep Neural Networks, and Deep Belief Networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, and music/audio signal recognition where they have been shown to produce state-of-the-art results on various tasks.

Literature

Deep Machine Learning – A New Frontier in Artificial Intelligence Research – Itamar Arel, Derek C. Rose, and Thomas P. Karnowski.
Info: Briefly introduce some popular deep learning algorithms, such as Convolutional Neural Networks and Deep Belief Networks ...

Gradient-Based Learning Applied to Document Recognition - Yann Lecun , Léon Bottou , Yoshua Bengio , Patrick Haffner
Info: Present traditional Convolutional Neural Networks ...

Info: Discuss the derivation and implementation of Convolutional Neural Networks, followed by a few straightforward extensions ...

Flexible, High Performance Convolutional Neural Networks for Image Classification - Dan C. Ciresan, Ueli Meier, Jonathan Masci, Luca Maria Gambardella, Jürgen Schmidhuber
Info: Present a fast, fully parameterizable GPU implementation of Convolutional Neural Network variants ...

Tiled Convolutional Neural Networks - Quoc V. Le, Jiquan Ngiam, Zhenghao Chen, Daniel Chia, Pang Wei Koh, Andrew Y. Ng
Info: Present Tiled Convolutional Neural Networks ...

ImageNet Classification with Deep Convolutional Neural Networks - Krizhevsky, A., Sutskever, I. and Hinton, G. E.
Info: Provide some improvements on traditional Convolutional Neural Networks for large image database training ...

Improving Neural Networks by Preventing Co-adaptation of Feature Detectors - G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever and R. R. Salakhutdinov
Info: Describe Dropout for deep learning ...

Maxout Networks - Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, Yoshua Bengio
Info: Describe Maxout for deep learning ...

Regularization of Neural Network using DropConnect - Li Wan, Matthew Zeiler, Sixin Zhang, Yann LeCun, Rob Fergus
Info: Describe DropConnect for deep learning ...

Convolutional Networks and Applications in Vision - Yann Lecun , Koray Kavukcuoglu , Clément Farabet
Info: Describe unsupervised Convolutional Neural Networks ...

Document

Deep Learning Document
Theano Document
Pylearn2 Document

Developing Environment

1. Anaconda (Python + Numpy + Scipy + Matplotlib + ...)

    Download: http://continuum.io/downloads

2. Theano

    Download: https://github.com/Theano/Theano

3. Pylearn2

    Download: https://github.com/lisa-lab/pylearn2


Architecture

1. Convolutional Neural Networks

2. Deep Belief Networks

Public Database

1. The CIFAR-10 dataset

    Download: http://www.cs.toronto.edu/~kriz/cifar.html

2. The CIFAR-100 dataset

    Download: http://www.cs.toronto.edu/~kriz/cifar.html

3. NORB Object Recognition Dataset

    Download: http://www.cs.nyu.edu/~ylclab/data/norb-v1.0/

Understanding of Convolutional Neural Networks

Convolutional Neural Networks (CNNs) is one of the popular deep learning algorithms. CNNs were proposed as a deep learning framework that is motivated by minimal data preprocessing requirements. It can also be regarded as a feature extraction process.

CNNs is a family of multi-layer neural networks. It is comprised of one or more convolutional layers (often with a subsampling layer) and then followed by one or more fully connected layers as in standard multilayer neural network.

The architecture of a CNNs is designed to take advantage of the 2D structure of  an input image. This is achieved with local connections and tied weights followed by some form of pooling which results in translation invariant features. Another benefit of CNNs is that they are easier to train and have many fewer parameters than fully connected networks with the same number of hidden units because of weight sharing strategy.

Given an input RGB image (M x N), the first convolutional layer has k filters (or weight matrix) of size m x n x q. Here, q equals 3 (the number of input maps: RGB); k is the number of output maps. Actually, these filters can be represented as a 4D tensor with size k x q x m x n. The size of output map should be (M-m+1) x (N-n+1) because of convolution. If the next layer is subsampling layer, each map is then subsampled typically with mean or max pooling over p x p contiguous regions. Either before or after the subsampling layer, an additive bias and sigmoid (or tanh) nonlinearity is applied to each feature map. The outputs of hyperbolic tangent function will typically be near zero, the outputs of a sigmoid will be non-zero on average. However, normalizing your training data to have mean 0 and variance 1 along the features can often improve convergence during gradient descent.