Deep Learning Reference

Introduction

Deep learning is a set of algorithms in machine learning that attempt to model high-level abstractions in data by using architectures composed of multiple non-linear transformations.

Deep learning is part of a broader family of machine learning methods based on learning representations. An observation (e.g., an image) can be represented in many ways (e.g., a vector of pixels), but some representations make it easier to learn tasks of interest (e.g., is this the image of a human face?) from examples, and research in this area attempts to define what makes better representations and how to create models to learn these representations.

Various deep learning architectures such as Deep Neural Networks, Convolutional Deep Neural Networks, and Deep Belief Networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, and music/audio signal recognition where they have been shown to produce state-of-the-art results on various tasks.

Literature

Deep Machine Learning – A New Frontier in Artificial Intelligence Research – Itamar Arel, Derek C. Rose, and Thomas P. Karnowski.

Info: Briefly introduce some popular deep learning algorithms, such as Convolutional Neural Networks and Deep Belief Networks ...

Gradient-Based Learning Applied to Document Recognition - Yann Lecun , Léon Bottou , Yoshua Bengio , Patrick Haffner

Info: Present traditional Convolutional Neural Networks ...

Notes on Convolutional Neural Networks - Jake Bouvrie

Info: Discuss the derivation and implementation of Convolutional Neural Networks, followed by a few straightforward extensions ...

Flexible, High Performance Convolutional Neural Networks for Image Classification - Dan C. Ciresan, Ueli Meier, Jonathan Masci, Luca Maria Gambardella, Jürgen Schmidhuber

Info: Present a fast, fully parameterizable GPU implementation of Convolutional Neural Network variants ...

Tiled Convolutional Neural Networks - Quoc V. Le, Jiquan Ngiam, Zhenghao Chen, Daniel Chia, Pang Wei Koh, Andrew Y. Ng

Info: Present Tiled Convolutional Neural Networks ...

ImageNet Classification with Deep Convolutional Neural Networks - Krizhevsky, A., Sutskever, I. and Hinton, G. E.

Info: Provide some improvements on traditional Convolutional Neural Networks for large image database training ...

Improving Neural Networks by Preventing Co-adaptation of Feature Detectors - G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever and R. R. Salakhutdinov

Info: Describe Dropout for deep learning ...

Maxout Networks - Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, Yoshua Bengio

Info: Describe Maxout for deep learning ...

Regularization of Neural Network using DropConnect - Li Wan, Matthew Zeiler, Sixin Zhang, Yann LeCun, Rob Fergus

Info: Describe DropConnect for deep learning ...

Convolutional Networks and Applications in Vision - Yann Lecun , Koray Kavukcuoglu , Clément Farabet

Info: Describe unsupervised Convolutional Neural Networks ...

Document

Deep Learning Document

Theano Document

Pylearn2 Document

Developing Environment

1. Anaconda (Python + Numpy + Scipy + Matplotlib + ...)

    Download: http://continuum.io/downloads

2. Theano

    Download: https://github.com/Theano/Theano

3. Pylearn2

    Download: https://github.com/lisa-lab/pylearn2

Architecture

1. Convolutional Neural Networks

2. Deep Belief Networks

Public Database

1. The CIFAR-10 dataset

    Download: http://www.cs.toronto.edu/~kriz/cifar.html

2. The CIFAR-100 dataset

    Download: http://www.cs.toronto.edu/~kriz/cifar.html

3. NORB Object Recognition Dataset

    Download: http://www.cs.nyu.edu/~ylclab/data/norb-v1.0/