[1504.00941] A Simple Way to Initialize Recurrent Networks of Rectified Linear Units

http://arxiv.org/abs/1504.00941

In this paper, we propose a simpler solution that use recurrent neural networks composed of rectified linear units.

Key to our solution is the use of the identity matrix or its scaled version to initialize the recurrent weight matrix.

再帰ネットに ReLU (rectified linear units) を使って長期の時間的構造を学習する。

Rectifier (neural networks) - Wikipedia, the free encyclopedia

http://en.wikipedia.org/wiki/Rectifier_%28neural_networks%29#Leaky_ReLUs

Leaky ReLUs allow a small, non-zero gradient when the unit is not active.

CIFAR-10 Competition Winners: Interviews with Dr. Ben Graham, Phil Culliton, & Zygmunt Zając | no free hunch

http://blog.kaggle.com/2015/01/02/cifar-10-competition-winners-interviews-with-dr-ben-graham-phil-culliton-zygmunt-zajac/

The "MP" entries above denote max pooling, and "VeryLeakyReLU" denotes a "leaky" ReLU with a fairly large (alpha was 0.33) non-zero gradient.