[1409.2329] Recurrent Neural Network Regularization

http://arxiv.org/abs/1409.2329

We present a simple regularization technique for Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units. Dropout, ...

LSTM に dropout を適用するには、リカレント（同じ層の1つ前の時間ステップ）の隠れユニットに適用するのではなく、入力（または１つ下の層の隠れユニット）に対して適用するとよいという話。

Figure 2 と p3 の下の式の太字 D の箇所を参照するとよい。

The Unreasonable Effectiveness of Recurrent Neural Networks

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

We'll train RNNs to generate text character by character and ponder the question "how is that even possible?"

Frontiers | MACOP modular architecture with control primitives | Frontiers in Computational Neuroscience

http://journal.frontiersin.org/article/10.3389/fncom.2013.00099/full

2.4.2. Echo state networks
We use an Echo State Network (ESN) (Jaeger, 2001) as inverse model. An ESN is composed of a discrete-time recurrent neural network [commonly called the reservoir because ESNs belong to the class of Reservoir Computing techniques (Schrauwen et al., 2007)] and a linear readout layer which maps the state of the reservoir to the desired output.

Download PDF
http://journal.frontiersin.org/article/10.3389/fncom.2013.00099/pdf

[1504.00941] A Simple Way to Initialize Recurrent Networks of Rectified Linear Units

http://arxiv.org/abs/1504.00941

In this paper, we propose a simpler solution that use recurrent neural networks composed of rectified linear units.

Key to our solution is the use of the identity matrix or its scaled version to initialize the recurrent weight matrix.

再帰ネットに ReLU (rectified linear units) を使って長期の時間的構造を学習する。