2015-03-16

Rectifier (neural networks) - Wikipedia, the free encyclopedia

http://en.wikipedia.org/wiki/Rectifier_%28neural_networks%29#Leaky_ReLUs
Leaky ReLUs allow a small, non-zero gradient when the unit is not active.