On the Importance of the Activation Function in Training Deep Neural Networks

This study investigated the Fundamental Deep Learning Problem, namely the problem of vanishing gradients when training deep neural networks. It identified the importance of the activation function, the impact this has on performance, and proposed a novel piecewise linear activation which outperforms current state-of-the-art results from rectifier units.