How to choose the right activation function？

The activation function is decided depending upon the objective of the problem statement and the concerned properties. Some of the inferences are as follows:

• Sigmoid functions work very well in the case of shallow networks and binary classifiers. Deeper networks may lead to vanishing gradients.

• The ReLU function is the most widely used, and try using Leaky ReLU to avoid the case of dead neurons. Thus, start with ReLU, then move to another activation function if ReLU doesn't provide good results.

• Use softmax in the outer layer for the multi-class classification.

• Avoid using ReLU in the outer layer.