January 05, 2024
An activation function plays an important role in creating the output of a node in the neural network.
An activation function takes in the matrix output of the node and determines if and how the node will propagate its information to the network layer.
Activation function act as filters to reduce noise and also normalize the output, which can get fairly large due to matrix multiplications.
It converts the outputs to a nonlinear value.
They serve as a critical step in helping a neural network learn specific patterns in data.
Here is the list of some of the most popular activation functions. Each of these functions have specific advantages, shortcomings and applications. They can take in an output matrix and deliver another output matrix of the same dimension.
A sigmoid function delivers an output in the range of zero to one, based on the input values. When it has a value of zero, it means that it does not pass its learnings to the next layer.
A Tanh function normalizes the output in the range of minus one to plus one.
A Rectified Linear Unit or ReLU, produces a zero if the output is negative. Else, it reproduces the same input verbatim.
A Softmax activation function is used in case of classification problems. it produces a vector of probabilities for each of the possible classes in the outcome. The sum of probabilities will be equal to one.
The class with the highest probability will be considered for prediction. These functions needed to be just added as a hyperparameter to the model.