Explain Leaky ReLU activation functions. Write mathematical expressions, range with its graph.

Leaky ReLU (Rectified Linear Unit) is a type of activation function that is used in artificial neural networks. It is a variant of the standard ReLU function, designed to address the "dying ReLU" problem where neurons can sometimes become inactive and no longer learn from the data.

The mathematical expression for Leaky ReLU is:

f(x) = max(0.01x, x)

This means that if the input x is positive, the output is x. If the input x is negative, the output is 0.01x. This small slope for negative values (0.01 in this case) ensures that Leaky ReLU never becomes inactive and can continue learning.

The range of Leaky ReLU is (-∞, ∞). This means it can output any real number, unlike the standard ReLU function which can only output positive numbers.

The graph of the Leaky ReLU function is a straight line in the second quadrant (because of the 0.01x term for negative x) and a straight line along the line y=x in the first quadrant (because the output is x for positive x). This makes it look like a "leaky" version of the standard ReLU function, hence the name.