Implement the ELU Activation Function

Problem

Implement the ELU (Exponential Linear Unit) activation function. ELU is defined as x for x > 0 and alpha * (exp(x) - 1) for x <= 0, where alpha is a hyperparameter (default 1.0).

Solution

import numpy as np

def elu(x: np.ndarray, alpha: float = 1.0) -> np.ndarray:
    return np.where(x > 0, x, alpha * (np.exp(x) - 1))

Explanation

1. For positive inputs, ELU acts as the identity function (like ReLU).
2. For negative inputs, instead of clamping to 0 (ReLU) or a straight line (Leaky ReLU), ELU uses an exponential curve alpha * (exp(x) - 1).
3. This smoothly saturates to -alpha for large negative values, which helps push mean activations closer to zero and improves learning.
4. The alpha parameter controls the saturation value for negative inputs.

Complexity

• Time: O(n) where n is the number of elements
• Space: O(n) for the output array