Implement the PReLU Activation Function

Problem

Implement the PReLU (Parametric Rectified Linear Unit) activation function. PReLU generalizes ReLU by learning the slope for negative inputs: f(x) = x if x > 0, else f(x) = alpha * x, where alpha is a learnable parameter.

Solution

import numpy as np

def prelu(x: np.ndarray, alpha: float = 0.25) -> np.ndarray:
    return np.where(x > 0, x, alpha * x)

Explanation

1. For positive inputs, PReLU returns the input unchanged (identity function).
2. For negative inputs, the output is alpha * x where alpha is typically a small positive value.
3. Unlike Leaky ReLU where alpha is a fixed hyperparameter, in PReLU alpha is learned during training via backpropagation.
4. This prevents the "dying ReLU" problem where neurons can become permanently inactive if they always output zero.

Complexity

• Time: O(n) where n is the number of elements
• Space: O(n) for the output array