Implement the Softsign Activation Function

Problem

Implement the Softsign activation function. Softsign is defined as f(x) = x / (1 + |x|). It is similar to tanh but converges to its asymptotes more slowly.

Solution

import numpy as np

def softsign(x: np.ndarray) -> np.ndarray:
    return x / (1 + np.abs(x))

Explanation

1. Softsign maps inputs to the range (-1, 1), similar to tanh.
2. The formula x / (1 + |x|) is computationally simpler than tanh since it avoids exponential operations.
3. Softsign approaches its asymptotes polynomially (as 1 - 1/|x|), while tanh approaches exponentially. This means softsign has longer tails.
4. The slower saturation can help with gradient flow in deep networks.

Complexity

• Time: O(n) where n is the number of elements
• Space: O(n) for the output array