← back

Derivatives of Activation Functions

#217 · Deep Learning · Easy

⊣ Solve on deep-ml.com

Problem

Compute the derivatives of common activation functions (Sigmoid, Tanh, ReLU, and Leaky ReLU) at a given input value.

Solution

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
import math

def activation_derivatives(x: float) -> dict:
    # Sigmoid
    sig = 1.0 / (1.0 + math.exp(-x))
    d_sigmoid = sig * (1.0 - sig)

    # Tanh
    t = math.tanh(x)
    d_tanh = 1.0 - t ** 2

    # ReLU
    d_relu = 1.0 if x > 0 else 0.0

    # Leaky ReLU (alpha = 0.01)
    alpha = 0.01
    d_leaky_relu = 1.0 if x > 0 else alpha

    return {
        "sigmoid": round(d_sigmoid, 4),
        "tanh": round(d_tanh, 4),
        "relu": round(d_relu, 4),
        "leaky_relu": round(d_leaky_relu, 4),
    }

Explanation

  1. Sigmoid derivative: sigma(x) * (1 - sigma(x)), where sigma(x) = 1 / (1 + e^(-x)).
  2. Tanh derivative: 1 - tanh(x)^2.
  3. ReLU derivative: 1 if x > 0, else 0.
  4. Leaky ReLU derivative: 1 if x > 0, else alpha (typically 0.01).

Complexity

  • Time: O(1)
  • Space: O(1)
← #216#218 →