← back

Compute the Hessian Matrix

#218 · Deep Learning · Medium

⊣ Solve on deep-ml.com

Problem

Compute the Hessian matrix (matrix of second-order partial derivatives) of a given multivariable function at a specified point using numerical differentiation.

Solution

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
def compute_hessian(f, point: list[float], h: float = 1e-5) -> list[list[float]]:
    n = len(point)
    hessian = [[0.0] * n for _ in range(n)]

    for i in range(n):
        for j in range(n):
            # f(x + ei*h + ej*h)
            pp = point[:]
            pp[i] += h
            pp[j] += h
            f_pp = f(pp)

            # f(x + ei*h - ej*h)
            pm = point[:]
            pm[i] += h
            pm[j] -= h
            f_pm = f(pm)

            # f(x - ei*h + ej*h)
            mp = point[:]
            mp[i] -= h
            mp[j] += h
            f_mp = f(mp)

            # f(x - ei*h - ej*h)
            mm = point[:]
            mm[i] -= h
            mm[j] -= h
            f_mm = f(mm)

            hessian[i][j] = round((f_pp - f_pm - f_mp + f_mm) / (4 * h * h), 4)

    return hessian

Explanation

  1. Use the finite difference formula for mixed second-order partial derivatives.
  2. For each pair (i, j), compute d^2f / (dx_i dx_j) using the four-point stencil: (f(x+h_i+h_j) - f(x+h_i-h_j) - f(x-h_i+h_j) + f(x-h_i-h_j)) / (4h^2).
  3. This produces a symmetric n x n matrix of second derivatives.

Complexity

  • Time: O(n^2) function evaluations where n is the dimensionality
  • Space: O(n^2) for the Hessian matrix
← #217#219 →