Gradient Direction and Magnitude

Problem

Compute the gradient direction and magnitude for a given multivariable function at a specific point. The gradient points in the direction of steepest ascent.

Solution

import numpy as np

def numerical_gradient(f, x: np.ndarray, h: float = 1e-7) -> np.ndarray:
    grad = np.zeros_like(x, dtype=float)
    for i in range(len(x)):
        x_plus = x.copy()
        x_minus = x.copy()
        x_plus[i] += h
        x_minus[i] -= h
        grad[i] = (f(x_plus) - f(x_minus)) / (2 * h)
    return grad

def gradient_direction_and_magnitude(f, x: np.ndarray) -> dict:
    grad = numerical_gradient(f, x)
    magnitude = float(np.linalg.norm(grad))
    if magnitude > 0:
        direction = grad / magnitude
    else:
        direction = grad
    return {
        "gradient": grad,
        "magnitude": magnitude,
        "direction": direction,
    }

Explanation

1. Numerical gradient uses the central difference formula: df/dx_i approx (f(x + he_i) - f(x - he_i)) / (2h) for each dimension.
2. Magnitude is the L2 norm of the gradient vector, indicating how steep the function is at that point.
3. Direction is the unit vector in the gradient direction (gradient / magnitude), pointing toward steepest ascent.
4. In gradient descent, we move in the negative gradient direction to minimize the function.

Complexity

• Time: O(n) function evaluations where n is the number of dimensions
• Space: O(n) for the gradient vector