KL Divergence Between Two Normal Distributions

Problem

Compute the Kullback-Leibler (KL) divergence between two univariate normal distributions. Given the means and variances of two Gaussian distributions, compute KL(P || Q).

Solution

import numpy as np

def kl_divergence_normal(mu_p, sigma2_p, mu_q, sigma2_q):
    mu_p = float(mu_p)
    mu_q = float(mu_q)
    sigma2_p = float(sigma2_p)
    sigma2_q = float(sigma2_q)

    kl = 0.5 * (
        np.log(sigma2_q / sigma2_p)
        + sigma2_p / sigma2_q
        + (mu_p - mu_q) ** 2 / sigma2_q
        - 1
    )
    return float(kl)

Explanation

1. The KL divergence from P = N(mu_p, sigma2_p) to Q = N(mu_q, sigma2_q) has a closed-form solution.
2. The formula is: KL(P || Q) = 0.5 * (log(sigma2_q / sigma2_p) + sigma2_p / sigma2_q + (mu_p - mu_q)^2 / sigma2_q - 1).
3. This measures how much information is lost when Q is used to approximate P.
4. KL divergence is always non-negative and equals zero only when P and Q are identical.

Complexity

• Time: O(1)
• Space: O(1)