Compute Multi-class Cross-Entropy Loss

Problem

Compute the multi-class cross-entropy loss given predicted probabilities and true labels. The cross-entropy loss measures the difference between the predicted probability distribution and the true distribution.

Solution

import numpy as np

def cross_entropy_loss(predictions: np.ndarray, targets: np.ndarray, epsilon: float = 1e-12) -> float:
    # predictions: (N, C) predicted probabilities
    # targets: (N,) integer class labels or (N, C) one-hot encoded
    predictions = np.clip(predictions, epsilon, 1 - epsilon)

    if targets.ndim == 1:
        # Integer labels
        N = predictions.shape[0]
        log_probs = -np.log(predictions[np.arange(N), targets])
        return float(np.mean(log_probs))
    else:
        # One-hot encoded
        return float(-np.mean(np.sum(targets * np.log(predictions), axis=1)))

Explanation

1. Clip predictions to avoid log(0) which gives -infinity.
2. If targets are integer class labels, index into predictions to get the predicted probability for the true class, then take -log.
3. If targets are one-hot encoded, compute the element-wise product with log-predictions and sum over classes.
4. Average the loss over all samples in the batch.

Complexity

• Time: O(N * C) where N = number of samples, C = number of classes
• Space: O(N) for the per-sample losses