Calculate Perplexity for Language Models

Problem

Calculate the perplexity of a language model given a list of predicted probabilities for the actual next tokens in a sequence. Perplexity measures how well a probability model predicts a sample and is defined as the exponentiation of the average negative log-likelihood.

Solution

import math
from typing import List

def calculate_perplexity(probabilities: List[float]) -> float:
    n = len(probabilities)
    if n == 0:
        raise ValueError("Empty probability list")

    log_likelihood = 0.0
    for p in probabilities:
        if p <= 0:
            raise ValueError("Probabilities must be positive")
        log_likelihood += math.log(p)

    avg_neg_log_likelihood = -log_likelihood / n
    perplexity = math.exp(avg_neg_log_likelihood)
    return round(perplexity, 4)

Explanation

1. Perplexity is defined as: PP = exp(-1/N * sum(log(p_i))) where p_i is the probability assigned to the i-th actual token.
2. Sum the log-probabilities of all tokens.
3. Divide the negative sum by N to get the average negative log-likelihood.
4. Exponentiate to get perplexity. Lower perplexity means the model predicts the sequence better.

Complexity

• Time: O(n) where n is the number of tokens
• Space: O(1)