Positional Encoding Calculator

Problem

Implement the positional encoding used in Transformer models. Given a sequence length and embedding dimension, generate the positional encoding matrix using sine and cosine functions at different frequencies.

Solution

import numpy as np

def positional_encoding(seq_len, d_model):
    PE = np.zeros((seq_len, d_model))

    for pos in range(seq_len):
        for i in range(0, d_model, 2):
            denominator = 10000 ** (i / d_model)
            PE[pos, i] = np.sin(pos / denominator)
            if i + 1 < d_model:
                PE[pos, i + 1] = np.cos(pos / denominator)

    return PE.tolist()

Explanation

1. Create a matrix of shape (seq_len, d_model) initialized to zeros.
2. For each position pos and each dimension index i:
3. - Even indices: PE(pos, i) = sin(pos / 10000^(i/d_model))
4. - Odd indices: PE(pos, i+1) = cos(pos / 10000^(i/d_model))
5. The wavelengths form a geometric progression from 2pi to 100002*pi.
6. Different frequencies allow the model to learn to attend to relative positions.
7. This encoding is added to the input embeddings before being fed to the Transformer.

Complexity

• Time: O(seq_len * d_model)
• Space: O(seq_len * d_model)