Instance Normalization (IN) Implementation

Problem

Implement Instance Normalization. Unlike Batch Normalization which normalizes across the batch, Instance Normalization normalizes each sample independently across spatial dimensions. It is commonly used in style transfer networks.

Solution

import numpy as np

def instance_norm(x: np.ndarray, gamma: np.ndarray = None, beta: np.ndarray = None, eps: float = 1e-5) -> np.ndarray:
    # x shape: (N, C, H, W)
    N, C, H, W = x.shape

    # Compute mean and variance per instance per channel
    mean = np.mean(x, axis=(2, 3), keepdims=True)  # (N, C, 1, 1)
    var = np.var(x, axis=(2, 3), keepdims=True)      # (N, C, 1, 1)

    # Normalize
    x_norm = (x - mean) / np.sqrt(var + eps)

    # Apply optional affine parameters (per channel)
    if gamma is not None:
        x_norm = x_norm * gamma.reshape(1, C, 1, 1)
    if beta is not None:
        x_norm = x_norm + beta.reshape(1, C, 1, 1)

    return x_norm

Explanation

1. For each sample and each channel independently, compute the mean and variance over the spatial (H, W) dimensions.
2. Normalize each spatial feature map to zero mean and unit variance.
3. Optionally apply a per-channel learnable scale (gamma) and shift (beta).
4. Unlike Batch Norm, Instance Norm does not depend on batch statistics, making it suitable for style transfer where per-instance style matters.

Complexity

• Time: O(N C H * W)
• Space: O(N C H * W) for the output