Implement Local Response Normalization (LRN)

#189 · Deep Learning · Medium

Problem

Implement Local Response Normalization (LRN) as described in the AlexNet paper. Given a 4D input tensor (batch, channels, height, width) and parameters n, alpha, beta, and k, normalize each value using the activities of its neighboring channels.

Solution

import numpy as np

def local_response_normalization(x: np.ndarray, n: int = 5,
                                  alpha: float = 1e-4,
                                  beta: float = 0.75,
                                  k: float = 2.0) -> np.ndarray:
    # x shape: (batch, channels, height, width)
    batch, channels, height, width = x.shape
    output = np.zeros_like(x)
    half_n = n // 2

    for b in range(batch):
        for c in range(channels):
            c_start = max(0, c - half_n)
            c_end = min(channels, c + half_n + 1)
            neighborhood = x[b, c_start:c_end, :, :]
            sum_sq = np.sum(neighborhood ** 2, axis=0)
            denom = (k + alpha * sum_sq) ** beta
            output[b, c, :, :] = x[b, c, :, :] / denom

    return output

Explanation

For each activation at position (b, c, h, w), consider a window of n neighboring channels centered at c.
Compute the sum of squares across those channels: sum_sq = sum(x[b, j, h, w]^2) for j in the neighborhood.
Normalize: output = x[b, c, h, w] / (k + alpha * sum_sq)^beta.
The hyperparameters k, alpha, beta, and n control the strength and scope of the normalization.

Complexity

Time: O(B C H W n) where B is batch size, C is channels, H/W are spatial dimensions
Space: O(B C H * W) for the output tensor

← #188 #190 →