Implement Gradient Clipping by Value

Problem

Implement gradient clipping by value. Given a list of gradients, clip each element so that its value falls within the range [-clip_value, clip_value].

Solution

import numpy as np

def gradient_clipping_by_value(gradients: list[np.ndarray], clip_value: float) -> list[np.ndarray]:
    return [np.clip(g, -clip_value, clip_value) for g in gradients]

Explanation

1. For each gradient tensor, apply element-wise clipping using np.clip.
2. Any value below -clip_value is set to -clip_value; any value above clip_value is set to clip_value.
3. This prevents exploding gradients during backpropagation while preserving the gradient direction for values within the range.
4. Unlike gradient clipping by norm, this method clips each element independently, which can change the gradient direction.

Complexity

• Time: O(n) where n is the total number of gradient elements across all tensors
• Space: O(n) for the clipped gradient copies