Single Neuron with Backpropagation

#25 · Deep Learning · Medium

Problem

Implement a single neuron with backpropagation. Given inputs, weights, bias, and true labels, perform a forward pass and then update weights and bias using gradient descent.

Solution

import math

def single_neuron_backprop(features: list[list[float]], labels: list[int],
                            weights: list[float], bias: float,
                            learning_rate: float, epochs: int) -> tuple[list[float], float, list[float]]:
    n = len(features)
    num_features = len(weights)
    w = list(weights)
    b = bias

    for _ in range(epochs):
        # Forward pass
        outputs = []
        for x in features:
            z = sum(x[j] * w[j] for j in range(num_features)) + b
            a = 1 / (1 + math.exp(-z))
            outputs.append(a)

        # Compute gradients
        dw = [0.0] * num_features
        db = 0.0
        for i in range(n):
            error = outputs[i] - labels[i]
            deriv = outputs[i] * (1 - outputs[i])  # sigmoid derivative
            delta = error * deriv
            for j in range(num_features):
                dw[j] += delta * features[i][j]
            db += delta

        # Average gradients
        dw = [d / n for d in dw]
        db /= n

        # Update weights and bias
        for j in range(num_features):
            w[j] -= learning_rate * dw[j]
        b -= learning_rate * db

    # Final MSE
    mse_loss = sum((outputs[i] - labels[i]) ** 2 for i in range(n)) / n
    return [round(wi, 4) for wi in w], round(b, 4), round(mse_loss, 4)

Explanation

Forward pass: Compute z = w*x + b, then apply sigmoid to get the output.
Loss: Mean squared error between predictions and labels.
Backward pass: Compute gradients of MSE w.r.t. weights and bias using the chain rule through the sigmoid.
Update: Adjust weights and bias in the direction that reduces the loss.
Repeat for the specified number of epochs.

Complexity

Time: O(epochs n f) where n is samples and f is features
Space: O(f) for weights and gradients

← #24 #26 →