Saba Shahrukh July 8, 2026 0 If you want to keep track of your post-reading status, please register on the site.

Going through a neural network from scratch is one of the best ways to truly understand how they work under the hood.

Instead of going strictly one single line at a time (which can get a bit tedious with blank lines and brackets), I have grouped the code into small, logical chunks and explained what the lines inside each chunk are doing.

1. Imports and Activation Functions

Python

import math
import random
  • import math: We bring in Python’s built-in math module to use mathematical operations like the hyperbolic tangent (math.tanh).
  • import random: We need this to generate random initial values for our network’s weights and biases.

Python

def tanh(x): return math.tanh(x)
def tanh_deriv(x): return 1.0 - math.tanh(x)**2
  • These define the Tanh activation function and its derivative. The derivative formula $1 – \tanh(x)^2$ is required later for backpropagation to calculate how much we need to adjust the weights.

Python

def relu(x): return max(0.0, x)
def relu_deriv(x): return 1.0 if x > 0 else 0.0
  • These define ReLU (Rectified Linear Unit). If the input $x$ is negative, it outputs $0$. If it is positive, it outputs $x$. The derivative is simply $1$ if $x > 0$, and $0$ otherwise.

Python

def leaky_relu(x): return x if x > 0 else 0.01 * x
def leaky_relu_deriv(x): return 1.0 if x > 0 else 0.01
  • Leaky ReLU is similar to ReLU, but instead of outputting exactly $0$ for negative numbers, it allows a small “leak” (multiplying by $0.01$). This helps prevent neurons from “dying” (getting stuck at zero).

2. Setting Up the Network Architecture

Python

class MultilayerPerceptron:
    def __init__(self, input_nodes, hidden_nodes, output_nodes, activation='tanh'):
        self.input_nodes = input_nodes
        self.hidden_nodes = hidden_nodes
        self.output_nodes = output_nodes
  • We define a class to hold our network. The __init__ method sets it up. We store the number of neurons we want in our input layer, hidden layer, and output layer.

Python

        if activation == 'tanh':
            self.act, self.act_deriv = tanh, tanh_deriv
        # ... (other elif branches for relu/leaky_relu)
  • We check which activation function you requested when creating the network and assign the math functions we defined earlier to self.act and self.act_deriv.

Python

        self.weights_ih = [[random.uniform(-0.5, 0.5) for _ in range(input_nodes)] for _ in range(hidden_nodes)]
        self.weights_ho = [[random.uniform(-0.5, 0.5) for _ in range(hidden_nodes)] for _ in range(output_nodes)]
  • self.weights_ih: Creates the weights between the Input and Hidden layers. It creates a list of lists (a matrix) filled with random numbers between $-0.5$ and $0.5$.
  • self.weights_ho: Does the exact same thing, but for the weights connecting the Hidden and Output layers.

Python

        self.bias_h = [random.uniform(-0.5, 0.5) for _ in range(hidden_nodes)]
        self.bias_o = [random.uniform(-0.5, 0.5) for _ in range(output_nodes)]
  • Every neuron (except inputs) gets a “bias” value, which shifts the activation function left or right. We initialize these with random numbers just like the weights.

3. The Forward Pass (Making Predictions)

Python

    def forward(self, inputs):
        if len(inputs) != self.input_nodes:
            raise ValueError("Input feature count must match input nodes.")
  • We start the forward pass (pushing data through the network). We first check to make sure the data you gave us matches the number of input nodes we created.

Python

        self.hidden_inputs = []
        self.hidden_outputs = []
        for i in range(self.hidden_nodes):
            z = self.bias_h[i]
            for j in range(self.input_nodes):
                z += self.weights_ih[i][j] * inputs[j]
            self.hidden_inputs.append(z)
            self.hidden_outputs.append(self.act(z))
  • This block calculates the values for the hidden layer.
  • We loop through every hidden neuron i.
  • We start the total value z with the neuron’s bias.
  • We loop through every input j, multiply the input by the connecting weight, and add it to z. Mathematically, this is $z = \text{bias} + \sum (\text{weight} \times \text{input})$.
  • We save the raw sum (self.hidden_inputs) and then apply our activation function to get the final output for that hidden neuron (self.hidden_outputs).

Python

        self.output_inputs = []
        self.output_outputs = []
        for i in range(self.output_nodes):
            z = self.bias_o[i]
            for j in range(self.hidden_nodes):
                z += self.weights_ho[i][j] * self.hidden_outputs[j]
            self.output_inputs.append(z)
            self.output_outputs.append(self.act(z))
            
        return self.output_outputs
  • This does the exact same process as the block above, but it moves data from the hidden layer to the output layer. Finally, it returns the network’s prediction.

4. Backpropagation (Learning from Mistakes)

Python

    def backward(self, inputs, targets, learning_rate):
        outputs = self.forward(inputs)
  • To learn, we first need to make a prediction. We call our forward method to see what the network currently thinks the answer is.

Python

        output_deltas = []
        for i in range(self.output_nodes):
            error = targets[i] - outputs[i]
            delta = error * self.act_deriv(self.output_inputs[i])
            output_deltas.append(delta)
  • We calculate how wrong the output was (targets - outputs).
  • We calculate the delta (the gradient) by multiplying the error by the derivative of the activation function. This tells us which direction and how much to change the weights.

Python

        hidden_deltas = []
        for i in range(self.hidden_nodes):
            error = 0.0
            for j in range(self.output_nodes):
                error += output_deltas[j] * self.weights_ho[j][i]
            delta = error * self.act_deriv(self.hidden_inputs[i])
            hidden_deltas.append(delta)
  • Now we work backwards to the hidden layer. Hidden neurons do not have a direct “target”, so their error is calculated by looking at how much they contributed to the error of the output neurons they connect to.

Python

        for i in range(self.output_nodes):
            for j in range(self.hidden_nodes):
                self.weights_ho[i][j] += learning_rate * output_deltas[i] * self.hidden_outputs[j]
            self.bias_o[i] += learning_rate * output_deltas[i]
  • Gradient Descent: Now we actually update the weights! We change the Hidden-to-Output weights by adding a fraction (learning_rate) of the delta multiplied by the output of the hidden neuron. We also update the bias.

Python

        for i in range(self.hidden_nodes):
            for j in range(self.input_nodes):
                self.weights_ih[i][j] += learning_rate * hidden_deltas[i] * inputs[j]
            self.bias_h[i] += learning_rate * hidden_deltas[i]
  • We do the exact same update process for the Input-to-Hidden layer weights and biases.

5. Training Loop and Preprocessing

Python

    def train(self, training_data, epochs, learning_rate):
        for epoch in range(epochs):
            for inputs, targets in training_data:
                self.backward(inputs, targets, learning_rate)
  • This is a helper function. An “epoch” is one full pass through the training data. We loop over the number of epochs requested, and for every piece of data, we run the backward function to update the weights.

Python

def convert_categorical_to_numeric(dataset):
    processed_dataset = []
    vocab = {"Red": 0.0, "Blue": 1.0, "Green": 2.0} 
    
    for row, target in dataset:
        numeric_row = []
        for feature in row:
            if isinstance(feature, str) and feature in vocab:
                numeric_row.append(vocab[feature])
            else:
                numeric_row.append(feature)
        processed_dataset.append((numeric_row, target))
    return processed_dataset
  • Neural networks can only understand numbers. This function loops through a dataset. If it sees a string (like “Red”), it looks it up in a dictionary (vocab) and replaces it with a number ($0.0$). If it’s already a number, it leaves it alone.

6. The Execution Block

Python

if __name__ == "__main__":
    random.seed(42)
  • We set a “seed” for the random number generator. This forces the random weights to be exactly the same every time you run the script, which is helpful for testing and debugging.

The rest of the __main__ block simply creates instances of the MultilayerPerceptron class, feeds it the specific datasets (like the AND gate and XOR gate data), calls the .train() method to make it learn, and then prints out the final predictions to prove that the math worked!

Here is a complete, from-scratch Python implementation of a Multilayer Perceptron (MLP) that covers all the concepts mentioned in your handwritten notes.

This code uses only standard, built-in Python constructs (using the math and random modules for basic math operations and weight initialization) and avoids any external dependencies like NumPy, PyTorch, or TensorFlow.

The Plain Python MLP Implementation

Python

import math
import random

# =====================================================================
# 1. ACTIVATION FUNCTIONS & DERIVATIVES
# Notes reference: "Activation function - typically non-linear such as 
# tanh, ReLU, Leaky ReLU, etc."
# =====================================================================

def tanh(x):
    return math.tanh(x)

def tanh_deriv(x):
    # Derivative of tanh(x) is 1 - tanh(x)^2
    return 1.0 - math.tanh(x)**2

def relu(x):
    return max(0.0, x)

def relu_deriv(x):
    return 1.0 if x > 0 else 0.0

def leaky_relu(x):
    return x if x > 0 else 0.01 * x

def leaky_relu_deriv(x):
    return 1.0 if x > 0 else 0.01


# =====================================================================
# 2. MULTILAYER PERCEPTRON CLASS
# Notes reference: "Architecture - input layer, one or more hidden layers, 
# output layer."
# =====================================================================

class MultilayerPerceptron:
    def __init__(self, input_nodes, hidden_nodes, output_nodes, activation='tanh'):
        self.input_nodes = input_nodes
        self.hidden_nodes = hidden_nodes
        self.output_nodes = output_nodes
        
        # Select the non-linear activation function mapped from the notes
        if activation == 'tanh':
            self.act, self.act_deriv = tanh, tanh_deriv
        elif activation == 'relu':
            self.act, self.act_deriv = relu, relu_deriv
        elif activation == 'leaky_relu':
            self.act, self.act_deriv = leaky_relu, leaky_relu_deriv
        else:
            raise ValueError("Unsupported activation function")

        # Initialize Weights (Random values between -0.5 and 0.5)
        # Weights from Input layer to Hidden layer
        self.weights_ih = [[random.uniform(-0.5, 0.5) for _ in range(input_nodes)] for _ in range(hidden_nodes)]
        # Weights from Hidden layer to Output layer
        self.weights_ho = [[random.uniform(-0.5, 0.5) for _ in range(hidden_nodes)] for _ in range(output_nodes)]

        # Initialize Biases
        self.bias_h = [random.uniform(-0.5, 0.5) for _ in range(hidden_nodes)]
        self.bias_o = [random.uniform(-0.5, 0.5) for _ in range(output_nodes)]

    def forward(self, inputs):
        """
        Forward Pass computing the output of the network.
        Notes reference: The diagram showing nodes and weights connected.
        """
        if len(inputs) != self.input_nodes:
            raise ValueError("Input feature count must match input nodes.")

        # --- Input to Hidden Layer ---
        self.hidden_inputs = []
        self.hidden_outputs = []
        for i in range(self.hidden_nodes):
            # Sum of (weight * input) + bias for the hidden neuron
            z = self.bias_h[i]
            for j in range(self.input_nodes):
                z += self.weights_ih[i][j] * inputs[j]
            self.hidden_inputs.append(z)
            self.hidden_outputs.append(self.act(z)) # Apply non-linearity

        # --- Hidden to Output Layer ---
        self.output_inputs = []
        self.output_outputs = []
        for i in range(self.output_nodes):
            # Sum of (weight * hidden_output) + bias for the output neuron
            z = self.bias_o[i]
            for j in range(self.hidden_nodes):
                z += self.weights_ho[i][j] * self.hidden_outputs[j]
            self.output_inputs.append(z)
            self.output_outputs.append(self.act(z))

        return self.output_outputs

    def backward(self, inputs, targets, learning_rate):
        """
        Notes reference: "MLP is trained using backpropagation algorithm 
        with gradient descent or its variants"
        """
        # 1. Forward pass to get current predictions
        outputs = self.forward(inputs)

        # 2. Calculate Output Layer Errors & Deltas
        output_deltas = []
        for i in range(self.output_nodes):
            error = targets[i] - outputs[i] # Difference between target and actual
            delta = error * self.act_deriv(self.output_inputs[i])
            output_deltas.append(delta)

        # 3. Calculate Hidden Layer Errors & Deltas
        hidden_deltas = []
        for i in range(self.hidden_nodes):
            error = 0.0
            # Backpropagate the error from output neurons connected to this hidden neuron
            for j in range(self.output_nodes):
                error += output_deltas[j] * self.weights_ho[j][i]
            delta = error * self.act_deriv(self.hidden_inputs[i])
            hidden_deltas.append(delta)

        # 4. Gradient Descent: Update Weights and Biases
        
        # Update Hidden-to-Output Weights & Biases
        for i in range(self.output_nodes):
            for j in range(self.hidden_nodes):
                self.weights_ho[i][j] += learning_rate * output_deltas[i] * self.hidden_outputs[j]
            self.bias_o[i] += learning_rate * output_deltas[i]

        # Update Input-to-Hidden Weights & Biases
        for i in range(self.hidden_nodes):
            for j in range(self.input_nodes):
                self.weights_ih[i][j] += learning_rate * hidden_deltas[i] * inputs[j]
            self.bias_h[i] += learning_rate * hidden_deltas[i]

    def train(self, training_data, epochs, learning_rate):
        """ Trains the network over a given dataset """
        for epoch in range(epochs):
            for inputs, targets in training_data:
                self.backward(inputs, targets, learning_rate)

# =====================================================================
# 3. DATA PREPROCESSING (CATEGORICAL TO NUMERIC)
# Notes reference: "Input features must be numeric", 
# "Non numeric input features to be converted to numeric ones"
# =====================================================================

def convert_categorical_to_numeric(dataset):
    """
    A simple function demonstrating how to convert non-numeric features 
    (like "Red", "Blue") to numeric values (Label Encoding).
    """
    processed_dataset = []
    # Create a simple vocabulary mapping
    vocab = {"Red": 0.0, "Blue": 1.0, "Green": 2.0} 
    
    for row, target in dataset:
        numeric_row = []
        for feature in row:
            if isinstance(feature, str) and feature in vocab:
                numeric_row.append(vocab[feature])
            else:
                numeric_row.append(feature)
        processed_dataset.append((numeric_row, target))
    return processed_dataset


# =====================================================================
# 4. TESTING THE CONCEPTS (RUNNING THE CODE)
# Notes reference: "Can solve complex real world problems involving 
# non-linear decision boundary" & "Linearly Separable" vs "Non-Linearly"
# =====================================================================

if __name__ == "__main__":
    # Ensure reproducibility
    random.seed(42)
    
    print("--- Demonstrating Preprocessing ---")
    raw_data = [(["Red", 1.5], [1]), (["Blue", 2.2], [0])]
    numeric_data = convert_categorical_to_numeric(raw_data)
    print(f"Raw Non-Numeric: {raw_data}")
    print(f"Converted Numeric: {numeric_data}\n")

    # --- Problem 1: Linearly Separable Data (AND Gate) ---
    print("--- Training on Linearly Separable Data (AND Gate) ---")
    and_data = [
        ([0, 0], [0]),
        ([0, 1], [0]),
        ([1, 0], [0]),
        ([1, 1], [1])
    ]
    
    # Architecture: 2 Input, 2 Hidden, 1 Output (Suitable for Classification)
    linear_mlp = MultilayerPerceptron(2, 2, 1, activation='tanh')
    linear_mlp.train(and_data, epochs=2000, learning_rate=0.1)
    
    for x, _ in and_data:
        pred = linear_mlp.forward(x)[0]
        print(f"Input: {x} -> Predicted: {pred:.3f} (Rounded: {round(max(0, pred))})")

    print("\n--- Training on Non-Linearly Separable Data (XOR Gate) ---")
    # --- Problem 2: Non-Linearly Separable Data (XOR Gate) ---
    # This requires the hidden layer to solve the complex decision boundary
    xor_data = [
        ([0, 0], [0]),
        ([0, 1], [1]),
        ([1, 0], [1]),
        ([1, 1], [0])
    ]
    
    # Architecture: 2 Input, 4 Hidden, 1 Output
    non_linear_mlp = MultilayerPerceptron(2, 4, 1, activation='tanh')
    non_linear_mlp.train(xor_data, epochs=4000, learning_rate=0.1)
    
    for x, _ in xor_data:
        pred = non_linear_mlp.forward(x)[0]
        # Using a slight threshold logic for continuous output formatting
        print(f"Input: {x} -> Predicted: {pred:.3f} (Rounded: {round(max(0, pred))})")

How This Code Maps to Your Notes:

  • Architecture (__init__): Follows the exact diagram layout (Input Neurons > Weights > Hidden Neurons > Weights > Output Neurons).
  • Activation Functions: Includes custom tanh, relu, and leaky_relu standard logic and their respective derivatives natively.
  • Backpropagation & Gradient Descent (backward): The math computes the error signals traversing backwards layer by layer, adjusting network weights by multiplying the error delta, the gradient descent learning_rate, and the inputs of that layer.
  • Numeric Constraints (convert_categorical_to_numeric): Resolves the rule at the bottom of the page noting that inputs must be numeric, using a small helper function to process string data.
  • Linearly vs. Non-Linearly Separable (__main__): The script directly tests the network’s ability against both an AND operation (linearly separable, as drawn on the right side of the graph) and an XOR operation (non-linearly separable, as drawn on the left side).
Category: 

Leave a Comment