Deep Neural Nets – Scaling to 3D Volumes

In the early stages of CNN hardware design, we conceptualized convolutions as 2D sliding windows over flat images. However, real-world deep learning operates on high-dimensional data. A true convolutional layer receives a 4D input tensor (Batch, Channels, Height, Width) and applies a 4D weight tensor (Filters, Channels, Kernel_Height, Kernel_Width).

No matter how many dimensions your tensors have, they must all be flattened into exactly TWO 2-dimensional matrices before hitting the GPU.

This article walks through the exact step-by-step physical memory unrolling required to achieve this, complete with isolated code implementations for each step.

Step 1: Handling Multiple Input Channels (The C Dimension)

When an input image has multiple channels (e.g., RGB, or deep feature maps), the sliding window is no longer a flat square; it is a 3D block.

• If our kernel is $2 \times 2$ and we have $2$ input channels, the receptive field extracts a $2 \times 2 \times 2$ volume.
• The Rule of Flattening: We must flatten this entire 3D volume into a single 1D column. We do this by taking the elements of Channel 1, flattening them, and then appending the flattened elements of Channel 2 directly underneath them.
• A single column’s height goes from $R \times S$ to $C \times R \times S$.

Code Execution: Flattening a Multi-Channel Patch

import numpy as np

print("--- STEP 1: MULTI-CHANNEL PATCH FLATTENING ---")
# 1. Define dimensions
C = 2  # Channels (e.g., Black and Yellow)
R, S = 2, 2  # Kernel Height and Width

# 2. Simulate a single 3D receptive field extracted from the image
# Shape: (C, R, S)
receptive_field = np.array([
    [[1, 2], [4, 5]],  # Channel 1 (Black grid from slide)
    [[1, 2], [4, 5]]   # Channel 2 (Yellow grid from slide)
])

# 3. Simulate the 3D Filter for a single output channel
filter_3d = np.array([
    [[1, 2], [3, 4]],  # Filter weights for Channel 1
    [[1, 2], [3, 4]]   # Filter weights for Channel 2
])

# 4. The Hardware Unrolling Step
# Flatten the entire 3D block into a 1D array of size (C * R * S)
unrolled_column = receptive_field.flatten()
flattened_filter = filter_3d.flatten()

print(f"Original Patch Shape: {receptive_field.shape}")
print(f"Unrolled Column Shape: {unrolled_column.shape} -> Expected (C*R*S) = 8")
print(f"Unrolled Column Data: {unrolled_column}")
print(f"Flattened Filter Data: {flattened_filter}")

# The spatial 3D dot product is now a simple 1D linear dot product!
output_scalar = np.dot(flattened_filter, unrolled_column)
print(f"Output computed scalar: {output_scalar}")

Step 2: Handling Multiple Filters (The M Dimension)

A convolutional layer doesn’t just look for one feature; it looks for many. If we have $M$ distinct filters, we want to apply all of them to the exact same input image.

• Since each filter has a shape of (C, R, S), we flatten each filter into a 1D row vector of length $C \times R \times S$.
• We stack these $M$ row vectors vertically.
• The Weight Matrix: Our weights are now a massive 2D matrix of shape [M, C * R * S].

Code Execution: Constructing the Weight Matrix

print("\n--- STEP 2: CONSTRUCTING THE WEIGHT MATRIX ---")
M = 2  # Number of output filters (Chnl 1 and Chnl 2 in output)
C, R, S = 2, 2, 2

# Create a 4D weight tensor (M, C, R, S)
# We will use the exact values from Slide 16
weights_4d = np.zeros((M, C, R, S))

# Filter 1
weights_4d[0, 0] = [[1, 2], [3, 4]] # Ch 1
weights_4d[0, 1] = [[1, 2], [3, 4]] # Ch 2
# Filter 2
weights_4d[1, 0] = [[1, 2], [3, 4]] # Ch 1
weights_4d[1, 1] = [[1, 2], [3, 4]] # Ch 2

# The Hardware Unrolling Step
# Reshape to (M, C * R * S)
weight_matrix = weights_4d.reshape(M, -1)

print(f"4D Weights Shape: {weights_4d.shape}")
print(f"2D Weight Matrix Shape: {weight_matrix.shape} -> Expected [M, C*R*S]")
print("Weight Matrix (Rows = Filters, Cols = Channel Weights):")
print(weight_matrix)

Step 3: The Ultimate Hardware Equation (The Final GEMM)

Now we combine everything, including the Batch dimension ($N$). Instead of looking at a single patch, we slide our window across all $N$ images, extracting every spatial location, and stacking them horizontally as columns. The total number of sliding window positions per image is $H_{out} \times W_{out}$. For $N$ images, it is $N \times H_{out} \times W_{out}$.

This gives us the final, optimized hardware execution shapes:

1. Filter Matrix: [M, C * R * S]
2. Toeplitz Input Matrix: [C * R * S, N * H_out * W_out]
3. Output Matrix: The matrix multiplication of the above yields [M, N * H_out * W_out].

Code Execution: The End-to-End Volumetric Im2Col Engine

print("\n--- STEP 3: THE ULTIMATE HARDWARE EQUATION ---")

# 1. Dimensions based on Slide 17
N = 1          # Batch Size
C = 2          # Input Channels
H_in, W_in = 3, 3 # Input Spatial Dimensions
M = 2          # Number of Filters
R, S = 2, 2    # Filter Spatial Dimensions
stride = 1

H_out = int((H_in - R) / stride) + 1
W_out = int((W_in - S) / stride) + 1

# 2. Initialize Tensors
# Input Image (N, C, H, W)
I = np.zeros((N, C, H_in, W_in))
I[0, 0] = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
I[0, 1] = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]

# Filters (M, C, R, S)
W = np.zeros((M, C, R, S))
for m in range(M):
    for c in range(C):
        W[m, c] = [[1, 2], [3, 4]]

# 3. PRE-PROCESSING: Build the Matrices
# Filter Matrix: [M, C*R*S]
filter_matrix = W.reshape(M, -1)

# Toeplitz Input Matrix: [C*R*S, N*H_out*W_out]
CRS = C * R * S
NHW = N * H_out * W_out
input_matrix = np.zeros((CRS, NHW))

col_idx = 0
for n in range(N):
    for h in range(H_out):
        for w in range(W_out):
            # Extract 3D volume (C, R, S) and flatten to 1D column
            patch = I[n, :, h:h+R, w:w+S]
            input_matrix[:, col_idx] = patch.flatten()
            col_idx += 1

print(f"Filter Matrix Shape: {filter_matrix.shape} -> Expected [M, CRS]")
print(f"Input Matrix Shape: {input_matrix.shape} -> Expected [CRS, N * H_out * W_out]")

# 4. EXECUTION: The GEMM
# [M, CRS] @ [CRS, NHW] = [M, NHW]
output_matrix = np.matmul(filter_matrix, input_matrix)
print(f"\nGEMM Output Matrix Shape: {output_matrix.shape}")
print("GEMM Output Matrix:\n", output_matrix)

# 5. POST-PROCESSING: Fold back to 4D Tensor
# We want (N, M, H_out, W_out)
output_reshaped = output_matrix.reshape(M, N, H_out, W_out)
output_tensor = np.transpose(output_reshaped, (1, 0, 2, 3))

print(f"\nFinal 4D Output Tensor Shape: {output_tensor.shape}")

Architectural Summary

By following this exact sequence, modern hardware compilers (like cuDNN or TVM) convert highly complex, deeply nested spatial mathematics into massive, flat memory blocks. The GPU’s arithmetic logic units (ALUs) don’t know what an “image” or a “channel” is—they only know how to multiply rows by columns at lightning speed.

Understanding this translation layer is what separates high-level application developers from true system architects.