Convolutional Neural Networks: Discrete Spatial Convolutions and Feature Extraction

Why Dense Layers Fail on Two Dimensional Images

Before convolutional networks became widespread, computer vision attempted to process images using standard multilayer perceptrons (dense fully connected neural networks). In this approach, a 2D image matrix of height $H$, width $W$, and channels $C$ was flattened into a single one dimensional vector of length $H \times W \times C$.

This naive transformation introduces two critical design flaws:

1. Destruction of spatial topology: In images, adjacent pixels carry strong semantic correlations. A pixel at coordinate $(x, y)$ is intimately related to its neighbor at $(x+1, y)$ and $(x, y+1)$. Flattening an image into a 1D vector spreads vertical neighbors hundreds or thousands of elements apart, forcing the network to relearn basic 2D geometric concepts from scratch.
2. Explosion of trainable parameters: Consider a modest color image of resolution $1000 \times 1000$ pixels with 3 color channels. Flattening produces a vector of $3,000,000$ values. Connecting this input to a single hidden layer with 1,000 neurons requires $3,000,000 \times 1,000 = 3 \text{ billion}$ weight parameters. Such models overfit instantly and exceed modern memory capacities.

Convolutional Neural Networks (CNNs) resolve these issues through two fundamental inductive biases: local receptive fields and weight sharing.

The Mathematical Mechanics of 2D Discrete Convolution

In continuous mathematics, convolution blends two functions. In deep learning and digital image processing, we work with discrete matrices where the operation is technically a cross correlation:

$$S(i, j) = (I * K)(i, j) = \sum_{m=0}^{k_h - 1} \sum_{n=0}^{k_w - 1} I(i + m, j + n) \cdot K(m, n) + b$$

Where:

• $I$ is the input image or incoming feature map of dimensions $(H, W)$.
• $K$ is the learnable kernel (filter) matrix of spatial dimensions $(k_h, k_w)$, typically $3 \times 3$ or $5 \times 5$.
• $b$ is an optional scalar bias term.
• $S(i, j)$ is the output scalar at coordinate $(i, j)$ in the resulting activation feature map.

The Frobenius Inner Product

At each spatial coordinate $(i, j)$, the kernel slides over a matching submatrix of the input image called the local receptive field. The convolution operation computes the Frobenius inner product between this local patch and the kernel filter:

$$\langle A, B \rangle_F = \sum_{m} \sum_{n} A_{m,n} \cdot B_{m,n}$$

Consider an example local image patch and a $3 \times 3$ vertical edge detection kernel:

$$\text{Image Patch} = \begin{bmatrix} 120 & 130 & 40 \ 115 & 125 & 35 \ 110 & 120 & 30 \end{bmatrix}, \quad \text{Sobel Kernel} = \begin{bmatrix} -1 & 0 & 1 \ -2 & 0 & 2 \ -1 & 0 & 1 \end{bmatrix}$$

We multiply corresponding elements and calculate their sum:

$$\begin{aligned} \text{Output} &= (120 \times -1) + (130 \times 0) + (40 \times 1) \ &+ (115 \times -2) + (125 \times 0) + (35 \times 2) \ &+ (110 \times -1) + (120 \times 0) + (30 \times 1) \ &= -120 + 0 + 40 - 230 + 0 + 70 - 110 + 0 + 30 = -320 \end{aligned}$$

Because the left side of the patch has high brightness (~120) and the right side has low brightness (~35), the filter produces a strong negative response, identifying a prominent vertical boundary transition.

Spatial Dimensions: Padding and Stride

The dimensions of an output feature map depend on four parameters: input size $W$, kernel size $K$, padding $P$, and stride $S$:

$$\text{Output Width} = \left\lfloor \frac{W - K + 2P}{S} \right\rfloor + 1$$

Stride ($S$)

Stride determines the step size in pixels by which the kernel shifts after each dot product. A stride of 1 slides the filter pixel by pixel. A stride of 2 skips every alternate pixel, halving both spatial dimensions and downsampling the representation.

Padding ($P$)

Without padding, sliding a $3 \times 3$ kernel across an $N \times N$ image loses 1 pixel around every border, producing an output of size $(N - 2) \times (N - 2)$. After ten consecutive convolution layers, the image would shrink to nothing.

Adding zero padding of $P = 1$ around all borders preserves the original spatial dimensions ($W_{\text{out}} = W_{\text{in}}$). This is commonly termed Same Padding.

Common Spatial Kernels in Computer Vision

Before modern networks learned kernel weights automatically through backpropagation, engineers designed spatial filters by hand:

• Identity Filter: Leaves the input unchanged. Center pixel has weight 1.0, all other pixels zero.
• Ridge and Edge Detection: Highlights intensity discontinuities. A center value of +4 surrounded by four -1 values acts as a discrete Laplacian operator.
• Sobel Filters: Approximates the directional gradient of image intensity. Sobel Horizontal detects horizontal contours, while Sobel Vertical detects vertical structures.
• Gaussian Blur: Applies a weighted distance average to soften noise and suppress high frequency pixel artifacts.
• Sharpening: Emphasizes high frequency transitions by subtracting neighboring blurred values from the central pixel.

In deep neural networks, early layers automatically converge on these same edge filters during training, discovering that edge boundaries are the most reliable basic primitives for understanding the visual world.

Downsampling with Pooling Layers

While convolutions extract local patterns, vision networks must also achieve translational invariance and reduce spatial dimensions. This is achieved using pooling layers.

Max Pooling

Max pooling slides a window (commonly $2 \times 2$ with stride 2) across the feature map and selects the single maximum activation value in each quadrant:

$$\begin{bmatrix} 12 & 25 & 80 & 14 \ 9 & 44 & 75 & 33 \ 60 & 70 & 5 & 10 \ 55 & 85 & 8 & 12 \end{bmatrix} \xrightarrow{\text{Max Pool } 2 \times 2} \begin{bmatrix} 44 & 80 \ 85 & 12 \end{bmatrix}$$

Max pooling provides two advantages:

1. Translational robustness: If a cat ear or wheel shifts slightly by 1 or 2 pixels, the maximum activation in the region remains identical.
2. Computational savings: Each pooling step reduces the total number of activations by 75%, allowing subsequent layers to increase their filter count without exceeding memory limits.

The Hierarchical Feature Pyramid

Deep vision networks (such as ResNet, VGG, or EfficientNet) arrange convolutions in a hierarchical pyramid:

• Layer 1 to 2 (Low Level Features): Tiny receptive fields (3x3 pixels). Kernels detect localized edges, corners, color gradients, and line orientations.
• Layer 3 to 5 (Mid Level Features): Through repeated pooling and stride, the effective receptive field expands to 20x20 pixels. Kernels detect shapes, circular textures, mesh grids, and basic contours.
• Layer 6+ (High Level Semantics): Receptive fields cover the entire input scene. Kernels activate for whole semantic objects such as dog snouts, car wheels, clock faces, and letters.

Finally, global average pooling flattens these high level feature maps into a vector for a final linear layer that outputs class probabilities.

Summary and Practical Takeaway

Discrete spatial convolutions allow neural networks to analyze images with parameter efficiency and spatial awareness. By replacing dense connections with localized filters and weight sharing, CNNs provide the architectural foundation for computer vision.

You can inspect sliding window convolutions, switch between 12 preset edge kernels, and watch real time Frobenius math breakdowns on MNIST digits in our interactive CNN Convolution Lab.