Exact Filters

ReLU on the Feature Map

ReLU is an exact clamp after convolution. Negative feature values become zero; positive values would pass through unchanged.

highlighted = computed this step

ReLU is an exact clamp

ReLU keeps positive values and replaces negative values with 0. It is a sign check, so it stays in the exact integer register.

ReLU⁡(z)=max⁡(0,z)\operatorname{ReLU}(z)=\max(0,z)
ReLU on the feature mapNegative cells clamp to zero exactly.before ReLUc1c29-66-9→after ReLUc1c29060negative cells clamp to 0kernel is given; valid padding and stride one; one exact forward pass; head softmax is NAMED;NOT training; NOT learning

Clamp the feature map

The feature map has positive and negative entries. The values 9 and 6 pass through, while -6 and -9 clamp to 0.

ReLU⁡[9−66−9]=[9060]\operatorname{ReLU}\begin{bmatrix}9&-6\\6&-9\end{bmatrix}=\begin{bmatrix}9&0\\6&0\end{bmatrix}
ReLU on the feature mapNegative cells clamp to zero exactly.before ReLUc1c29-66-9→after ReLUc1c29060negative cells clamp to 0kernel is given; valid padding and stride one; one exact forward pass; head softmax is NAMED;NOT training; NOT learning

Summary

This is the same exact ReLU idea used in the neural-net books: a discrete gate, not a transcendental function.

ReLU is exact\text{ReLU is exact}
ReLU on the feature mapNegative cells clamp to zero exactly.before ReLUc1c29-66-9→after ReLUc1c29060negative cells clamp to 0kernel is given; valid padding and stride one; one exact forward pass; head softmax is NAMED;NOT training; NOT learning