Backpropagation, Exactly

The Backward Pass

• The Gradient Question
• Backward from the Loss
• ReLU's Derivative is 0 or 1
• Same Weights, Different Gate
• Gradients for the Weights
• One Weight Gradient Product
• One Weight Update
• Check the Updated Forward Pass
• The Gradient-Annotated Graph
• What One Backprop Step Is and Is Not