Backpropagation, Exactly

The Backward Pass

• The Gradient Question
• Backward from the Loss
• ReLU's Derivative is 0 or 1
• Gradients for the Weights
• The Gradient-Annotated Graph
• What One Backprop Step Is and Is Not