Named Log Loss

Soft Targets Weight Terms

A soft target keeps exact rational target weights. The cross-entropy sum becomes a weighted sum of named log terms instead of one selected term.

highlighted = computed this step

A soft target has weights

Instead of a one-hot target, this displayed target is y=(0,3/4,1/4). The weights are exact and sum to 1.

y=(0,3/4,1/4)y=(0,3/4,1/4)
Soft Targets Weight TermsExact soft target weights make a weighted sum of named log terms.soft target weights named log termsclasstarget weightp_iweighted named termA01/40 (target weight drop)B3/41/23/4*-log(1/2)C1/41/41/4*-log(1/4)H=3/4*-log(1/2)+1/4*-log(1/4)soft target weights + named log boundaryone displayed target distribution; one displayed probability distributionNOT training; NOT learning; NOT a softmax computation; NOT generalization

Weights multiply named log terms

Class A has target weight 0, so it drops. Class B contributes 3/4*-log(1/2) and class C contributes 1/4*-log(1/4).

3/4⋅[−log⁡(1/2)]  +  1/4⋅[−log⁡(1/4)]3/4\cdot[-\log(1/2)]\;+\;1/4\cdot[-\log(1/4)]
Soft Targets Weight TermsExact soft target weights make a weighted sum of named log terms.soft target weights named log termsclasstarget weightp_iweighted named termA01/40 (target weight drop)B3/41/23/4*-log(1/2)C1/41/41/4*-log(1/4)H=3/4*-log(1/2)+1/4*-log(1/4)soft target weights + named log boundaryone displayed target distribution; one displayed probability distributionNOT training; NOT learning; NOT a softmax computation; NOT generalization

Add the symbolic terms

The total is the symbolic weighted sum 3/4*-log(1/2) + 1/4*-log(1/4). No log decimal is evaluated.

H=3/4⋅[−log⁡(1/2)]  +  1/4⋅[−log⁡(1/4)]H=3/4\cdot[-\log(1/2)]\;+\;1/4\cdot[-\log(1/4)]
Soft Targets Weight TermsExact soft target weights make a weighted sum of named log terms.soft target weights named log termsclasstarget weightp_iweighted named termA01/40 (target weight drop)B3/41/23/4*-log(1/2)C1/41/41/4*-log(1/4)H=3/4*-log(1/2)+1/4*-log(1/4)soft target weights + named log boundaryone displayed target distribution; one displayed probability distributionNOT training; NOT learning; NOT a softmax computation; NOT generalization

Boundary

This is one displayed target distribution and one displayed probability distribution. It is NOT training, NOT learning, NOT a softmax computation, and NOT generalization.

exact weights; named log terms\text{exact weights; named log terms}
Soft Targets Weight TermsExact soft target weights make a weighted sum of named log terms.soft target weights named log termsclasstarget weightp_iweighted named termA01/40 (target weight drop)B3/41/23/4*-log(1/2)C1/41/41/4*-log(1/4)H=3/4*-log(1/2)+1/4*-log(1/4)soft target weights + named log boundaryone displayed target distribution; one displayed probability distributionNOT training; NOT learning; NOT a softmax computation; NOT generalization