Gini Splits

Information Gain

Choosing the Split

Information gain compares candidate questions by exact subtraction. Feature A and Feature B are both rendered so the comparison is visible.

highlighted = computed this step

Information gain

Gain subtracts the weighted child impurity from the parent impurity. Feature A has exact gain 1/18.

gainA=1/18\text{gain}_A=1/18
Compare splitsFeature A and Feature B are scored by exact gain.Feature A splitnodecountsGnoteparent[3, 3]1/2left[2, 1]4/9right[1, 2]4/9weighted4/9weighted childrengain1/18parent - weightedFeature B splitnodecountsGnoteparent[3, 3]1/2left[2, 2]1/2right[1, 1]1/2weighted1/2weighted childrengain0parent - weighted

Why compare exact gains

Feature B leaves weighted impurity at 1/2 and gain 0. The larger exact gain picks the split.

gainA=1/18>0\text{gain}_A=1/18>0
Compare splitsFeature A and Feature B are scored by exact gain.Feature A splitnodecountsGnoteparent[3, 3]1/2left[2, 1]4/9right[1, 2]4/9weighted4/9weighted childrengain1/18parent - weightedFeature B splitnodecountsGnoteparent[3, 3]1/2left[2, 2]1/2right[1, 1]1/2weighted1/2weighted childrengain0parent - weighted

Summary

Feature A wins this exact comparison because 1/18 is greater than 0.

A chosenA\text{ chosen}
Compare splitsFeature A and Feature B are scored by exact gain.Feature A splitnodecountsGnoteparent[3, 3]1/2left[2, 1]4/9right[1, 2]4/9weighted4/9weighted childrengain1/18parent - weightedFeature B splitnodecountsGnoteparent[3, 3]1/2left[2, 2]1/2right[1, 1]1/2weighted1/2weighted childrengain0parent - weighted