Illustration of progress and generalization loss measures stopping an optimization early as described in [46].

A) Sample normalized training and validation costs during towards the end of an optimization. The training is slow, but steadily declining, while the validation cost is changing erratically at various optimization epochs. B) Measures of progress, generalization loss, and their ratio, Q, over the optimization time period as in A. Progress quantifies how much the average training cost is larger than the minimum cost seen in last k optimization iterations. Generalization loss quantifies how much larger the current validation cost is compared to the minimum validation cost seen across all iterations seen so far. In the example shown, progress (green dashes) stays relatively steady at various epochs reflecting the slow steady decline in training cost. Generalization loss widely fluctuates along with the validation error (red dashes). The ratio of the progress and generalization loss, Q (purple), steadily increases three simulation epochs in a row (as indicated by the black arrows), which signals that early stopping should occur.

(TIF)