Keras Early Stopping
I'm training neural network for my project using Keras. Keras has provided a function for early stopping. May I know what parameters should be observed to avoid my neural network from overfitting by using early stopping?
Early stopping is basically stopping the training once your loss starts to increase (or in other words validation accuracy starts to decrease). According to documents it is used as follows;
keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0, patience=0, verbose=0, mode='auto')
Values depends on your implementation (problem, batch size etc...) but generally to prevent overfitting I would use;
- Monitor the validation loss (need to use cross validation or at least train/test sets) by setting the
min_deltais a threshold to whether quantify a loss at some epoch as improvement or not. If the difference of loss is below
min_delta, it is quantified as no improvement. Better to leave it as 0 since we're interested in when loss becomes worse.
patienceargument represents the number of epochs before stopping once your loss starts to increase (stops improving). This depends on your implementation, if you use very small batches or a large learning rate your loss zig-zag (accuracy will be more noisy) so better set a large
patienceargument. If you use large batches and a small learning rate your loss will be smoother so you can use a smaller
patienceargument. Either way I'll leave it as 2 so I would give the model more chance.
verbosedecides what to print, leave it at default (0).
modeargument depends on what direction your monitored quantity has (is it supposed to be decreasing or increasing), since we monitor the loss, we can use
min. But let's leave keras handle that for us and set that to
So I would use something like this and experiment by plotting the error loss with and without early stopping.
keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0, patience=2, verbose=0, mode='auto')
For possible ambiguity on how callbacks work, I'll try to explain more. Once you call
fit(... callbacks=[es]) on your model, Keras calls given callback objects predetermined functions. These functions can be called
on_batch_end. Early stopping callback is called on every epoch end, compares the best monitored value with the current one and stops if conditions are met (how many epochs have past since the observation of the best monitored value and is it more than patience argument, the difference between last value is bigger than min_delta etc..).
As pointed by @BrentFaust in comments, model's training will continue until either Early Stopping conditions are met or
epochs parameter (default=10) in
fit() is satisfied. Setting an Early Stopping callback will not make the model to train beyond its
epochs parameter. So calling
fit() function with a larger
epochs value would benefit more from Early Stopping callback.