Various forms of regularization are used to prevent overfitting. One of the most important is dropout: randomly dropping some units and their connections from dropout the network during training (Hinton et al. 2012 , Srivastava et al. 2014 . Tuning of hyperparameters is also important. The parameters of a neural network are the hyperparameter weights W and biases b; those are learned by gradient descent. The hyperparameters are things that are chosen by the algorithm designer; optimal values are tuned on a devset rather than by gradient descent learning on the training set. Hyperparameters include the learning rate η, the mini-batch size, the model architecture (the number of layers, the number of hidden nodes per layer, the choice of activation functions), how to regularize, and so on. Gradient descent itself also has many architectural variants such as Adam (Kingma and Ba, 2015) .