Layer normalization (or layer norm) is one of many forms of normalization that layer norm can be used to improve training performance in deep neural networks by keeping the values of a hidden layer in a range that facilitates gradient-based training. Layer norm is a variation of the standard score, or z-score, from statistics applied to a single hidden layer. The first step in layer normalization is to calculate the mean, µ, and standard deviation, σ , over the elements of the vector to be normalized. Given a hidden layer with dimensionality d h , these values are calculated as follows.