These kinds of regularization come from statistics, where L1 regularization is called lasso regression (Tibshirani, 1996) and L2 regularization is called ridge regression, lasso ridge and both are commonly used in language processing. L2 regularization is easier to optimize because of its simple derivative (the derivative of θ 2 is just 2θ ), while L1 regularization is more complex (the derivative of |θ | is non-continuous at zero).