• Neural networks are built out of neural units, originally inspired by human neurons but now simply an abstract computational device. • Each neural unit multiplies input values by a weight vector, adds a bias, and then applies a non-linear activation function like sigmoid, tanh, or rectified linear unit. • In a fully-connected, feedforward network, each unit in layer i is connected to each unit in layer i + 1, and there are no cycles. • The power of neural networks comes from the ability of early layers to learn representations that can be utilized by later layers in the network. • Neural networks are trained by optimization algorithms like gradient descent. • Error backpropagation, backward differentiation on a computation graph, is used to compute the gradients of the loss function for a network. • Neural language models use a neural network as a probabilistic classifier, to compute the probability of the next word given the previous n words. • Neural language models can use pretrained embeddings, or can learn embeddings from scratch in the process of language modeling.