text: Clearly presented paper, including a number of reasonable techniques to improve LSTM-LMs. The proposed techniques are heuristic, but are reasonable and appear to yield improvements in perplexity. Some specific comments follow. re. "ASGD" for Averaged SGD: ASGD usually stands for Asynchronous SGD, have the authors considered an alternative acronym? AvSGD? re. Optimization criterion on page 2, note that SGD is usually taken to minimizing expected loss, not just empirical loss (Bottou thesis 1991). Is there any theoretical analysis of convergence for Averaged SGD? re. paragraph starting with "To prevent such inefficient data usage, we randomly select the sequence length for the forward and backward pass in two steps": the explanation is a bit unclear. What is the "base sequence length" exactly? Also, re. the motivation above this paragraph, I'm not sure what "elements" really refers to, though I can guess. What is the number of training tokens of the datasets used, PTB and WT2? Can the authors provide more explanation for what "neural cache models" are, and how they relate to "pointer models"? Why do the sections "Pointer models", "Ablation analysis", and "AWD-QRNN" come after the Experiments section?