An Empirical Study of Memorization in NLP

A recent study by Feldman (2020) proposed a <PRE_TAG>long-tail theory</POST_TAG> to explain the <PRE_TAG>memorization</POST_TAG> behavior of deep learning models. However, <PRE_TAG>memorization</POST_TAG> has not been empirically verified in the context of NLP, a gap addressed by this work. In this paper, we use three different NLP tasks to check if the long-tail theory holds. Our experiments demonstrate that top-ranked memorized training instances are likely atypical, and removing the top-memorized training instances leads to a more serious drop in <PRE_TAG>test accuracy</POST_TAG> compared with removing training instances randomly. Furthermore, we develop an attribution method to better understand why a training instance is memorized. We empirically show that our <PRE_TAG><PRE_TAG><PRE_TAG>memorization</POST_TAG> attribution method</POST_TAG></POST_TAG> is faithful, and share our interesting finding that the top-memorized parts of a training instance tend to be features negatively correlated with the class label.