text: This paper proposes another entropic regularization term for deep neural nets. The key idea can be stated as follows: Let X denote the observed input, C the hidden class label taking values in a finite set, and Y the representation computed by a neural net. Then C -> X -> Y is a Markov chain. Moreover, assuming that the mapping X -> Y is deterministic (as is the case with neural nets or any other deterministic representations), we can write down the mutual information between X and Y as I(X;Y) = H(Y) - H(Y|X) = H(Y). A simple manipulation shows that H(Y) = I(C;Y) + H(Y|C). The authors interpret the first term, I(C;Y), as a data fit term that quantifies the statistical correlations between the class label C and the representation Y, whereas the second term, H(Y|C), is the amount by which the representation Y can be compressed knowing the class label C. The authors then propose to 'explicitly decouple' the data-fit term I(C;Y) from the regularization penalty and focus on minimizing H(Y|C). In fact, they replace this term by the sum of conditional entropies of the form H(Y_{i,k}|C), where Y_{i,k} is the activation of the ith neuron in the kth layer of the neural net. The final step is to recognize that the conditional entropy may not admit a scalable and differentiable estimator, so they use the relation between a quantity called entropy power and second moments to replace the entropic penalty with the conditional variance penalty Var[Y_{i,k}|C]. Since the class-conditional distributions are unknown, a surrogate model Q_{Y|C} is used. The authors present some experimental results as well. However, this approach has a number of serious flaws. First of all, if the distribution of X is nonatomic and the mapping X -> Y is continuous (in the case of neural nets, it is even Lipschitz), then the mutual information I(X;Y) is infinite. In that case, the representation of I(X;Y) in terms of entropies is not valid -- indeed, one can write the mutual information between two jointly distributed random variables X and Y in terms of differential entropies as I(X;Y) = h(Y) - h(Y|X), but this is possible only if both terms on the right-hand side exist. This is not the case here, so, in particular, one cannot relate I(X;Y) to I(C;Y). Ironically, I(C;Y) is finite, because C takes values in a finite set, so I(C;Y) is at most the log cardinality of the set of labels. One can start, then, simply with I(C;Y) and express it as H(C) - H(C|Y). Both terms are well-defined Shannon entropies, where the first one does not depend on the representation, whereas the second one involves the representation. But then, if the goal is to _minimize_ the mutual information between I(C;Y), it makes sense to _maximize_ the conditional entropy H(C|Y). In short, the line of reasoning that leads to minimizing H(Y|C) is not convincing. Moreover, why is it a good idea to _minimize_ I(C;Y) in the first place? Shouldn't one aim to maximize it subject to structural constraints on the representation, along the lines of InfoMax? The next issue is the chain of reasoning that leads to replacing H(Y|C) with Var[Y|C]. One could start with that instead without changing the essence of the approach, but then the magic words "Shannon decay" would have to disappear altogether, and the proposed method would lose all of its appeal.