text: This paper leverages how deep Bayesian NNs, in the limit of infinite width, are Gaussian processes (GPs).  After characterizing the kernel function, this allows us to use the GP framework for prediction, model selection, uncertainty estimation, etc. - Pros of this work The paper provides a specific method to efficiently compute the covariance matrix of the equivalent GP and shows experimentally on CIFAR and MNIST the benefits of using the this GP as opposed to a finite-width non-Bayesian NN. The provided phase analysis and its relation to the depth of the network is also very interesting. Both are useful contributions as long as deep wide Bayesian NNs are concerned.  A different question is whether that regime is actually useful. - Cons of this work Although this work introduces a new GP covariance function inspired by deep wide NNs,  I am unconvinced of the usefulness of this regime for the cases in which deep learning is useful.  For instance, looking at the experiments, we can see that on MNIST-50k (the one with most data, and therefore, the one that best informs about the "true" underlying NN structure) the inferred depth is 1 for the GP and 2 for the NN, i.e., not deep.  Similarly for CIFAR, where only up to depth 3 is used.  None of these results beat state-of-the-art deep NNs. Also, the results about the phase structure show how increased depth makes the parameter regime in which these networks work more and more constrained.  In [1], it is argued that kernel machines with fixed kernels do not learn a hierarchical representation.  And such representation is generally regarded as essential for the success of deep learning.  My impression is that the present line of work will not be relevant for deep learning and will not beat state-of-the-art results  because of the lack of a structured prior.  In that sense, to me this work is more of a negative result informing that to be successful, deep Bayesian NNs should not be wide and should have more structure to avoid reaching the GP regime. - Other comments:In Fig. 5, use a consistent naming for the axes (bias and variances). In Fig. 1, I didn't find the meaning of the acronym NN with no specified width. Does the unit norm normalization used to construct the covariance disallow ARD input selection? [1] Yoshua Bengio, Olivier Delalleau, and Nicolas Le Roux. The Curse of Dimensionality for Local Kernel Machines. 2005.