text: This paper proposed a new parametrization scheme for weight matrices in neural network based on the Householder  reflectors to solve the gradient vanishing and exploding problems in training.  The proposed method improved two previous papers: 1) stronger expressive power than Mahammedi et al. (2017), 2) faster gradient update than Vorontsov et al. (2017). The proposed parametrization scheme is natrual from numerical linear algebra point of view and authors did a good job in Section 3 in explaining the corresponding expressive power.  The experimental results also look promising.  It would be nice if the authors can analyze the spectral properties of the saddle points in linear RNN (nonlinear is better but it's too difficult I believe).  If the authors can show the strict saddle properties then as a corollary, (stochastic) gradient descent finds a global minimum.  Overall this is a strong paper  and I recommend to accept.