I have experiments with LSTMs as well, but I didn't include them in the paper because I couldn't get them to work significantly better than the RNN, which I consider to be a simpler model (they work comparably). That is, a simple RNN with RMSProp, gradient clipping, dropout on non-recurrent connections and careful cross-validation of hyperparameters has so far given me the best results. This might be because sentences are not actually very long structures, so the advantages of LSTM might not be as large. I'm not certain on this point. I'm currently running more extensive experiments with LSTMs and it might turn out that once I tweak everything properly they might work better, but I think it's still valuable to have the RNN numbers as a baseline if nothing else.