Abstract:Previous studies proved that, adding part of speech tag information to the input layer of neural language model, can improve the performance significantly. But part of speech tag need hand-annotated data to train the tag model, which consumes a lot and the extra tagger also makes the model more complicated. To solve the problem, this article propose adding the results of brown clustering, instead of part of speech tag information to the input layer of the recurrent network language model. In the Penn Treebank corpus, the relative improvement over the original recurrent neural network language model reaches 8%~9%.