The recognition rate of the Speaker Recognition System under ideal condition can reach more than 90%, but the recognition rate will decrease rapidly under the traffic environment. In this paper, we discuss the robust speaker recognition under the channel mismatch environment. First, a speaker recognition system based on Gaussian mixture model(GMM) is set up. Then, two methods of improvement are put forward through testing and analysis of the actual communication channel. One is that we establish a general channel model to filter the clean speech which is regarded as the training speech. The other is that we put forward to abandon the first and second dimensional characteristic parameters through comparing the Mel-frequency cepstral coefficient of the measured channel, the ideal low-pass channel and the common speech. The experimental results show that the recognition rate under the channel mismatch environment is improved by 20% after processing and 9%-12% comparing with the traditional Cepstral Mean Subtraction(CMS).