At present, the voiceprint recognition system has achieved high identification precision. But with the increase of the number of target speakers, general system has difficulty in satisfying the need of real time. Therefore, a two-layer recognition model is raised in this paper. The first layer based on VQ-VPT model quickly sorts out K target speakers' voiceprint models which are most similar to the speaker's voiceprint characteristics. In the second layer, the GMM-UBM model matches the K voiceprint models to make a final judgment. Via experimental verification, under the premise of ensuring high recognition accuracy, the two-layer recognition model has greatly improved the recognition speed of the system.