The Speaker Recognition System is significantly affected by the Multi-Sound sources problem. In order to overcome this problem, a target sound extraction algorithm named time-frequency masking is proposed. The proposed algorithm is based on the sound source azimuth information and the approximate sparse nature of sound. A Mel-frequency cepstral coefficient (MFCC) based Gaussian mixture model (GMM) speaker recognition system is presented to improve the recognition robustness. The proposed algorithm has been tested on the simulated data through a number of experiments which shows the efficiency and robustness of the proposed algorithm. In the Multi-Sound sources environment, the recognition rate of the proposed algorithm can be improved by about 25%.