The SIFT(Scale Invariant Feature Transform) algorithm is now regarded as the best local feature extraction and matching algorithm. However, in the traditional SIFT algorithm, the Euclidean distance which could not change the high-dimensional feature vector into a low-dimensional geometry structure is used to measure the SSD(Sum of Square Differences) between two image features to match and results into mismatching. To overcome the shortcoming, an SIFT matching algorithm based on diffusion distance is proposed in this paper which replaces the Euclidean distance with the diffusion one. At the same time, RANSAC(Random Sample Consensus) is presented to exclude the mismatching points. Experimental results show that the proposed algorithm has more efficiency to deal with image deformation, illumination change and image noise than the traditional one.