To solve the problems of long registration time and many wrong registrations in using the infrared image stitching algorithm. This paper improves the infrared image stitching algorithm which based on SIFT algorithm. Firstly, the improved algorithm uses the Gaussian difference pyramid to establish the scale space. Secondly, this algorithm uses FAST algorithm to extract the feature points from the Gaussian pyramid difference images. In order to find the initial matching point pairs, the method of this paper calculates the feature vectors distance to measure the similarity of feature points. Thirdly, RANSAC algorithm and direction consistency method are used to remove the wrong matching point pairs. Finally, this algorithm uses weighted average method to mosaic images. The result of the infrared image mosaic experiments shows that the improved method is more stable and more rapid.