In order to solve the problem of low accuracy and speed of traditional SURF (Speeded-Up Robust Features) algorithm, an optimized image matching algorithm is proposed. A local two-dimensional entropy is introduced to characterize the uniqueness of feature points in the stage of feature point extracting. Some error points are eliminated by calculating the local two-dimensional entropy of the feature points and setting appropriate thresholds. The Euclidean distance is replaced by the Manhattan distance during the matching phase. The concept of the nearest neighbors and nearer neighbors is introduced. The first two points are extracted with the closest Manhattan points between the feature points in the template image and one in the image to be matched. If the ratio obtained by dividing the nearest distance by nearer distance is less than the threshold T, then this pair of matches is accepted to reduce mismatches. The experimental results show that the algorithm is superior to the traditional algorithm, and the accuracy is improved while the speed is also improved.