In order to study the correlation between infrared and visible image fusion quality evaluation indexes, the similarity of the fusion quality evaluation indexes of infrared and visible images based on cluster analysis is put forward. In this paper, eleven evaluation indexes are listed, and correlation matrix are respectively established by using Spearman rank correlation coefficient and grey correlation degree. Through the analysis of change rate of threshold, the optimal clustering threshold is selected and the evaluation index of the degree of similarity is given. In the experiment, 10 groups of commonly used infrared and visible fusion images are respectively evaluated by 11 evaluation indicators, and then 11 evaluation indicators are divided into 6 categories by cluster analysis method. The results can be used as the basis for selecting a reasonable objective evaluation index set.