Traditional image fusion algorithms have many shortcomings, such as high computational complexity and inability to effectively extract image texture. To compensate these shortcomings of above traditional algorithms, an image fusion method is proposed based on the Siamese Convolutional Neural Network (Siamese CNN). First, we use the Siamese CNN to generate a weight graph, which contains all pixel information from the two images to be fused. Then, the image pyramid is fused in a multi-scale way, and the local similarity strategy is adopted to adjust the decomposition coefficient adaptively. Finally, several existing image fusion methods are compared. Experimental results show that the proposed method has sound fusion effect and is practical to some extent.