Light-weight image fusion algorithm is very important for human eye observation and machine recognition. By studying the importance of visual saliency in infrared and visible image fusion, a visual saliency map (VSM)-guided MSDNet fusion network is optimized and designed based on the SDNet fusion network. Firstly, the structure and channel numbers of SDNet are reduced to accelerate training and inference speed, and the learning ability of the light-weight model is enhanced by structural parameterization and reverse parameterization techniques. Then, for model training, the loss function guided by VSM is used to achieve model self-supervised training. Finally, at the end of the training, the image reconstruction branch is deleted. So the final light-weight model is obtained by the fusion of convolution parameters. Experiments show that the light-weight network can not only ensure image fusion quality but greatly improve the speed, making its porting in mobile terminals possible.