This study proposes an infrared and visible image fusion method based on latent low-rank representation and guided filtering to address the serious detail loss and the poor visual quality in the fusion. First of all, the source image is decomposed by latent low-rank representation into low-rank layers and salient layers. Then the low-rank layers are decomposed by guided filtering into basic layers and structural layers with the aim of extracting more structural information from low-rank layers. According to the characteristics of basic layers, structural layers, and salient layers, visual saliency weighting, gradient saliency weighting, and absolute maximum selection are used as fusion rules, respectively. In particular, since the initial weight is noisy and unaligned with the object boundary, it is optimized by guided filtering. Finally, the basic fusion layer, the structural fusion layer, and the salient fusion layer are overlapped to yield the fused image. The subjective and objective evaluation results of several groups of fused images are compared. The proposed method is found able to effectively extract the detail information of source images and superior to other image fusion methods in terms of visual quality and objective evaluation.