Pyramid Detail Fusion Network for SAR and Optical Image Based on Dual Non-local Attention Perception
CSTR:
Author:
  • Article
  • | |
  • Metrics
  • |
  • Reference [25]
  • |
  • Related [20]
  • | | |
  • Comments
    Abstract:

    Synthetic aperture radar (SAR) and optical image fusion aim to leverage the imaging complementarity of satellite sensors for generating more comprehensive geomorphological information. However, existing network models often exhibit low imaging accuracy during the fusion process due to the heterogeneity in data distribution of each single satellite sensor and differences in imaging physical mechanisms. This study proposes the DNAP-Fusion, a novel SAR and optical image fusion network that incorporates dual non-local attention perception. The proposed method utilizes a dual non-local perceptual attention module to extract structural information from SAR images and texture details from optical images within a multi-level image pyramid with a gradually decreasing spatial scale. It then fuses their complementary features in both spatial and channel dimensions. Subsequently, the fused features are injected into the upsampled optical image through image reconstruction, resulting in the final fusion outcome. Additionally, before network training, image encapsulation decisions are employed to enhance the commonality between objects in SAR and optical images within the same scene. Qualitative and quantitative experimental results demonstrate that the proposed method outperforms state-of-the-art (SOTA) multisensor fusion methods. The correlation coefficient (CC) in the objective evaluation indices is 0.990 6, and the peak signal to noise ratio (PSNR) is 32.156 0 dB. Moreover, the proposed method effectively fuses the complementary features of SAR and optical images, offering a valuable idea and method for enhancing the accuracy and effectiveness of remote sensing image fusion.

    Reference
    [1] 金益如, 杨学志, 董张玉, 等. 一种NSST与稀疏表示相结合的遥感图像融合算法. 地理与地理信息科学, 2016, 32(2): 60–66.
    [2] 邰建豪, 潘斌, 赵珊珊, 等. 基于Shearlet变换的SAR与多光谱遥感影像融合. 武汉大学学报(信息科学版), 2017, 42(4): 468–474.
    [3] 童涛, 杨桄, 谭海峰, 等. 基于NSCT变换的多传感器图像融合算法. 地理与地理信息科学, 2013, 29(2): 22–25, 59.
    [4] 易维, 曾湧, 原征. 基于NSCT变换的高分三号SAR与光学图像融合. 光学学报, 2018, 38(11): 1110002.
    [5] Zhang Q, Liu Y, Blum RS, et al. Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: A review. Information Fusion, 2018, 40: 57–75.
    [6] Shah VP, Younan NH, King RL. An efficient pan-sharpening method via a combined adaptive PCA approach and contourlets. IEEE Transactions on Geoscience and Remote Sensing, 2008, 46(5): 1323–1335.
    [7] Yang Y, Zhang YM, Huang SY, et al. Infrared and visible image fusion using visual saliency sparse representation and detail injection model. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 5001715.
    [8] Masi G, Cozzolino D, Verdoliva L, et al. Pansharpening by convolutional neural networks. Remote Sensing, 2016, 8(7): 594.
    [9] Yang JF, Fu XY, Hu YW, et al. PanNet: A deep network architecture for pan-sharpening. Proceedings of the 2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017. 1753–1761.
    [10] Liu L, Wang J, Zhang EL, et al. Shallow-deep convolutional network and spectral-discrimination-based detail injection for multispectral imagery pan-sharpening. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13: 1772–1783.
    [11] Chen YX, Bruzzone L. Self-supervised SAR-optical data fusion of Sentinel-1/-2 images. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5406011.
    [12] Jin C, Deng LJ, Huang TZ, et al. Laplacian pyramid networks: A new approach for multispectral pansharpening. Information Fusion, 2022, 78: 158–170.
    [13] 吴佼华, 杨学志, 方帅, 等. 基于双分支卷积神经网络的SAR与多光谱图像融合实验. 地理与地理信息科学, 2021, 37(2): 22–30.
    [14] Ye YX, Liu WC, Zhou L, et al. An unsupervised SAR and optical image fusion network based on structure-texture decomposition. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 4028305.
    [15] Zhang YZ, Zhang HS, Lin H. Improving the impervious surface estimation with combined use of optical and SAR remote sensing images. Remote Sensing of Environment, 2014, 141: 155–167.
    [16] Fang FM, Li JC, Yuan YT, et al. Multilevel edge features guided network for image denoising. IEEE Transactions on Neural Networks and Learning Systems, 2021, 32(9): 3956–3970.
    [17] Wang XL, Girshick R, Gupta A, et al. Non-local neural networks. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018. 7794–7803.
    [18] 范艺华, 董张玉, 杨学志. 结合上下文编码与特征融合的SAR图像分割. 中国图象图形学报, 2022, 27(8): 2527–2536.
    [19] Deng LJ, Vivone G, Jin C, et al. Detail injection-based deep convolutional neural networks for pansharpening. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(8): 6995–7010.
    [20] 董张玉, 许道礼, 张晋, 等. 基于双分支多尺度残差融合嵌套的SAR和多光谱图像融合架构与实验. 地理与地理信息科学, 2023, 39(1): 23–30.
    [21] Wald L, Ranchin T, Mangolini M. Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images. Photogrammetric Engineering and Remote Sensing, 1997, 63(6): 691–699.
    [22] 盛佳佳, 杨学志, 董张玉, 等. 基于NSST-IHS变换稀疏表示的SAR与可见光图像融合. 图学学报, 2018, 39(2): 201–208.
    [23] Shao ZF, Cai JJ. Remote sensing image fusion with deep convolutional neural network. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(5): 1656–1669.
    [24] Li H, Wu XJ, Kittler J. RFN-Nest: An end-to-end residual fusion network for infrared and visible images. Information Fusion, 2021, 73: 72–86.
    [25] Cao Y, Xu JR, Lin S, et al. Global context networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(6): 6881–6895.
    Cited by
    Comments
    Comments
    分享到微博
    Submit
Get Citation

朱佳佳,杨学志,梁宏博,杨翔宇.结合双非局部注意力感知的SAR和光学图像金字塔细节融合网络.计算机系统应用,2024,33(8):155-165

Copy
Share
Article Metrics
  • Abstract:312
  • PDF: 792
  • HTML: 443
  • Cited by: 0
History
  • Received:December 15,2023
  • Revised:January 17,2024
  • Online: June 28,2024
Article QR Code
You are the first990420Visitors
Copyright: Institute of Software, Chinese Academy of Sciences Beijing ICP No. 05046678-3
Address:4# South Fourth Street, Zhongguancun,Haidian, Beijing,Postal Code:100190
Phone:010-62661041 Fax: Email:csa (a) iscas.ac.cn
Technical Support:Beijing Qinyun Technology Development Co., Ltd.

Beijing Public Network Security No. 11040202500063