AIPUB归智期刊联盟

微信公众号

网站二维码

2025年4月17日 6:00 星期四
首页 > 过刊浏览>2022年第31卷第7期 >23-34. DOI:10.15888/j.cnki.csa.008566
PDF HTML阅读 XML下载 导出引用 引用提醒
基于深度学习的单图像超分辨率重建综述
作者:
基金项目:

国家自然科学基金(62172093)


Survey on Single Image Super-resolution Reconstruction Based on Deep Learning
Author:
  • 摘要
    • | |
  • 访问统计
    • |
  • 参考文献 [71]
    • |
  • 相似文献 [20]
    • | | |
  • 文章评论
    摘要:

    图像超分辨率重建是用于提高图像质量的一项重要技术, 得益于深度学习在计算机视觉领域的成功应用和快速发展, 单图像超分辨率重建的效果得到了显著提升. 因此, 本文针对基于深度学习的单图像超分辨率重建方法展开深入研究, 首先综合介绍了用于该领域的基准数据集、性能评价指标、损失函数等相关知识, 然后对有监督学习和无监督学习下单图像超分辨率重建技术的最新算法进行分类讨论, 并且比较分析了不同模型之间的异同点与优缺点, 最后对该领域面临的问题和未来的发展方向进行了总结与展望.

    Abstract:

    Image super-resolution reconstruction is an important technique to improve image quality. Thanks to the successful application and rapid development of deep learning in the field of computer vision, significant improvement in single image super-resolution (SISR) reconstruction has been achieved. In response, this study explores SISR reconstruction methods based on deep learning in depth. Relevant background knowledge such as benchmark data sets, performance evaluation indexes, and the loss function used in this field are outlined. Then, the latest algorithms for SISR reconstruction techniques with supervised and unsupervised learning are discussed respectively, and the differences and similarities among different models as well as their advantages and disadvantages are compared. Finally, the existing problems in this field are summarized, and future trends are proposed.

    参考文献
    [1] 唐艳秋, 潘泓, 朱亚平, 等. 图像超分辨率重建研究综述. 电子学报, 2020, 48(7): 1407–1420. [doi: 10.3969/j.issn.0372-2112.2020.07.022
    [2] 刘朋伟, 高媛, 秦品乐, 等. 基于多感受野的生成对抗网络医学MRI影像超分辨率重建. 计算机应用. http://kns.cnki.net/kcms/detail/51.1307.TP.20210708.1626.008.html. [2021-10-07].
    [3] 蒋文杰, 罗晓曙, 戴沁璇. 基于对抗网络人脸超分辨率重建算法研究. 计算机工程与应用, 2021, 57(11): 219–223. [doi: 10.3778/j.issn.1002-8331.2002-0370
    [4] Kappeler A, Yoo S, Dai QQ, et al. Video super-resolution with convolutional neural networks. IEEE Transactions on Computational Imaging, 2016, 2(2): 109–122. [doi: 10.1109/TCI.2016.2532323
    [5] Yuan QQ, Li Y, Li JC, et al. Remote sensing image super-resolution via regional spatially adaptive total variation model. Proceedings of 2014 IEEE Geoscience and Remote Sensing Symposium. Quebec City: IEEE, 2014. 3073–3076.
    [6] Wang ZH, Chen J, Hoi SCH. Deep learning for image super-resolution: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(10): 3365–3387. [doi: 10.1109/TPAMI.2020.2982166
    [7] Bashir SMA, Wang Y, Khan M, et al. A comprehensive review of deep learning-based single image super-resolution. PeerJ Computer Science, 2021, 7: e621. [doi: 10.7717/peerj-cs.621
    [8] 向海燕. 超分辨率图像恢复方法综述. 重庆理工大学学报(自然科学), 2014, 28(9): 72–76
    [9] 陈文静, 唐轶. 基于深度学习的图像超分辨率重建方法综述. 云南民族大学学报(自然科学版), 2019, 28(6): 597–605
    [10] Zhang K, Zuo WM, Zhang L. Learning a single convolutional super-resolution network for multiple degradations. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018. 3262–3271.
    [11] Bevilacqua M, Roumy A, Guillemot C, et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. Proceedings of British Machine Vision Conference. Surrey: BMVC, 2012. 1–10.
    [12] Zeyde R, Elad M, Protter M. On single image scale-up using sparse-representations. Proceedings of the 7th International Conference on Curves and Surfaces. Avignon: Springer, 2010. 711–730.
    [13] Huang JB, Singh A, Ahuja N. Single image super-resolution from transformed self-exemplars. Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015. 5197–5206.
    [14] Martin D, Fowlkes C, Tal D, et al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. Proceedings of the 8th IEEE International Conference on Computer Vision. Vancouver: IEEE, 2001. 416–423.
    [15] Ogawa T, Otsubo A, Narita R, et al. Object detection for comics using Manga109 annotations. arXiv: 1803.08670, 2018.
    [16] Yang JC, Wright J, Huang TS, et al. Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 2010, 19(11): 2861–2873. [doi: 10.1109/TIP.2010.2050625
    [17] Agustsson E, Timofte R. NTIRE 2017 challenge on single image super-resolution: Dataset and study. Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu: IEEE, 2017. 1122–1131.
    [18] Lim B, Son S, Kim H, et al. Enhanced deep residual networks for single image super-resolution. Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu: IEEE, 2017. 1132–1140.
    [19] Deng J, Dong W, Socher R, et al. ImageNet: A large-scale hierarchical image database. Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami: IEEE, 2009. 248–255.
    [20] Lin TY, Maire M, Belongie S, et al. Microsoft COCO: Common objects in context. Proceedings of the 13th European Conference on Computer Vision. Zurich: Springer, 2014. 740–755.
    [21] Everingham M, Eslami SMA, van Gool L, et al. The PASCAL visual object classes challenge: A retrospective. International Journal of Computer Vision, 2015, 111(1): 98–136. [doi: 10.1007/s11263-014-0733-5
    [22] Tolosana R, Vera-Rodriguez R, Fierrez J, et al. Deepfakes and beyond: A survey of face manipulation and fake detection. Information Fusion, 2020, 64: 131–148. [doi: 10.1016/j.inffus.2020.06.014
    [23] Timofte R, Rothe R, van Gool L. Seven ways to improve example-based single image super resolution. Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016. 1865–1873.
    [24] Chao D, Loy CC, Tang XO. Accelerating the super-resolution convolutional neural network. Proceedings of the 14th European Conference on Computer Vision. Amsterdam: Springer, 2016. 391–407.
    [25] Arbeláez P, Maire M, Fowlkes C, et al. Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(5): 898–916. [doi: 10.1109/TPAMI.2010.161
    [26] Dong C, Loy CC, He KM, et al. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295–307. [doi: 10.1109/TPAMI.2015.2439281
    [27] Blau Y, Mechrez R, Timofte R, et al. The 2018 PIRM challenge on perceptual image super-resolution. Proceedings of European Conference on Computer Vision. Munich: Springer, 2018. 334–355.
    [28] Wang XT, Yu K, Dong C, et al. Recovering realistic texture in image super-resolution by deep spatial feature transform. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018. 606–615.
    [29] Sun LB, Hays J. Super-resolution from Internet-scale scene matching. Proceedings of 2012 IEEE International Conference on Computational Photography. Seattle: IEEE, 2012. 1–12.
    [30] Ledig C, Theis L, Huszár F, et al. Photo-realistic single image super-resolution using a generative adversarial network. IEEE Computer Society. Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017. 105–114.
    [31] Almohammad A, Ghinea G. Stego image quality and the reliability of PSNR. Proceedings of the 2010 2nd International Conference on Image Processing Theory, Tools and Applications. Paris: IEEE, 2010. 215–220.
    [32] Marabini R, Sorzano COS, Matej S, et al. 3-D reconstruction of 2-D crystals in real space. IEEE Transactions on Image Processing, 2004, 13(4): 549–561. [doi: 10.1109/TIP.2003.822620
    [33] Viswanathan M, Viswanathan M. Measuring speech quality for text-to-speech systems: Development and assessment of a modified mean opinion score (MOS) scale. Computer Speech & Language, 2005, 19(1): 55–83
    [34] Kim J, Lee S. Deep learning of human visual sensitivity in image quality assessment framework. Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017. 1969–1977.
    [35] Ma C, Yang CY, Yang XK, et al. Learning a no-reference quality metric for single-image super-resolution. Computer Vision and Image Understanding, 2017, 158: 1–16. [doi: 10.1016/j.cviu.2016.12.009
    [36] Talebi H, Milanfar P. NIMA: Neural image assessment. IEEE Transactions on Image Processing, 2018, 27(8): 3998–4011. [doi: 10.1109/TIP.2018.2831899
    [37] Zhang L, Zhang L, Mou XQ, et al. FSIM: A feature similarity index for image quality assessment. IEEE Transactions on Image Processing, 2011, 20(8): 2378–2386. [doi: 10.1109/TIP.2011.2109730
    [38] Blau Y, Michaeli T. The perception-distortion tradeoff. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018. 6228–6237.
    [39] Ahn N, Kang B, Sohn KA. Fast, accurate, and lightweight super-resolution with cascading residual network. Proceedings of the 15th European Conference on Computer Vision. Munich: Springer, 2018. 256–272.
    [40] Lai WS, Huang JB, Ahuja N, et al. Deep Laplacian pyramid networks for fast and accurate super-resolution. Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017. 5835–5843.
    [41] Johnson J, Alahi A, Li FF. Perceptual losses for real-time style transfer and super-resolution. Proceedings of the 14th European Conference on Computer Vision. Amsterdam: Springer, 2016. 694?711.
    [42] Gatys LA, Ecker AS, Bethge M. Image style transfer using convolutional neural networks. Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016. 2414–2423.
    [43] Goodfellow IJ, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets. Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal: NIPS, 2014. 2672–2680.
    [44] Bulat A, Tzimiropoulos G. Super-FAN: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with GANs. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018. 109–117.
    [45] Dong C, Loy CC, He KM, et al. Learning a deep convolutional network for image super-resolution. ECCV. Proceedings of the 13th European Conference on Computer Vision. Zurich: Springer, 2014. 184–199.
    [46] Kim J, Lee JK, Lee KM. Deeply-recursive convolutional network for image super-resolution. Proceedings of the IEEE conference on computer vision and pattern recognition. Las Vegas: IEEE, 2016. 1637–1645.
    [47] Zhang YL, Tian YP, Kong Y, et al. Residual dense network for image super-resolution. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018. 2472–2481.
    [48] Shi W, Caballero J, Huszár F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016. 1874–1883.
    [49] Hu XC, Mu HY, Zhang XY, et al. Meta-SR: A magnification-arbitrary network for super-resolution. Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019. 1575–1584.
    [50] Haris M, Shakhnarovich G, Ukita N. Deep back-projection networks for super-resolution. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018. 1664–1673.
    [51] Guo Y, Chen J, Wang JD, et al. Closed-loop matters: Dual regression networks for single image super-resolution. Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020. 5406–5415.
    [52] Li Z, Yang JL, Liu Z, et al. Feedback network for image super-resolution. Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019. 3862–3871.
    [53] Zhang YL, Li KP, Li K, et al. Image super-resolution using very deep residual channel attention networks. Proceedings of the 15th European Conference on Computer Vision. Munich: Springer, 2018. 294–310.
    [54] Dai T, Cai JR, Zhang YB, et al. Second-order attention network for single image super-resolution. Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019. 11057–11066.
    [55] Niu B, Wen WL, Ren WQ, et al. Single image super-resolution via a holistic attention network. Proceedings of the 16th European Conference on Computer Vision. Glasgow: Springer, 2020. 191–207.
    [56] Wang XT, Yu K, Wu SX, et al. ESRGAN: Enhanced super-resolution generative adversarial networks. Proceedings of European Conference on Computer Vision. Munich: Springer, 2019. 63–79.
    [57] Soh JW, Park GY, Jo J, et al. Natural and realistic single image super-resolution with explicit natural manifold discrimination. Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019. 8114–8123.
    [58] Ma C, Rao YM, Cheng YA, et al. Structure-preserving super resolution with gradient guidance. Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020. 7766–7775.
    [59] Lugmayr A, Danelljan M, van Gool L, et al. SRFlow: Learning the super-resolution space with normalizing flow. Proceedings of the 16th European Conference on Computer Vision. Glasgow: Springer, 2020. 715–732.
    [60] Tong T, Li G, Liu XJ, et al. Image super-resolution using dense skip connections. Proceedings of 2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017. 4809–4817.
    [61] Han W, Chang SY, Liu D, et al. Image super-resolution via dual-state recurrent networks. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018. 1654–1663.
    [62] Lai WS, Huang JB, Ahuja N, et al. Fast and accurate image super-resolution with deep Laplacian pyramid networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(11): 2599–2613. [doi: 10.1109/TPAMI.2018.2865304
    [63] Wang YF, Perazzi F, McWilliams B, et al. A fully progressive approach to single-image super-resolution. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Salt Lake City: IEEE, 2018. 977–986.
    [64] Wei YX, Gu SH, Li YW, et al. Unsupervised real-world image super resolution via domain-distance aware training. Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021. 13380–13389.
    [65] Yuan Y, Liu SY, Zhang JW, et al. Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Salt Lake City: IEEE, 2018. 814–823.
    [66] Zhu JY, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of 2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017. 2242–2251.
    [67] Shocher A, Cohen N, Irani M. Zero-shot super-resolution using deep internal learning. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018. 3118–3126.
    [68] Soh JW, Cho S, Cho NI. Meta-transfer learning for zero-shot super-resolution. Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020. 3513–3522.
    [69] Liang JY, Cao JZ, Sun GL, et al. SwinIR: Image restoration using swin transformer. Proceedings of 2021 IEEE/CVF International Conference on Computer Vision Workshops. Montreal: IEEE, 2021. 1833–1844.
    [70] Gu JJ, Lu HN, Zuo WM, et al. Blind super-resolution with iterative kernel correction. Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019. 1604–1613.
    [71] Luo ZX, Huang Y, Li S, et al. Unfolding the alternating optimization for blind super resolution. Proceedings of the 34th Conference on Neural Information Processing Systems. Vancouver: NeurIPS, 2020. 1–12.
    引证文献
引用本文

邢苏霄,陈金玲,李锡超,陈彤.基于深度学习的单图像超分辨率重建综述.计算机系统应用,2022,31(7):23-34

复制
分享
文章指标
  • 点击次数:
  • 下载次数:
  • HTML阅读次数:
  • 引用次数:
历史
  • 收稿日期:2021-10-12
  • 最后修改日期:2021-11-08
  • 在线发布日期: 2022-05-31
文章二维码
友情链接:中国科学院软件研究所中国计算机学会中国科学院国家自然科学基金委员会软件学报
您是第11366329位访问者
版权所有：中国科学院软件研究所 京ICP备05046678号-3
地址：北京海淀区中关村南四街4号 中科院软件园区 7号楼305房间,邮政编码：100190
电话：010-62661041 传真： Email：csa (a) iscas.ac.cn
技术支持：北京勤云科技发展有限公司

京公网安备 11040202500063号