基于自监督多模态语义通信的人体3D重建
作者:
作者单位:

作者简介:

通讯作者:

中图分类号:

基金项目:

2024年度广东高校科研平台和项目 (自然科学类) (2024ZDZX3035)


3D Reconstruction of Human Based on Self-supervised Multimodal Semantic Communication
Author:
Affiliation:

Fund Project:

  • 摘要
  • |
  • 图/表
  • |
  • 访问统计
  • |
  • 参考文献
  • |
  • 相似文献
  • |
  • 引证文献
  • |
  • 资源附件
  • |
  • 文章评论
    摘要:

    针对多模态数据传输中语义信息丢失及信道噪声干扰导致人体3D重建精度下降的问题, 本文提出了一种基于自监督多模态语义通信的人体3D重建方法(SMH3D). 本方法利用跨模态对比自监督学习, 实现RGB图像与深度图等多模态数据的特征对齐, 并通过变分自编码器对特征进行压缩; 随后引入知识图谱增强的语义编码机制和可变长度编码策略, 以在有限带宽下优先保护关键语义信息. 针对真实信道条件的不稳定性, 本文设计了端到端的信道编码与解码模块, 并采用信道状态反馈自适应调整传输参数, 从而保证在噪声、衰落等恶劣环境下语义特征的高保真传输. 通过基于Transformer与图神经网络相结合的多模态特征融合策略, 并利用条件隐式函数实现高精度人体3D模型重建. 实验结果表明, 在不同信噪比条件下, SMH3D在IoU、PSNR和语义保真度等指标上均显著优于传统的点云传输方法SemCom, 及面向实时三维重建任务的语义通信方法SCS, 尤其在低信噪比环境下表现出更强的鲁棒性和稳定性.

    Abstract:

    Given semantic information loss in multimodal data transmission and 3D reconstruction accuracy degradation of human bodies caused by channel noise interference, this study proposes a 3D reconstruction method for human bodies based on self-supervised multimodal semantic communication (SMH3D). This method utilizes cross-modal contrastive self-supervised learning to achieve feature alignment of multimodal data, such as RGB images and depth maps, and compresses the features by a variational self-encoder. Then, a knowledge graph-enhanced semantic coding mechanism and a variable-length coding strategy are introduced to prioritize the protection of key semantic information under limited bandwidth. To solve the instability of real channel conditions, this study designs the end-to-end channel coding and decoding modules and employs channel state feedback to adaptively adjust the transmission parameters, thus ensuring the high-fidelity transmission of semantic features in harsh environments such as noise and fading. A multimodal feature fusion strategy based on the combination of Transformer and graph neural networks, and the conditional implicit function is adopted to realize high-accuracy 3D model reconstruction of human bodies. The experimental results show that SMH3D significantly outperforms the traditional point cloud transmission method SemCom and the semantic communication method SCS for real-time 3D reconstruction tasks in terms of IoU, PSNR, and semantic fidelity, showing stronger robustness and stability in low signal-to-noise ratio environments.

    参考文献
    相似文献
    引证文献
引用本文

唐显锋,叶仕通.基于自监督多模态语义通信的人体3D重建.计算机系统应用,2025,34(11):279-288

复制
分享
相关视频

文章指标
  • 点击次数:
  • 下载次数:
  • HTML阅读次数:
  • 引用次数:
历史
  • 收稿日期:2025-03-08
  • 最后修改日期:2025-03-31
  • 录用日期:
  • 在线发布日期: 2025-09-30
  • 出版日期:
文章二维码
您是第位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京市海淀区中关村南四街4号,邮政编码:100190
电话:010-62661041 传真: Email:csa@iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号