本文已被:浏览 1141次 下载 2026次
Received:January 04, 2020 Revised:January 22, 2020
Received:January 04, 2020 Revised:January 22, 2020
中文摘要: 自动生成图片描述是自然语言处理和计算机视觉的热点研究话题,要求计算机理解图像语义信息并用人类自然语言的形式进行文字表述.针对当前生成中文图像描述整体质量不高的问题,提出首先利用FastText生成词向量,利用卷积神经网络提取图像全局特征;然后将成对的语句和图像〈S,I〉进行编码,并融合为两者的多模态特征矩阵;最后模型采用多层的长短时记忆网络对多模态特征矩阵进行解码,并通过计算余弦相似度得到解码的结果.通过对比发现所提模型在双语评估研究(BLEU)指标上优于其他模型,生成的中文描述可以准确概括图像的语义信息.
中文关键词: 图像中文描述 FastText语言模型 卷积神经网络 长短时记忆网络
Abstract:Automatic image captioning is a hot topic which connects natural language processing and computer vision. It mainly completes the task of understanding image semantic information and expressing it in the form of human natural language. For the overall quality of Chinese image captioning is not very high, this study uses FastText to generate word vector, uses convolution neural network to extract the global features of the image, then encodes the pairs of sentences and images〈S, I〉, and finally merges them into a feature matrix containing both Chinese description and image information. Decoder uses LSTM model to decode the feature matrix, and obtains the decoding result by calculating cosine similarity. Through comparison, we find that the model proposed in this study is better than other models in BiLingual Evaluation Understudy (BLEU). The Chinese description generated by the model can accurately summarize the semantic information of the image.
keywords: Chinese image captioning FastText Convolutional Neural Network (CNN) Long and Short Term Memory network (LSTM)
文章编号: 中图分类号: 文献标志码:
基金项目:
Author Name | Affiliation | |
CHEN Xing | College of Computer and Information, Hohai University, Nanjing 211100, China | hsingchen@hhu.edu.cn |
Author Name | Affiliation | |
CHEN Xing | College of Computer and Information, Hohai University, Nanjing 211100, China | hsingchen@hhu.edu.cn |
引用文本:
陈兴.基于多模态神经网络生成图像中文描述.计算机系统应用,2020,29(9):191-197
CHEN Xing.Generation of Chinese Image Description by Multimodal Neural Network.COMPUTER SYSTEMS APPLICATIONS,2020,29(9):191-197
陈兴.基于多模态神经网络生成图像中文描述.计算机系统应用,2020,29(9):191-197
CHEN Xing.Generation of Chinese Image Description by Multimodal Neural Network.COMPUTER SYSTEMS APPLICATIONS,2020,29(9):191-197