本文已被:浏览 1387次 下载 3119次
Received:November 17, 2015 Revised:December 21, 2015
Received:November 17, 2015 Revised:December 21, 2015
中文摘要: 随着微博等社交网络的普及,新词源源不断涌现,分词系统经常将新词错误切分为单字.新词发现已经成为中文自然语言处理领域的研究热点.现有新词识别方法依赖大规模语料统计数据,对低频新词识别能力差.本文提出一种扩展Skip-gram模型和词向量投影方法,将两者结合后能缓解自然语言处理中常见的数据稀疏问题,有效识别低频新词,进而提高分词系统的准确率和召回率.
Abstract:With the popularity of microblog and other social networks, a steady stream of new words emerge, Chinese word segmentation systems often cut the new words into Chinese characters. The new word discovery has become a hot topic in the field of Chinese natural language processing. Existing new word recognition methods rely on the statistical data of large-scale corpus, the ability of new low-frequency word recognition is poor. This paper presents an extension of skip-gram model and word vector projection method, after the combination of the this two methods can ease the data sparseness problem effectively in natural language processing, to identify new low-frequency words, and to improve the precision and recall rate of Chinese word segmentation system.
keywords: skip-gram SOM word vector microblog new word recognition
文章编号: 中图分类号: 文献标志码:
基金项目:福建省教育厅科技项目(JA11304)
Author Name | Affiliation |
YU Jie | Computer Engineering Department, Fujian Polytechnic of Information Technology, Fuzhou 350003, China |
Author Name | Affiliation |
YU Jie | Computer Engineering Department, Fujian Polytechnic of Information Technology, Fuzhou 350003, China |
引用文本:
于洁.Skip-Gram模型融合词向量投影的微博新词发现.计算机系统应用,2016,25(7):130-136
YU Jie.Microblog New Word Recognition Combining Skip-Gram Model and Word Vector Projection.COMPUTER SYSTEMS APPLICATIONS,2016,25(7):130-136
于洁.Skip-Gram模型融合词向量投影的微博新词发现.计算机系统应用,2016,25(7):130-136
YU Jie.Microblog New Word Recognition Combining Skip-Gram Model and Word Vector Projection.COMPUTER SYSTEMS APPLICATIONS,2016,25(7):130-136