基于差分融合句法特征的英语语法纠错模型
作者:
基金项目:

国家自然科学基金(62272308)


Grammatical Error Correction Model Based on Differential Fusion Syntactic Feature
Author:
  • 摘要
  • | |
  • 访问统计
  • |
  • 参考文献 [20]
  • |
  • 相似文献 [20]
  • | | |
  • 文章评论
    摘要:

    当前的英文语法纠错模型往往忽略了有利于语法纠错的文本句法知识, 从而使得英语语法纠错模型的纠错能力受到影响. 针对上述问题, 提出一种基于差分融合句法特征的英语语法纠错模型. 首先, 本文提出的句法编码器不仅可以直接从文本中无监督地生成依存关系图和成分句法树信息, 而且还能将上述两种异构的句法结构进行特征融合, 编码成高维的句法表征. 其次, 为了同时利用文本中的语义和句法信息, 差分融合模块先使用差分正则化加强语义编码器捕获句法编码器未能生成的语义特征, 然后采用协同注意力将句法表征和语义表征进一步融合, 作为Transformer编码端的输出特征, 最终输入到解码端, 从而生成语法正确的文本. 在CoNLL-2014 英文纠错任务数据集上进行对比实验, 结果表明, 该方法的准确率和F0.5值优于基于Copy-Augmented Transformer的语法纠错模型, 其F0.5值提升了5.2个百分点, 并且句法知识避免了标注数据过少问题, 具有更优的文本纠错效果.

    Abstract:

    Current English GEC methods tend to ignore the syntactic knowledge in texts, which plays an important role in grammatical error correction, and thus the error correction ability of English GEC models is affected. To address this problem, the study proposes a GEC method which is based on the differential fusion syntactic features. First, the proposed syntactic encoder can generate dependency graph and constituency syntactic tree information from raw data in an unsupervised way and conduct the feature fusion of these two heterogeneous syntactic structures to encode high-dimensional syntactic representation. Second, to utilize both semantic and syntactic information in the text, the differential fusion module first uses differential regularization to enhance the semantic encoder to capture the semantic features that the syntactic encoder fails to generate. Then the syntactic representation and semantic representation are further fused by cross attention as the output features of the Transformer encoder, which are finally input to the decoder to generate grammatically correct text. The comparison experiment on the CoNLL-2014 task dataset shows that the precision and F0.5 value of this method are better than those of the GEC model based on the Copy-Augmented Transformer, and the F0.5 value of this method is improved by 5.2 percentage points. The syntactic knowledge avoids the problem of lacking high-quality annotated training corpora and has a better performance in text error correction.

    参考文献
    [1] Zhao W, Wang L, Shen KW, et al. Improving grammatical error correction via pre-training a copy-augmented architecture with unlabeled data. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis: Association for Computational Linguistics, 2019. 156–165.
    [2] Junczys-Dowmunt M, Grundkiewicz R, Guha S, et al. Approaching neural grammatical error correction as a low-resource machine translation task. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). New Orleans: Association for Computational Linguistics, 2018. 595–606.
    [3] Kaneko M, Mita M, Kiyono S, et al. Encoder-decoder models can benefit from pre-trained masked language models in grammatical error correction. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020. 4248–4254.
    [4] Bugliarello E, Okazaki N. Enhancing machine translation with dependency-aware self-attention. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020. 1618–1627.
    [5] He SX, Li ZC, Zhao H. Syntax-aware multilingual semantic role labeling. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Hong Kong: Association for Computational Linguistics, 2019. 5350–5359.
    [6] Li RF, Chen H, Feng FX, et al. Dual graph convolutional networks for aspect-based sentiment analysis. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Association for Computational Linguistics, 2021. 6319–6329.
    [7] Chollampatt S, Ng HT. A multilayer convolutional encoder-decoder neural network for grammatical error correction. Proceedings of the 32nd AAAI Conference on Artificial Intelligence and 30th Innovative Applications of Artificial Intelligence Conference and 8th AAAI Symposium on Educational Advances in Artificial Intelligence. New Orleans: AAAI Press, 2018. 706.
    [8] Rothe S, Mallinson J, Malmi E, et al. A simple recipe for multilingual grammatical error correction. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Association for Computational Linguistics, 2021. 702–707.
    [9] Devlin J, Chang MW, Lee K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis: Association for Computational Linguistics, 2019. 4171–4186.
    [10] Lewis M, Liu YH, Goyal N, et al. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020. 7871–7880.
    [11] Yasunaga M, Leskovec J, Liang P. LM-Critic: Language models for unsupervised grammatical error correction. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Punta Cana: Association for Computational Linguistics, 2021. 7752–7763.
    [12] Raheja V, Alikaniotis D. Adversarial grammatical error correction. Proceedings of the 2020 Findings of the Association for Computational Linguistics. Association for Computational Linguistics, 2020. 3075–3087.
    [13] Awasthi A, Sarawagi S, Goyal R, et al. Parallel iterative edit models for local sequence transduction. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Hong Kong: Association for Computational Linguistics, 2019. 4260–4270.
    [14] Chollampatt S, Wang WQ, Ng HT. Cross-sentence grammatical error correction. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence: Association for Computational Linguistics, 2019. 435–445.
    [15] Grundkiewicz R, Junczys-Dowmunt M. Near human-level performance in grammatical error correction with hybrid machine translation. Proceedings of the 2018 conference of the North American chapter of the association for computational linguistics: Human Language Technologies, Volume 2 (Short Papers). New Orleans: Association for Computational Linguistics, 2018. 284–290.
    [16] Li ZC, Parnow K, Zhao H. Incorporating rich syntax information in grammatical error correction. Information Processing & Management, 2022, 59(3): 102891
    [17] Fei H, Wu SQ, Ren YF et al. Better combine them together! Integrating syntactic constituency and dependency representations for semantic role labeling. Proceedings of the 2021 Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Association for Computational Linguistics, 2021. 549–559.
    [18] Shen YK, Lin ZH, Jacob AP, et al. Straight to the tree: Constituency parsing with neural syntactic distance. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Melbourne: Association for Computational Linguistics, 2019. 1171–1180.
    [19] Luo HY, Jiang L, Belinkov Y, et al. Improving neural language models by segmenting, attending, and predicting the future. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence: Association for Computational Linguistics, 2019. 1483–1493.
    [20] Shen YK, Tay Y, Zheng C, et al. StructFormer: Joint unsupervised induction of dependency and constituency structure from masked language modeling. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Association for Computational Linguistics, 2021. 7196–7209.
    引证文献
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

罗松,汪春梅,袁非牛,戴维.基于差分融合句法特征的英语语法纠错模型.计算机系统应用,2023,32(10):293-300

复制
分享
文章指标
  • 点击次数:537
  • 下载次数: 1102
  • HTML阅读次数: 983
  • 引用次数: 0
历史
  • 收稿日期:2023-03-24
  • 最后修改日期:2023-04-28
  • 在线发布日期: 2023-07-14
文章二维码
您是第11275056位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京海淀区中关村南四街4号 中科院软件园区 7号楼305房间,邮政编码:100190
电话:010-62661041 传真: Email:csa (a) iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号