###
计算机系统应用英文版:2024,33(12):115-122
本文二维码信息
码上扫一扫!
双注意力记忆多智能体强化学习
(大连海事大学 人工智能学院, 大连 116026)
BiTransformer Memory for Multi-agent Reinforcement Learning
(College of Artificial Intelligence, Dalian Maritime University, Dalian 116026, China)
摘要
图/表
参考文献
相似文献
本文已被:浏览 25次   下载 189
Received:May 22, 2024    Revised:June 17, 2024
中文摘要: 多智能体协同在强化学习研究领域占据重要地位, 旨在深入探讨智能体如何通过相互协作实现共同目标. 大部分协作多智能体算法注重合作的构建, 但忽略了个体策略的强化. 为解决上述问题, 本文提出一种BiTransformer记忆(BTM)在线强化学习模型, 该模型不仅考虑多智能体之间的协同, 还利用记忆模块辅助个体决策. BTM由双注意力编码器和双注意力解码器组成, 分别用于个体策略的增强和多智能体系统的协作. 在双注意力编码器中, 受人类的决策经验依赖的启发, 提出记忆注意力模块为当前决策提供历史决策经验. 与传统利用RNN的方法不同, BTM为每一个提供的是一个显式历史决策经验库, 而非隐藏单元. 此外, 提出融合注意力模块, 在历史决策经验的辅助下处理当下的局部观测信息, 从而获取环境中最具决策价值的信息, 进一步提高智能体个体的决策能力. 在双注意力解码器中, 本文提出了决策注意力模块和合作注意力模块两个模块, 通过综合考虑其他已经做出决策智能体与当前智能体的合作收益以及带有历史决策经验的局部观察, 从而促进历史决策辅助下的多智能体潜在合作的形成. 最终本文在星际争霸中的多个场景下对BTM进行了测试, 取得了93%的平均胜率.
Abstract:Multi-agent collaboration plays a crucial role in the field of reinforcement learning, focusing on how agents cooperate to achieve common goals. Most collaborative multi-agent algorithms emphasize the construction of collaboration but overlook the reinforcement of individual decision-making. To address this issue, this study proposes an online reinforcement learning model, BiTransformer memory (BTM), which not only considers the collaboration among multiple agents but also uses a memory module to assist individual decision-making. The BTM model is composed of a BiTransformer encoder and a BiTransformer decoder, which are utilized to improve individual decision-making and collaboration within the multi-agent system, respectively. Inspired by human reliance on historical decision-making experience, the BiTransformer encoder introduces a memory attention module to aid current decisions with a library of explicit historical decision-making experience rather than hidden units, differing from the conventional RNN-based method. Additionally, an attention fusion module is proposed to process partial observations with the assistance of historical decision experience, to obtain the most valuable information for decision-making from the environment, thereby enhancing the decision-making capabilities of individual agents. In the BiTransformer decoder, two modules are proposed: a decision attention module and a collaborative attention module. They are used to foster potential cooperation among agents by considering the collaborative benefits between other decision-making agents and the current agent, as well as partial observations with historical decision-making experience. BTM is tested in multiple scenes of StarCraft, achieving an average win rate of 93%.
文章编号:     中图分类号:    文献标志码:
基金项目:
引用文本:
马裕博,周长东,张志文,杨培泽,张博.双注意力记忆多智能体强化学习.计算机系统应用,2024,33(12):115-122
MA Yu-Bo,ZHOU Chang-Dong,ZHANG Zhi-Wen,YANG Pei-Ze,ZHANG Bo.BiTransformer Memory for Multi-agent Reinforcement Learning.COMPUTER SYSTEMS APPLICATIONS,2024,33(12):115-122