The IEC 60870-5-103 protocol is an information interface supporting standard applied to relay protection equipment and transmits mainly the information related to relay protection. The message is transmitted in plain text and has poor security for a lack of encryption measures and digital signature mechanism. A communication experiment environment between the master station and the DTU terminal is built to verify that there are hidden dangers in the 103 protocol of Ethernet transmission. A man-in-the-middle attack test is carried out on the system by detecting ARP spoofing. The experimental results show that the 103 protocol of Ethernet transmission faces the risk of man-in-the-middle attack. In order to improve the security of the protocol, we propose a two-way identity authentication mechanism based on an asymmetric cryptographic algorithm and rely on a symmetric encryption mechanism and digital signature technology to ensure the confidentiality and integrity of the transmitted message. Finally, the method is validated through simulation tests.