Instant messaging system (IMS) has become an important communication method due to its real-time characteristics, which can improve work efficiency and reduce communication costs. It plays an increasingly important role in enterprises, schools, governments and other organizations. However, while instant communication brings convenience, its inherent security weaknesses hinder its further development. In order to ensure the security of the instant messaging system, some advanced security encryption algorithms are used in the communication system to prevent attacks and information leakage. However, these algorithms have their own shortcomings in terms of encryption strength or encryption speed. After understanding the limitations of these encryption algorithms, in this study, we proposed an alternative algorithm that aims to leverage and combine the best features of both these algorithms and provide much better security than either of them, namely 3DES-RC4 hybrid encryption algorithm, is an algorithm with a 256-byte key space. The complexity is increased from O(2¹⁶⁸) to O(2⁵¹⁰⁰) compared with 3DES algorithm. Based on this algorithm, an instant communication system is designed. The encryption and decryption functions of the system are tested, and the performance and strength of the proposed algorithm are analyzed. By comparing to the 3DES algorithm, it is proved that the algorithm proposed in this paper retains the characteristics of 3DES encryption strength and RC4 pseudo-randomness, and is superior to the constituent algorithms in terms of encryption strength and adaptability.