With the development of technologies such as computer networks and wireless communication, copyright protection and information security issues of video media documents have become increasingly the focus of people’s attention, and video media file encryption is a way to effectively protect information security. Traditional video file encryption methods need to encrypt all video frame data in video media files. The efficiency of file encryption is relatively low, and the encryption process is time-consuming. Therefore, a method for improving the efficiency of video media file encryption based on the Chinese SM2 algorithm is proposed according to the structural characteristics of H.264/AVC video frames. This method only encrypts the NALU Header information of the key frame in the encrypted video media during video media file encryption. In the case of detecting H.264 shards, it is also necessary to encrypt the non-IDR Header information. The experimental results show that the method can effectively encrypt video media files while reducing the time required for encryption, thus significantly improving the encryption efficiency of video media files.