In the traditional first-order hidden Markov model (HMM1), each state in the state sequence is assumed to be only related to the previous state. In this way, although the model learning and recognition algorithm can be simply and effectively deduced, a lot of information passed down from the above is lost. Therefore, in view of the traditional HMM1, a continuous sign language recognition method based on the second-order hidden Markov model (HMM2) is proposed to solve the problems of the difficulty and low accuracy of sign language recognition. In this method, a video of sign language is divided into several short videos by the sliding window algorithm, and the feature vectors of the short videos and word videos of sign language are obtained through the 3D convolution model. The relevant parameters of the HMM2 are thereby calculated, and continuous sign language recognition is achieved via the Viterbi algorithm. Experimental results show that the accuracy of sign language recognition based on the HMM2 is 88.6%, which is higher than that of the traditional HMM1.