Aiming at the low accuracy of students’ classroom behavioral state recognition, an improved model based on YOLOv4 is proposed. We establish a dataset for students’ classroom behavioral states, adjust the YOLOv4 algorithm to train the parameters of the model, and modify the activation function of the convolutional block as an ELU function to optimize the model. Meanwhile, DIoU-soft-NMS is proposed as a non-maximum suppression mechanism to identify and analyze students’ classroom behavioral states. According to the duration of each state and the frequency of state changes, the effective length of students’ attention on lectures is calculated, and by referring to the scoring principle of Shandong college entrance examinations, the quantitative evaluation criteria of students’ classroom attention and the quantitative evaluation criteria of teachers’ classroom teaching effects are established. The experimental results show that when the same evaluation index is used to measure the model, the mean average precision (mAP) of the model reaches 98.8% assumed that the detection rate of students’ classroom behaviors remains unchanged, which is 3.53% higher than that of the original YOLOv4 model. Students approve the quantitative evaluation criteria of classroom attention, and it has a high degree of agreement.