The research on the recognition of abnormal human behavior in video surveillance systems is of great significance. As traditional algorithms are easily affected by the environment and have poor timeliness and accuracy, an abnormal behavior recognition algorithm based on skeleton sequence extraction is proposed. Firstly, the improved YOLOv3 network is used to detect targets and is combined with the RT-MDNet algorithm to track them for target trajectories. Then, the OpenPose model is employed to extract the skeleton sequence of targets in the trajectories. Finally, the spatiotemporal graph convolutional network combined with clustering is applied to recognize the abnormal behavior of the targets. The experimental results indicate that the proposed algorithm has a processing speed of 18.25 fps and recognition accuracy of 94% under a complex background of light changes, which can accurately identify the abnormal behavior of various targets in real time.