Crossing behavior detection is of great significance for epidemic control and social security and can reduce accidents caused by illegal crossing behavior to a certain extent. In view of the problems of poor real-time performance and the need for prior knowledge in the current crossing behavior detection task, this study applies the Faster RCNN+SlowFast spatiotemporal behavior detection algorithm to the crossing behavior detection task to split and detect the crossing behavior. In order to improve the detection accuracy and speed of the target in the spatiotemporal behavior detection algorithm, the target detection module, namely Faster RCNN is changed to lightweight YOLOv5 with high real-time performance. Then, according to the extensive in-class diversity under different perspectives of the same behavior, the residual block of the Fast branch and Slow branch is changed to AC residual block and SE residual block, respectively, so as to strengthen the network’s learning ability to key features and fine-grained features. Finally, the crossing behavior detection algorithm is designed to detect the continuity of climbing and descending states. Experimental results show that the average accuracy of the network reaches 93.5%, which shows excellent performance in crossing behavior detection.