In the safety management of the operation sites, the supervision of fence crossing by non-construction personnel has always been essential. However, at present, there are many problems in the construction sites, such as a wide range of operation and a difficulty in the management of construction personnel, leading to the inefficiency of manual supervision. As an important research hot spot in the field of computer vision, video-based human action detection is widely used in public security monitoring. Therefore, in view of the shortcomings of the traditional manual supervision, in combination with the current computer vision technology, an intelligent detection and recognition method for fence crossing violations is proposed in this paper. In this method, video frames are acquired continuously through monitoring, and clips composed of video frames are taken as input. In addition, temporal and spatial features are extracted by 3D and 2D convolutions respectively. After fusion of the two parts of features, classification and boundary box regression are carried out. Furthermore, a comparative experiment is conducted to verify the effect of this method. The experimental results show that the proposed method can detect the fence crossing behavior accurately in a short time, featuring strong working ability.