The factory environment is complex and changeable, with many dangerous areas, and illegal entry can bring serious harm to the life and health of workers. Considering the complex operation and poor recognition effect of traditional detection methods, this study proposes an alarm system for workers’ intrusion in dangerous areas on the basis of the improved YOLOv5s model. Firstly, the binocular ranging technology based on the SGBM algorithm is integrated into YOLOv5s object detection, and the trigger condition of spatial distance is added. Hence, the sound and light alarm will be triggered only when workers approach the camera within a certain range. Furthermore, the attention mechanism is introduced into YOLOv5s. Comparative experiments prove that the introduction of the CA module improves the average accuracy of mAP@0.5 by 1.86%. The results show that this method can accurately identify the intrusion of a worker in dangerous areas and gives a sound and light alarm to remind the worker, which provides a new means for factory safety management.