The surrounding areas of underground infrastructure such as optical cables and high-pressure oil and gas pipelines are vulnerable to brutal invasion by excavators. This study proposes an excavator detection and working state discrimination method combined with Yolopose and a multilayer perceptron. First, the Yolopose-ex extraction network based on Yolopose’s six-point posture of the excavator is designed. Secondly, the Yolopose-ex model is utilized to extract the change information of the excavator’s working posture in the video, and the working state feature vector (MSV) of the excavator is constructed. Finally, the multilayer perceptron (MLP) is adopted to analyze the working status of the excavator in the video. The experimental results show that the proposed method overcomes the problem of difficult discrimination of complex backgrounds, and the accuracy of the identification of the working state of the excavator reaches 96.6%, which has a high reasoning speed and generalization ability.