In the field of security, fatigue is an important reason for the decrease in the attention of security staff and can cause various problems. The existing fatigue detection methods, however, have many problems, such as the strong invasiveness and high costs of physiological indexes for detection, the impact of individual differences and head poses on the facial fatigue detection results, and the short early-warning time of fatigue. Therefore, this study puts forward an early and late fatigue detection algorithm based on the adaptive threshold and the empirical fusion of multiple facial features. In this algorithm, the lightweight SCRFD model is used for face detection, and the MobileNetV2 model is used to locate the key points of the face. The gradient boosting decision tree (GBDT) is applied to learn the mapping relationship between the head pose information and the eye aspect ratio (EAR) threshold, and the recognition of movements such as blinking, yawning, and leaning forward and backward is achieved through the six degrees of freedom of the percentage of eyelid closure over the pupil time (PERCLOS), the frequency of opening mouth (FOM), and head pose, respectively. In the fatigue estimation stage, various fatigue behaviors should be fused and mapped into fatigue-related KSS values. Hence, the fatigue causality maps of various facial behaviors are constructed in advance according to expert experience. Then, the self-defined singleton, mutual, and activate/inhibit feature operators are used to calculate the KSS values at the early and late stages of fatigue from the facial behavior detection sequences in combination with the causality maps. Finally, the dual-scale KNN is employed to realize the early and late fatigue estimation. The experimental results show that the yawn detection accuracy of the proposed algorithm reaches 93.81% on the YawDD dataset, and the fatigue detection accuracy of the algorithm is 67.72% and 87.88% on UTA-RLDD and Drozy datasets, respectively. The real-time inference performance can reach 17.96 frames per second (FPS) only through the CPU.