In construction sites, many high fall accidents have occurred, so it is necessary to wear helmets. An improved algorithm based on YOLOX-s is proposed to deal with missing and omitted detection of small target samples encountered in helmet-wearing condition detection. First, the 160×160 feature layer in the Neck layer is introduced in the backbone feature extraction network for feature fusion, and a detection head for small targets is added; second, the SIoU loss function is used to calculate the loss value, which makes the loss term considered in the training process of the network more comprehensive, and the varifocal loss function is used to calculate the loss value of the confidence level to further reduce the imbalance of the positive and difficult samples in the training process; finally, coordinate attention (CA) mechanism is used to enhance the feature representation of the model. The experimental results show that the optimization of the Neck layer, detection layer, and loss function and the introduction of the CA mechanism lead to better convergence and regression performance of the network during the training process. The mAP value of the improved algorithm is 95.57%, which is 17.11% and 3.59% higher than that of YOLOv3 and the original YOLOX-s algorithm, respectively. The detection speed of the improved algorithm is 54.73 frames/s, which meets the real-time detection speed requirement.