This study proposes an improved lightweight pavement disease detection model called pavement disease-YOLOv5s (PD-YOLOv5s) to address the problem of low detection accuracy in pavement disease detection due to diverse disease forms, large-scale differences, and similar background grayscale values. Firstly, the model applies a three-dimensional parameter-free attention mechanism called SimAM to effectively enhance the feature extraction ability of the model in complex environments without increasing the number of model parameters. Secondly, the model integrates the residual block Res2NetBlock to expand its receptive field and improve its feature fusion at a finer granularity level. Finally, the SPD-GSConv module is constructed for downsampling to effectively capture target features of different scales and integrate the extracted features into the model to perform pavement disease classification detection. Experimental results on real pavement disease datasets show that the mean average precision (mAP) of the PD-YOLOv5s model is improved by 4.7% compared to that of the original YOLOv5s model. The parameters of the proposed model are reduced to 6.78M, and the detection speed reaches 53.97 f/s. The PD-YOLOv5s model has superior detection performance while reducing network computing costs, making it valuable for engineering applications in pavement disease detection.