Road defect detection, as an important method for measuring pavement damage and maintaining road maintenance, faces challenges, including extreme length-to-width ratios, varying defect sizes, and uneven distributions of easy versus difficult defects. Current convolution-based methods have achieved larger receptive fields to enhance perception, but at the expense of high-frequency components that contain small defects, making them unsuitable for road defect detection tasks. To address this, a road defect detection algorithm, FS-YOLO, based on frequency enhancement and synergy of geometric shape and category, is proposed. First, to balance the receptive field and high-frequency information, a frequency-adaptive dilation strategy is introduced, dynamically adjusting the spatial expansion rate according to local frequency components, and assigning appropriate convolutional kernels to defects of different sizes. Second, given that different types of defects have distinct geometric shapes and positions, an attention-based three-dimensional explicit synergy dynamic detection head is introduced to achieve explicit synergy between spatial geometric information and category information, enabling the model to leverage the inherent potential of defect categories and spatial locations. Finally, the Slide loss function is introduced to address the imbalance in the distribution of difficult and easy defects in real-world roads, particularly enhancing the model’s ability to handle difficult-to-distinguish samples. Experimental results show that FS-YOLO significantly outperforms the baseline model in terms of precision and recall on both the self-built dataset and the public road defect detection datasets RDD 2022 and UAV-PDD. It has also been effectively validated in practical applications on expressways and national and provincial roads, significantly improving the accuracy and efficiency of defect detection.