The samples to be tested for metal surface defects are often characterized by low resolution, fuzzy defect boundaries, dense defects, and small defect targets. At the same time, the constructed detection model has a large number of hyperparameters that need to be manually adjusted and lacks the adaptive parameter adjustment ability. In this study, a surface defect super-resolution detection algorithm based on Bayesian optimization is proposed. Through the design of fine layered structure, the receptive field of the backbone network feature map is enriched; the extraction of high-low frequency information is enhanced; the high-resolution image with clear edge texture is reconstructed. By constructing the bottleneck residual dense structure, the shallow and deep features of the backbone feature extraction network are enriched, and the classification ability and the localization ability of the model for small targets and dense targets are improved. The key hyperparameters of the detection model are optimized adaptively by a Bayesian optimization algorithm with low time cost. Experiments show that mAP_0.5 for six types of metal surface defects in the NEU-DET dataset can reach 0.782, and the detection speed can reach 102 f/s, which is superior to other detection algorithms.