Since the existing deep model faces many problems such as a large number of model parameters, insufficient feature fusion, and low detection accuracy for small targets in the field of industrial bearing appearance defect detection, a lightweight adaptive feature fusion detection network (Efficient-YOLO) is proposed. First of all, the network uses the EfficientNetV2 structure embedded in the CBAM attention mechanism for basic feature extraction to ensure model accuracy and significantly optimize the model parameters. Secondly, an adaptive feature fusion network (CBAM-BiFPN) is designed to strengthen the network’s extraction of effective feature information. Then, the Swin?Transformer mechanism is introduced in the downstream feature fusion network, and the Ghost convolution introduced by the upstream network is used to greatly improve the model’s global perception of bearing appearance defects. Finally, the improved non-maximum suppression method (Soft-CIoU-NMS) is applied in the inference phase, with distance-related weight evaluation factors added, so as to reduce missed detection of overlapping frames. The experimental results show that compared with the existing mainstream detection models, the method has a mAP of 90.1% on the bearing surface defect dataset. The number of parameters is reduced to 1.99M. and the calculation amount is 7 GFLOPs. The recognition rate of small targets with bearing defects is significantly improved, which meets the needs of industrial bearing appearance defect detection.