Rib fracture is a common disease in clinical medicine. However, the diagnosis of fracture with the manual method is heavy in workload and difficult. To help doctors to reduce workload and improve detection sensitivity , we present a rib fracture classification algorithm based on a rib fracture network (RF-Net). Firstly, generative adversarial networks are used to generate synthetic medical images to enlarge the data size. Then, these data are input into our RF-Net to yield classification results. The data augmentation method can ease the overfitting phenomenon and improve the model training. In RF-Net, we use RF-block to replace the ordinary depth wise separable convolution , which can extract multi-scale features to strengthen the feature extraction ability of the entire network. Furthermore, considering the high requirement for fast speed, we apply the compression strategy to optimize some high-dimension modules to decrease the computation cost. The comparison with the existing deep learning models demonstrates that our method achieves the best result in multiple indicators, including accuracy, the area under the curve (AUC), sensitivity, and specificity. Besides, ablation experiments are conducted to verify the robustness of the algorithm and the effectiveness of each module. Finally, the results show that our method can classify the disease more accurately and faster than existing state-of-the-art approaches, which can provide a reliable basis for diagnosis.