The textile industry is a pillar of China’s economy. In the supply chain of fabrics, fabric defects largely affect their quality. At present, textile production enterprises mainly rely on people’s eyes to check the quality of fabrics. This method has high cost of employment, low efficiency, and a high rate of missed and false detections, failing to meet the requirement for high-speed industrial development. This study enhances the data to address the uneven number of categories in the data set. Categories of fabric defects are identified by the Faster RCNN model, and the RPN network in the Faster RCNN model is improved regarding the small concentrated defect targets in the dataset. In addition, this study develops a fabric defect recognition system to display categories of fabric defects and precisely locate the defects. Through the comparison of experimental results, the average detection rate of this method is 79.3%, which is 5.75% higher than that of Fast RCNN.