To solve the difficulty in detecting the defects in fabrics of unknown styles in the automatic fabric defect detection algorithm, this study proposes a fabric defect detection method based on feature residuals. First, the defect residuals of the defective and template fabric images are fused with that of the normal unlabeled fabric image to generate a new defective fabric sample. Then, the improved feature extraction network uses the shared weight method to extract features from the defective and template fabrics and calculate the feature residuals. Finally, the ROIAlign method is used to mix the global context information and the region of interest for feature fusion. The fused features are subject to defect classification and location return. Experiments are separately conducted on two test sets containing and not containing fabrics of unknown styles. The results show that the improved algorithm can better eliminate the influence of fabric styles on the detection results. The accuracy is greatly improved in the test set that doesnot include unknown styles. In the test set containing unknown styles, defect detection maintains high accuracy. Compared with that of the general algorithm before the improvement, the final scores have increased by 15.4% and 16.2%, respectively.