Fabric defect detection is an important link in the quality management of the textile industry. Accurate and fast fabric defect detection on embedded devices can effectively reduce the detection cost, thus being of great value. Considering the structural characteristics of colored fabric defects in actual production, such as a complex background, large differences in the quantity of defects, an extremely high aspect ratio, and a high proportion of small defects, a colored fabric defect detection method based on a lightweight model is proposed and implemented on an embedded circuit board Raspberry Pi 4B. The lightweight feature extraction network ShuffleNetV2 is first used to extract the features of colored fabric defects on the basis of the one-stage target detection network, you only look once (YOLO), so as to reduce the complexity of the network structure and the number of parameters and improve the detection speed. Then, the detection head is decoupled to separate the classification and localization tasks so that the convergence speed of the model can be improved. In addition, the complete intersection over union (CIoU) is introduced as the loss function of defect location regression to improve the accuracy of defect location. The experimental results show that the proposed algorithm can achieve a detection speed of 8.6 FPS on Raspberry Pi 4B, which can meet the needs of the textile industry.