Due to the scarcity of defective fabric samples in real-world applications and the subtle nature of anomaly regions, existing methods may fail to detect defect areas accurately, leading to localization errors. To leverage the Transformer attention mechanism’s ability to capture fine-grained features, this study proposes a dynamic attention-sharing mechanism tailored for fabric defect detection tasks, enhancing the model’s ability to capture subtle textures. Specifically, a fabric defect detection framework based on a mix multi-head attention (MMHA) reconstruction network is proposed. First, a pre-trained Transformer encoder is used to extract features from fabric images. Next, a bottleneck layer with MeanDropout is employed to reduce the model’s reliance on redundant features. Furthermore, a decoder combining MMHA with LLamaMLP is introduced to facilitate the dynamic selection of relevant fabric features, thus improving attention to critical fine-grained textures. Finally, the decoder loosely reconstructs combinations of multi-level features to achieve defect detection and localization. Extensive experiments conducted on publicly available datasets demonstrate that the proposed method achieves state-of-the-art performance in fabric defect detection. Experimental results show that the proposed method improves image-level indicators by 2.1% and 0.6% on two datasets, respectively, and achieves leading performance with pixel-level indicators of 96.0% and 60.5%.