With regard to erroneous detection and low efficiency problems in the traditional manual detection of fabric defects, a self-adaptive detection method for fabric defect contours is proposed based on the visual perceptual mechanism. Firstly, the mechanism of visual information processing with the receptive field of the retina in the visual system is simulated to filter the fabric defect images and enhance the defects. Secondly, the edge information in the enhanced fabric defect images is detected according to the edge detection model of fabric defects proposed on the basis of the orientation selectivity mechanism in the area of primary visual cortex (V1). Lastly, the fabric defect contour is extracted by re-processing the detected edge images through self-adaptive threshold selection. To validate the effectiveness and accuracy of our method, this paper tests and compares four kinds of fabric defects from both qualitative and quantitative aspects. The results show that the proposed method performs well in detecting the contour information of fabric defects. This method not only acquires relatively high-quality detection images of fabric defects but also selects parameters adaptively in the whole process. It avoids the effect of subjective factors and implies practical application value.