In view of the fact that the current algorithm can not meet the needs of fabric defect classification detection with periodic pattern characteristics, a deep convolutional neural network fabric defect detection algorithm based on Fisher criterion is proposed. First, a small Deep Convolutional Neural Network (DCNN) is designed by using depthwise separable convolution. Further, the Softmax loss function of DCNN adds Fisher criterion constraint and updates the whole network parameters through gradient algorithm to get Deep Convolutional Neural Network (FDCNN). Finally, the classification rates of TILDA and pink plaid fabric database were 98.14% and 98.55%. The experimental results show that the FDCNN model can not only effectively reduce network parameters and running time, but also improve fabric defect classification rate.