In the electronic industry, defect detection of printed circuit board (PCB) has become more and more important. Some minor or irregular damage of PCBs is closely related to visual texture information, such as dense and complex PCB cables. Feature vectors extracted from the traditional convolutional neural network are prone to lose the intermediate visual feature information such as texture features, which results in an insignificant detection effect for minor and irregular damage. To solve this problem, this study proposes a PCB damage classification model based on a Siamese deep feature fusion residual network, and the model’s backbone network is ResNet50. In the feature extraction stage, the intermediate visual features such as texture information and the high-level semantic features finally output by the neural network are fused into a 32-dimensional feature vector. The similarity between the vectors of the two features is represented by the L2 distance, which is used to judge whether the PCB is defective. Triplet loss and cross-entropy loss are applied in the training phase, and the combination of multiple loss functions improves the accuracy of the network. The validity of the model is verified by experiments, and the accuracy on the test data set reaches (95.42±0.31)%. This indicates the feasibility of the model in PCB defect classification and detection.