For better use of high-quality graphite resources, this paper proposed a graphite classification and recognition algorithm based on transfer learning and focal loss convolutional neural network (CNN). The offline expansion and online enhancement of the self-built initial data set can effectively expand the data set and reduce the overfitting risk of deep CNN. With VGG16, ResNet34 and MobileNet V2 as basic models, a new output module is redesigned and loaded into the full connection layer, which improves the generalization ability and robustness of the model. Combined with the focal loss function, the hyperparameters of the model are modified and trained on the graphite data set. The simulation results show that the proposed method has the accuracy improved to above 95% with faster convergence and a more stable model, which proves the feasibility and effectiveness of the proposed algorithm.