China is a traditional agricultural country. Agriculture is not only the foundation for the construction and development of the national economy, but also the guarantee for the normal, stable, and orderly operation of the society. However, the annual losses due to crop pests and diseases are huge, and the traditional methods for identifying crop pests and diseases are not ideal. At the same time, the rapid development of deep learning in recent years has made great progress in image classification and recognition. Therefore, this study constructs an image recognition model for crop pests and diseases through deep learning based methods, and improves the convolution network loss function for sample imbalance problems. The model can effectively identify crop pests and diseases and the accuracy of the model is further improved after optimizing the loss function.