Accurate recognition of crop pests is essential for timely crop protection and treatment. However, crop pests in natural environment are small in size and have almost the same color as the environment. Moreover, crop pest images are affected by various factors such as noise and complex background. Therefore, it is difficult for existing crop pest recognition models related to deep learning to balance the requirements of recognition accuracy and robustness and be deployed on mobile devices with limited computational resources and low performance. In this study, ShuffleNetV2 0.5×, which has the fewest model parameters in the ShuffleNetV2 network structure, is selected as the benchmark network, and a lightweight crop pest recognition model based on high-order residual and attention mechanism (HOR-Shuffle-CANet) is designed. Specifically, the high-order residual in the early stage can provide rich pest features for the subsequent network layer, which significantly improves the recognition accuracy of the model. The coordinate attention (CA) mechanism can further suppress the background noise and focus on the key information about crop pests, which effectively enhances the robustness of the model. The bi-tempered logistic loss function with label smoothing regularization (LSR) can solve two shortcomings of logistic loss functions in training noisy data sets and make the model more adaptable to noise. The experimental results show that the HOR-Shuffle-CANet model achieves a recognition accuracy of 91.22% on the test dataset of ten types of common crop pest images in natural scenarios, which is 3.54 percentage points higher than the benchmark network. On the basis of maintaining lightweight computing, its recognition accuracy is also higher than that of the existing classical convolutional neural networks such as AlexNet, VGG-16, GoogLeNet, Xception, and ResNet-34, as well as lightweight network models such as MobileNetV3-Small, EfficientNet-B0, etc. Due to its high recognition accuracy, strong robustness, and excellent anti-interference performance, the proposed model can meet the practical application requirements of crop pest recognition.