To improve the identification accuracy of ordinary neural convolutional networks for tomato leaf disease, a new network based on the multi-scale fusion attention mechanism (MIPSANet) is proposed. The lightweight network is used as the main framework to reduce the network parameters in this network. To increase the depth and width of the network, the Inception structure is added to extract multi-scale feature information of data. Meanwhile, a more elaborate dual attention mechanism, polarized self-attention (PSA), is used in this process as a plug-and-play module to be embedded in the whole model, which improves the expressive power of important feature points. The lightweight PSA modules are also suitable for this model. A full connection layer is added after the convolution for classification. The proposed MIPSANet is applied to conduct experiments on Kaggle public dataset, tomato leaves dataset, with 30 batches of training, achieving an accuracy rate of 91.05%. The results show that this network is strikingly effective in the classification of tomato leaf diseases compared with other networks, which provides some reference value for the network structure and parameter configuration of the classification network.