Intelligent tongue diagnosis is of great significance in assisting doctors in medical treatment. At present, intelligent tongue diagnosis is mainly focused on the prediction and classification of single tongue image features, making it difficult to provide substantial help in the diagnostic process. To make up for this deficiency, research of accurate prediction and classification is carried out from the level of tongue image syndrome to assist doctors in diagnosing diseases. The TUNet is used to segment the tongue, and a parallel residual network PMANet integrated with the multi-attention mechanism is proposed to classify the syndrome of tongue image. the pixel accuracy (PA), mean intersection over union (MIoU) and Dice coefficient of TUNet reach 99.7%, 98.4%, and 99.2%, respectively, improved by 3.2%, 9.0%, and 4.8% compared with the baseline U-Net. In the research of tongue image syndrome classification, PMA’s total amount of parameters is 12.34M, slightly higher than that of EfficientNet, and its total amount of floating-point calculations is 1.021G, significantly lower than all compared networks. Under the background of a lower amount of both parameters and floating-point calculations, the classification accuracy of PMANet reaches 95.7%, achieving a balance between precision, parameter amount, and floating-point calculations amount. This method provides support for the research of intelligent tongue diagnosis and is expected to promote the modernization of TCM tongue diagnosis.