Image semantic segmentation methods based on deep convolutional neural network requires a large number of pixel-level annotation training data, but the labeling process is time-consuming and laborious. In this study, a semi-supervised image semantic segmentation method with encoder-decoder based on generative adversarial networks is proposed, in which the encoder-decoder as the generator. The entire network is trained by coupling the standard multi-class cross entropy loss with the adversarial loss. In order to make full use of the rich semantic information contained in the shallow layers, this study puts the features of multi-scales in the encoder into the classifier, and fuses the obtained classification results with different granularities to optimize the object boundaries. In addition, the discriminator enables semi-supervised learning by discovering the trusted regions in the unlabeled data segmentation results to provide additional supervisory signals. Experiments on PASCAL VOC 2012 and Cityscapes show that the proposed method is superior to the existing semi-supervised image semantic segmentation methods.