In recent years, image segmentation applications based on convolutional neural networks (CNNs) have been quite extensive, and great progress has been made in feature extraction. However, with convolutional layers increasingly deep, the receptive field is continually enlarged, which makes the model lose local feature information and affects model performance. Using graph convolution network (GCN) to process information on graph data structures preserves local features without losing local information as the layers deepen. This study focuses on combining U-Net (a kind of symmetric full convolutional networks) feature extraction based on CNN structure with GCN-based image segmentation to extract global and local, shallow, and deep multi-scale feature sets for multimodal glioma MR sequence image segmentation. The process can be divided into two stages. Firstly, U-Net is used to extract features from brain multimodal glioma MR sequence images, and multiple pooling layers are used to realize multi-scale feature extraction and up-sampling for feature fusion, in which the bottom layer outputs lower-level features, and the top layer outputs more abstract high-level features. Secondly, the feature map data obtained by U-Net is converted into the graph structure data required by GCN by dilating neighborhood and sparsification, and the image segmentation problem is converted into the graph node classification problem. Lastly, the graph structure data is classified by cosine similarity. Experimental results achieved segmentation accuracy of 0.996 and sensitivity of 0.892 on the BraTS 2018 public database. Compared with other deep learning models, this method, by fusing multi-scale features and using GCN to establish topological connections between high and low level features, ensures that local information is not lost to achieve better segmentation results, which meets the needs of analyzing clinical glioma MR images, and then effectively improves the diagnostic accuracy of gliomas.