Existing image restoration methods generally suffer from structural misalignment and blurred edges of the restored area, which is due to the under-utilization of structural information in known areas during image restoration. To this end, a color clustering-based image restoration algorithm with an encoder-decoder structure is proposed in this study. The algorithm uses a progressive image restoration network structure, taking the results of the images after color clustering as input, and the images processed by the clustering algorithm better preserve the structural information. Therefore, the structural information can be fully utilized in the subsequent image texture restoration process. Meanwhile, to further reduce the computational overhead of the network, a cross-attention mechanism is introduced to model the global dependence of images from both horizontal and vertical dimensions. The experimental results show that the image restoration algorithm proposed in this study can effectively alleviate the blurring of the edges in the restored areas, and compared with several mainstream image restoration algorithms, the proposed image restoration algorithm can produce more realistic output results in the case of large missing areas.