In the face of large-scale image defects and irregular damage areas, existing image restoration methods often produce results with structural inconsistencies and blurry texture details. This study proposes an image restoration algorithm using the generated edge map and multi-scale feature fusion—MSFGAN (multi-scale feature network model based on edge condition). The model adopts a two-stage network design, using the edge map as a restoration condition to constrain the structural aspects of the restoration results. Firstly, the Canny operator is used to extract the edge map of the image to be restored, generating a complete edge map. Then, the complete edge map is combined with the image to be restored for image restoration. To address common issues in image restoration algorithms, an Attention Mechanism Multi-Fusion convolution block (AM block) is proposed, integrating an attention mechanism for feature extraction and fusion of damaged images. Skip connections are introduced in the decoder part of the image restoration network to fuse high-level semantics and low-level features, achieving high-quality detail and texture restoration. Test results on the CelebA and Places2 datasets show that MSFGAN has improved restoration quality compared to current methods. In the 20%–30% mask ratio, the average improvement of SSIM is 0.0291, and PSNR improvement is 1.535 dB. Ablation experiments validate the effectiveness of the proposed optimization and innovations in image restoration tasks.