The original damage-free pavement image is of great significance for analyzing the evolution details of pavement damages and formulating the next maintenance plan. However, the initial state corresponding to a pavement crack image cannot be obtained in field acquisition. To obtain the corresponding damage-free pavement image, this study proposes a deep image prior-based unsupervised crack image inpainting algorithm for asphalt pavements that enables efficient semantic-level inpainting of cracks in a single pavement image. Specifically, a robust principal component analysis algorithm is used to remove the vertical stripe noise on the surface of the pavement crack image. Then, the maximum between-class variance method and morphological processing are employed to obtain a binary mask image of the crack area. Finally, the crack area is inpainted with the proposed deep image prior-based inpainting algorithm to obtain the final damage-free pavement image. The proposed method is evaluated on a dataset of self-collected pavement crack images. The experimental results show that the proposed method can effectively achieve semantic-level inpainting of pavement crack images as it significantly improves the peak signal-to-noise ratio and structural similarity to an average of 43.3823 dB and 0.9834, respectively, compared with those of the traditional methods and it also achieves a high speed.