Taking the laser point cloud of pavement elevation as the research object, this study proposes a method for extracting pavement potholes based on normal vector distance. Firstly, the cloud data of pavement elevation points are cleaned. Secondly, the PCA method in the adaptive optimal neighborhood is used to estimate the normal vector of the pavement point cloud data. The normal distance from the sampling point in the pavement point cloud to the tangent plane of its local quadric surface is calculated as the normal vector distance to describe the three-dimensional spatial features of the sampling points. Next, threshold segmentation is employed to automatically extract the pothole point cloud set, which is then segmented by the Mean-Shift clustering algorithm to obtain multiple pothole point sets. Finally, for each pothole point set, the Alpha Shape algorithm is used to extract pothole boundary points that are fitted by cubic spline interpolation three times to obtain the pothole contour. On this basis, the pothole size (length, width, and depth) and area are calculated. Experiments are carried out on regular pothole model point cloud data and real pavement point cloud data. The calculation shows that the average relative errors of pothole depth extracted by this method are 2.7% and 4.7%, respectively, and the average relative errors of pothole area extracted by this method are 6.8% and 4.3%, respectively. The experimental results show that the proposed method can accurately extract the boundary points and size information of pavement potholes and has a strong anti-interference ability for the recognition and extraction of irregular-shaped potholes.