The three-dimensional reconstruction method based on multi-view image motion structure restoration is quite time-consuming and has little robustness. Aiming at these issues, a new three-dimensional reconstruction method for physical geological specimen based on Kinect is proposed in this paper. First, the GrabCut algorithm is used to extract the foreground objects, combining with the original depth images to generate the point cloud corresponding point of view. Then, it uses RANSAC algorithm to perform point cloud alignment crudely based on the SIFT feature. Next, the ICP algorithm is improved by introducing the methods of rejecting outliers and adjusting the weight dynamically to perform fine registration of point cloud. Finally, a complete three-dimensional point cloud model is constructed. The experimental results demonstrate that the proposed method can reconstruct the three-dimensional point cloud of physical geological specimen quickly, excellently and robustly. When the specimens lack structural features, the proposed method can handle it effectively.