In view of the variety of buildings, and the confusion with the surrounding environment in the remote sensing image, the traditional methods are difficult to extract the buildings efficiently and accurately. This study proposes a method of building automatic extraction method based on the improved Mask-RCNN. In the proposed method, an improved Mask-RCNN network model framework is constructed by using PyTorch deep learning framework, path aggregation network and feature enhancement function are added in the network design, and multi-threaded iterative training and model optimization learning are conducted for the Inria aerial image tag data set by means of supervision and migration learning, thus automatic and accurate segmentation and extraction of buildings are achieved. The proposed method is compared with SVM, FCN, U-net, Mask-RCNN, and other building extraction algorithms with different open-source datasets. The experimental results show that the proposed method has competitive performance, which can extract buildings more efficiently, accurately, and quickly in different open-source datasets, and the four evaluation indexes of mAP, mRecall, mPrecision, and F1 scores extracted in the same dataset are better than the compared algorithms.