Compared with pixel-based building extraction methods, object-oriented methods can reduce the phenomena of “the same spectrum for different objects” and “different spectra for the same object” and improve extraction accuracy. To address the curse of feature dimensionality due to numerous features of remote sensing images, this study proposes an object-oriented feature optimization method for building extraction. First of all, minimum error automatic threshold segmentation is combined with multi-scale segmentation to optimize the segmentation technology. Then, features are selected by the Relief algorithm and fast correlation-based filter (FCBF) algorithm to construct the optimal feature subset. Finally, buildings are extracted by the random forest method, and building boundaries are optimized by the minimum bounding rectangle method. The results show that the importance of features varies greatly. An overall accuracy of 0.93 is achieved by building extraction based on the optimal feature subset, and the Kappa coefficient is 0.91, which is significantly higher than the extraction results of the original feature set and the optimized feature set.