It is important to understand the characteristics of rock porosity, pore size distribution, and pore connectivity for oil and gas exploration and exploitation, and the analysis and judgment of these characteristics need to rely on the image segmentation technology of rock thin sections. There are a large number of fine particles in the images of rock thin sections, and the edge features among these particles are very similar, which cannot be accurately distinguished. Meanwhile, uneven staining during section manufacturing will cause unbalanced color characteristics of the pores of the thin sections, resulting in the inability to segment. Therefore, to improve the segmentation effect of rock thin sections, this study proposes an improved segmentation algorithm based on U²Net. The main contents are as follows. (1) The U²Net network is adopted as the backbone to improve the model’s ability to express image features, and coordinate attention is combined to enhance the ability to express image features. (2) The introduction of a multi-scale feature extraction module enlarges the receptive field of the convolutional layers and enables the utilization of multi-scale feature information from the feature map. Empirical evaluations demonstrate that the proposed method outperforms conventional segmentation techniques and other state-of-the-art segmentation networks in small particle segmentation. Additionally, the proposed algorithm exhibits superior segmentation accuracy and robustness.