To solve unclear boundaries and incoherent, incomplete, or even lost segmentation results in the semantic segmentation task of colon polyp images, a colon polyp image segmentation network named colon polyp image segmentation network based on multi-scale features and contextual aggregation (MFCA-Net) is proposed. The network selects PvTv2 as the backbone network for feature extraction. The multi-scale feature complement module (MFCM) is designed to extract rich multi-scale local information and reduce the influence of polyp morphology changes on segmentation results. The global information enhancement module (GIEM) is designed. A large-kernel deep convolution embedded with positional attention is constructed to accurately locate polyps and improve the network’s ability to distinguish complex backgrounds. The high-level semantic-guided context aggregation module (HSCAM) is designed. It guides local features with global features, complements differences, and cross-fuses shallow details and deep semantic information to improve the coherence and integrity of segmentation. The boundary perception module (BPM) is designed. Boundary features are optimized by combining traditional image processing methods and deep learning methods to achieve fine-grained segmentation and obtain clearer boundaries. Experiments show that the proposed network obtains higher mDice and mIoU scores compared with current mainstream algorithms on the publicly available colon polyp image datasets such as Kvasir, ClinicDB, ColonDB, and ETIS, and has higher segmentation accuracy and robustness.