The hesitant fuzzy C-means (HFCM) clustering algorithm has addressed the uncertainty between different pixel blocks in an image to some extent. However, as its objective function does not contain any local information, it is very sensitive to noise and cannot achieve good segmentation accuracy when the noise is large. This study proposes an image segmentation method based on improved HFCM (IHFCM) to address the above issues. Firstly, the completion method of hesitant fuzzy elements is given, and then a similarity measure between hesitant fuzzy elements is defined. Using the defined similarity measure, the study constructs a novel fuzzy factor and fuses it into the objective function of HFCM. The new fuzzy factor considers not only spatial information in the local window but also the similarity between pixels, balancing the impact of noise while preserving image details. Finally, experimental results on synthesized images, BSDS500 dataset images, and natural images show that the proposed IHFCM algorithm has good robustness to noise and improves segmentation accuracy.