Specular highlight removal is a hot topic in the field of computer vision. Existing methods based on dichromatic reflection model,which separate diffuse and specular reflection components to remove specular highlights in a single image, tend to cause color distortion and texture loss. To relieve this problem, the pixel clustering algorithm is improved by using pixel intensity ratio to remove specular highlight, which can more accurately classify pixels and improve the image color distortion. Firstly, the difference between the original image and the single channel image of the minimum intensity value is calculated, to obtain the specular-free image. Secondly, the minimum and maximum diffuse chromaticity values for each pixel in the highlight area is estimated based on the specular-free image. Finally, the distribution pattern of the pixels in the highlight area are analyzed in a minimum-maximum chromaticity space and clustered by x-means method. The specular components of highlight area pixels can be easily separated by using the estimated intensity ratio of diffuse pixels, thereby getting a non-highlight image. Experimental results show that, compared with the existing method, the peak signal-to-noise ratio increases by 2% to 4% on average, and the image color distortion and texture loss are improved with better visual effects.