In this paper, a content-based image retrieval algorithm using texton and color is proposed. Firstly, it converts color image from RGB color space to HSV color space and quantizes the color image into 256 colors. Then texton-images can be extracted based on the five texton types which are defined for image analysis. Finally, color features of texton-images are represented by color histogram whose similarity can be measured by the improved cross color histogram. Experimental results indicate that: the proposed algorithm can effectively remove the effects of the background color and can make a better description on the color image texture and edge features possessing higher rates of precision and recall compared with the edge detection method.