The two major detection methods for table tennis ball by table tennis robot's visual system are based on motion analysis or detection by single color models. The motion analysis methods have poor robustness in moving background, while the similar color and light intensity change will interfere the performance of detection methods based on single color model. A quick detection method based on multi-color models was proposed, the RGB color model and HSV color model are combined to extract the region of table tennis ball as a binary image, and the center of ball is positioned by calculating the barycenter. On this basis, Region Of Interest (ROI) technology based on previous frame position was proposed for real-time tracking. Experiments show that this method can accurate positioning table tennis ball in complex environment, the algorithm processing time is less than 10 ms, and positioning error is less than 20 mm. Its precision meets the demand of table tennis robot.