The research on tracking moving objects commonly uses color histogram as the main statistical characteristic which has a good result but is also vulnerable to illumination changes. The fuzzy color histogram in tracking can solve the above problems. Since fuzzy clustering has some inherent disadvantages, RSA-FCM algorithm was proposed to track moving objects, which used random sampling strategy in fuzzy clustering. Meanwhile, the proposed tracking method based on the adaptive fuzzy clustering model improved tracking accuracy. The experimental comparisons show that this algorithm behaves better than traditional method both on tracking accuracy and real-time performance.