In order to solve the problem that object tracking only based on color information always fails under the conditions of cluttered backgrounds, a tracking algorithm combining the color and structural information was proposed. Weighted color histogram based on HSV was used to describe the color model of the target, a structural model was developed by using target gray level image. The two features were fused in the frame of particle filter, the link is the calculation of the particle weight. Meanwhile, the weighs of fusing for color and structural were adjusted adaptively. The experimental results show that the proposed algorithm has more stability and higher accuracy.