Vehicle object detection and tracking is a key step for the real-time monitoring and acquisition of traffic parameters in the video monitoring system of expressway. A vehicle tracking method based on trajectory temporal information with KCF algorithm is proposed to realize high precision continuous tracking. Firstly, data sets is established, the classification and detection of vehicle applicable to highway scenario are realized by using SSD algorithm based on deep learning. Then, based on the trajectory of temporal information, matching between object and trajectory is realized, and KCF algorithm is applied to forecast the missing object positioning, so as to realize the vehicle trajectory tracking. The experiment result shows that this tracking method has high precision, can adapt to many different scenarios, thus has high application value.