Single-object pedestrian tracking is one of the most basic and widely studied tasks in computer vision object tracking. However, most of the correlation filtering algorithms and deep learning algorithms currently used have insufficient tracking accuracy and real-time tracking performance. To solve the above problems, we propose a real-time single-object pedestrian tracking algorithm based on deep and shallow feature fusion. Firstly, this algorithm predicts the object location by Kalman filters and extracts the shallow color features of the object by calculating the four-part color histogram, and the prediction similarity is obtained to judge the reliability of prediction results. Then, YOLOv4 is used as a detector to extract deep features of the object and then calculate the distance metric of motion information and appearance information. Meanwhile, the shallow color features of the detection object are extracted to calculate the similarity distance metric, and the weighted fusion of the feature distance metric is employed to match the detection object and update the tracking trajectory. Finally, a trajectory updating strategy is put forward to coordinate the calling relationship between the prediction block and the detection block and to achieve a balance between tracking accuracy and speed. Testing experiments are conducted on the OTB100 and LaSOT datasets. The experimental results demonstrate that the tracking accuracy of the proposed algorithm on the above datasets reaches 0.581 and 0.453, respectively, and the tracking speed tested on GPU can achieve 33.64 FPS and 35.32 FPS, respectively, which meets the requirements of real-time tracking.