Using YOLOv5s as the object detector in DeepSORT presents challenges, including high computational costs, complex in structure, and exhibits limitations in detection accuracy. Firstly, the GhostNet lightweight module is introduced to lightweight the YOLOv5s model, reducing both the number of parameters and computational load of the model to meet the deployment requirements for mobile devices. Secondly, the ECA attention mechanism is incorporated to enhance the model’s perceptual capability, improving detection performance. Lastly, knowledge distillation is applied to the YOLOv5s model to further enhance detection accuracy. The improved YOLOv5s algorithm shows a 2% increase in precision, 1% increase in recall, and 0.8% increase in mAP@0.5, compared to the original algorithm. The model’s parameter count is reduced by 47%, and its complexity by 48%. When the enhanced YOLOv5s is combined with the DeepSORT algorithm, MOTA, MOTP, and IDF1 are improved by 1.2%, 3.1%, and 2.7%, respectively, while IDS is reduced by 35%, compared to the original version. Experimental results verify that the improved YOLOv5s, as a detector, enhances detection speed, reduces pedestrian ID switching, and can be effectively applied to pedestrian tracking.