Maintaining a safe social distance is one of the important means to effectively prevent the spread of the virus. Moreover, it can not only reduce the number of infected people and ease the medical burden but also greatly lower the mortality rate. On the basis of the you only look once version 4 (YOLOv4) framework, the lightweight network E-GhostNet is used to replace the CSPDarknet-53 in the original network. The E-GhostNet network establishes a relationship between the input data and the output features generated by the original Ghost module, thereby enabling the network to capture contextual features. Then, the coordinate attention (CA) mechanism is introduced to E-GhostNet to enhance the model’s attention on effective features. In addition, the complete intersection over union (CIoU) loss function is replaced by the soft intersection over union (SIoU) loss function to obtain a faster convergence speed and optimization effect. Finally, the DeepSORT multi-target tracking algorithm is utilized to detect and label pedestrians, and affine transformation (IPM) is employed to determine the violation of the required distance between pedestrians. The experimental results show that the network achieves real-time pedestrian distance detection with a detection speed of 40 FPS and an accuracy of 85.71%, which is 2.57% higher than that of the original GhostNet algorithm.