To achieve automatic garbage sorting and ensure correct garbage classification, this study proposes a lightweight garbage detection algorithm based on you only look once version 4 (YOLOv4). The algorithm replaces CSPDarknet53, the backbone network in YOLOv4, with a MobileNetV3 network after level adjustment, thereby making the network architecture more suitable for the YOLOv4 network and improving the detection speed of the network. Subsequently, a new bottleneck is designed by drawing on the idea of the Ghost module and the MobileNeXt network structure to replace the bottleneck in the backbone network and boost the detection accuracy of the model. Then, a large residual edge structure is added to the backbone network to raise the detection accuracy of the network. Furthermore, a coordinate attention (CA) mechanism is added before the neck network to further enhance the detection accuracy of the network. Finally, to prevent the K-means algorithm from falling into a local extremum in the clustering process, this study employs the bisecting K-means algorithm to re-cluster anchor boxes for the garbage detection dataset. Experimental results show that the redesigned network achieves a mean average precision (mAP) close to that of the YOLOv4 network but reduces the number of parameters by 89%. Moreover, it improves detection speed by 51% to 67.5 FPS (on NVIDIA GeForce RTX 3060). It can thus be deployed to embedded devices with low computing power and memory.