In water surface garbage detection, large differences occur in target shape and scale, and it is difficult to distinguish the background and the small target. Thus, this study proposes the SPMYOLOv3 detection algorithm to identify surface garbage. Firstly, massive surface garbage datasets are collected and annotated, and an improved K-means clustering method is applied to generate the priori boxes that better match the datasets. Secondly, the SE-PPM module is added after the backbone network of YOLOv3 for strengthening the feature information of the target, ensuring that the target scale remains unchanged and the global information is preserved. The multidirectional FPN is then applied to fuse the feature maps of different scales so that the feature maps after fusion contain richer context information. Finally, the Focal Loss is adopted to compute the confidence loss of negative samples, which alleviates the imbalance of positive and negative samples in YOLOv3. The modified algorithm is tested on the water surface garbage dataset, and the results show that the accuracy of the modified algorithm is 3.96% higher than that of the original YOLOv3 algorithm.