A lightweight YOLOv4 algorithm, MA-YOLOv4, is proposed to enhance feature fusion for addressing problems of complex detection networks, a large number of parameters, and poor real-time detection in deep learning-based maritime ship target detection tasks. Firstly, MobileNetv3 is employed to replace the backbone network, and a new activation function SiLU is introduced. The depthwise separable convolution is applied to replace the ordinary 3×3 convolution to reduce the number of network parameters. Secondly, an adaptive spatial feature fusion module is added to enhance feature fusion. Finally, the MDK-means clustering algorithm is adopted to get the anchor frame suitable for the ship target, and the Ship7000 dataset is utilized for training and evaluation. Experimental results show that compared with YOLOv4, the improved algorithm can reduce the number of model parameters by 82% and increase mAP by 2.57% and FPS by 30 f/s, which can achieve high-precision real-time detection of ships at sea.