For low-level computer vision tasks, image de-raining has always been a hot issue. However, due to the uneven density of rain lines in the image, it is a very challenging issue to remove rain from a single image. Attention to the target image often requires attention to two parts:the overall structure of the image and the details of the image. In this regard, a novel multi-stream feature fusion convolutional neural network algorithm is proposed. It presents superior performance through various network frameworks. The network algorithm uses three branch networks to extract complex multi-directional rain line features, concat features and combines with the original image to remove rain. The detail enhanced network can obtain high quality without rain. The de-raining performance under the synthesized dataset and the real rain dataset indicates that the proposed algorithm can preserve more details while removing the rain line than the existing deep learning-based rain removal algorithm, and it can keep the high quality of the picture.