Aiming at the image of ship target is small and far from the harbor surveillance, the interference of sea noise and other characteristics, a background suppression algorithm based on morphological filtering is proposed. According to the geometric features of the ship, the project first uses the weighted morphological filtering of multi group structure elements to separate the ship target and the sea background, then the adaptive and fast median filter based on neighborhood is used to filter the impulse noise, finally, due to the sea clutter in the frames is discontinuous and the area is lesser, thus the connected domain computing to eliminate the ship's model about morphological feature is also used. The experimental results show that the proposed detection algorithm can effectively suppress the background noise, the sea clutter, and quickly detect the ship target in the actual port video.