Ship trajectory prediction is the premise and basis of realizing intelligent ship navigation. At present, most studies on ship trajectory prediction only rely on historical data of the automatic identification system (AIS), without using other sensor information on the ship. This study proposes a multi-modal trajectory prediction model S-Transformer. In this network, the seawater/land in the electronic chart is segmented as an auxiliary training target and integrated with the real Zhoushan port AIS data to train the model. In addition, the future ship sailing trajectory is predicted. The study also introduces Segment Recurrence to capture long-term dependencies of AIS data. The experimental results show that the S-Transformer has excellent prediction results in different ship-traveling situations and outperforms the unimodal benchmark model for related prediction tasks.