The detection and recognition of ship numbers are of great significance for the intelligent management of ports and can solve the time-consuming and labor-intensive problems caused by the traditional manual supervision of fishing boats. Since the ship number plates feature non-uniform hanging positions, background colors, and numbers of characters, this study proposes a two-stage detection and recognition method with two models. First, the study introduces a DP-DBNet ship number location detection model that combines dual path networks (DPN) with a differentiable binarization network (DBNet). Secondly, the study presents an MHA-CRNN ship number recognition model that combines the multi-head-attention mechanism (MHA) with the improved convolutional recurrent neural network (CRNN). Finally, this study uses the data from the new modern smart fishing port project in Zhifu District of Yantai and carries out an algorithm comparison experiment analysis. The experimental results show that the two-stage recognition method with two models can make the recognition accuracy rate of the ship number reach 76.39%, which fully proves the effectiveness and application value of the model in marine port management.