Automatically identifying the types and counting the numbers of ships on waterways is of great significance for the construction of “smart waterways”, intelligent early warning regarding water surface, and navigation decision support. In this study, ship sample images are trained with the YOLOv3 pre-training model, and the detection model for ships on waterways is developed after parameter adjustment and optimization. Then, considering that the deep learning model is good at extracting target features, this study combines the target HSV color histogram features and LBP local features to achieve the target selection. In view of common drift and jitter of tracking targets, a correction network is designed with the integration of regression-based direction judgment and target counting with a variable time window, which realizes the automatic detection, tracking and self-correcting counting of moving targets on water surface. The test results show that the proposed method is stable and robust, suitable for automatically analyzing channel videos and extracting statistical data.