The tower crane hook video has poor video stabilization due to complex working conditions at the construction site. To solve the problem, this study proposes a hook video stabilization algorithm integrating the improved ORB feature matching and fixed-lag Kalman filter. In the image motion estimation, the classical ORB algorithm is improved by adopting image chunking and adaptive thresholding for feature point extraction and introducing the image quadratic tree algorithm to improve the uniformity of image feature point distribution; on this basis, the background compensation combined with inter-frame difference method is used to quickly identify local motion targets and reject them, which improves the accuracy of global motion parameter estimation; in the motion filter and compensation stage, a fixed-lag Kalman filter algorithm is used to remove the random jitter components, so as to obtain the motion compensation parameters for video stabilization, thus achieving video monitoring stabilization of the tower crane hook visualization system. The experimental results show that compared with that of the classical ORB and Kalman filter algorithm for image stabilization, the inter-frame transform fidelity (ITF) of the proposed algorithm is improved by about 9.12%, and the average value of structural similarity ($ \overline {{\text{SSIM}}} $) is improved by about 2.75%, resulting in a better image stabilization effect, and inter-frame processing speed (FPS) reaches 29.65 f/s, meeting the real-time monitoring requirements of the tower crane.