As an important load-bearing element of cable-stayed bridges, vibration testing of stay cables plays a key role in bridge health monitoring. Under ideal laboratory conditions, the traditional vibration detection algorithm with spatial phase can achieve high-accuracy measurement of structural vibration. However, in practical scenarios, environmental factors such as vehicles, wind excitation, and the angle between the cable and the ground can cause large errors in the measurement results. Therefore, the traditional algorithm is not suitable for cable vibration detection in these cases. To address this problem, this study proposes a cable vibration frequency detection algorithm based on directional adaptive complex steerable filters to precisely measure cable vibration in real scenarios. Firstly, the linear characteristics of the cable are used to detect the location of the cable and determine the main vibration direction of the cable; secondly, according to the vibration direction characteristics of the cable, a directional adaptive complex steerable filter is designed to decompose each frame of the video, so as to obtain the phase and amplitude spectra of the same direction at different scales and enhance the phase of the edge region of the cable. Finally, the spatial phase of each frame is averaged, and the phase sequences are arranged in time order to obtain the main frequency of cable vibration by Fourier transform. By comparing the results with those of acceleration sensors, it is proved that the proposed algorithm is highly robust and can meet the application requirements of bridge cable vibration measurement in real scenarios.