With the cost reduction and popularization of digital cameras, non-contact structural health monitoring technology based on computer vision has been paid more and more attention to. Traditional contact measuring instruments may cause mass loads on lightweight structures, and the installation and maintenance on large civil structures are costly and time-consuming, especially for long-term applications. As an alternative non-contact method, the computer vision method using digital cameras has relatively low cost and is flexible. In order to solve the problem of low accuracy and stability of the traditional optical flow method in micro displacement measurement, this study proposes an improved phase-based optical flow method to calculate the vibration displacement of structures. In this method, the partial complex steerable filter constructed in the phase-based video motion magnification method is introduced to convolute with the video sequence image to obtain the local spatial phase information of structures. Then, the phase-based optical flow method is used to calculate the pixel displacement of structural vibration. Finally, the pixel displacement is transformed into real displacement by noise reduction and the scale factor method. The experimental results show that the proposed method has higher accuracy and stability than other optical flow methods.