A recognition algorithm based on static gestures and dynamic gestures was proposed in this paper, combined with mouse class functions of Windows API to achieve the mouse operation. Firstly, the original image was captured from the camera was converted to the binary image of high credibility through the image processing technology; and furthermore, calling static gesture recognition algorithm for recognition of spreading fingers number, according to the finger number, combined with mouse class functions of Windows API achieve mouse double-click and move function; finally, when the number of detected fingers was 5, calling dynamic gesture recognition algorithm to recognize gestures up down left right four directions, combined with the mouse class functions of Windows API to simulate the press and lift of left-right mouse button and wheel sliding. The experiment results show that the gesture recognition algorithm recognition rate reached 94.11%, for some development platform without a mouse or inconvenient to use the mouse, using gestures instead of the mouse has a certain value and significance.