With the increasing demand for human-computer interaction, the vision-based gesture recognition has attracted much attention in many fields. The depth image is more and more popular in target recognition field for its good performance. The gesture region is divided from the depth image and normalized to obtain a unified specification gesture binary image, and then it detects the gesture edge. A progressive Hough transform algorithm is proposed to detect the finger edge curve and extract the finger information in the gesture image. Then it extracts the features based on the edge curve and accords this to establish 3D histogram. Finally, the two features are fused, and the gesture classification is carried out by the minimum closed ball support vector machine (MEB-SVM) according to the obtained eigenvector. And the recognition rate on the test set is 96.6%. The new method does not depend on color, detail texture, and other information, with good robustness. And the method is faster to meet the needs of general applications.