The optimized Mask R-CNN network based on deep learning is used to visual detection of the tiny defects on gears. Firstly, by comparing the detection effects of five kinds of residual neural network, resnet-101 is selected as the image sharing feature extraction network. Then, the detection rate for missing tooth is correspondingly improved by eliminating the unreasonable 3×3 convolution of feature map P₅ in the feature pyramid network. Finally, in order to effectively train the region proposal network, the appropriate anchor size and aspect ratio are set according to small fluctuation of annotation box in the designed sample labeling scheme. The optimized Mask R-CNN network eventually achieved 98.2% detection rate for missing tooth on gears.