At present, most of the pointer recognition methods are based on the traditional image processing technology, and the extraction process is complicated with many steps. To effectively solve the problems of difficult pointer axis extraction and poor reading recognition accuracy of a pointer instrument, this study introduces a method of pointer instrument recognition based on deep learning. First, the Faster R-CNN algorithm is used to detect the instrument disk, and then the method based on deep learning is adopted to detect the pointer. According to the position information of the target frame, the pointer image is obtained by clipping. The final reading of the instrument is identified by binarization, thinning, Hough transform, and the least square fitting line. Compared with the traditional image processing directly on the image of the panel target frame or the original image, this method greatly reduces the interference in the process of locating the line where the pointer axis is located. The experimental results show that the average accuracy of pointer detection based on deep learning proposed in this study is up to 96.55%. It has high accuracy and stability for pointer detection of the pointer instrument under a complex background.