Due to factors such as air pollution and smoking, pneumonia has become one of the diseases with the highest mortality rates in humans. The application of machine learning and deep learning technology in medical image detection has provided assistance for clinical experts in diagnosing various diseases. However, there is a lack of effective paired lung X-ray datasets, and existing methods for pneumonia detection use universal classification models that are not specific to pneumonia tasks. As a result, it is difficult to detect subtle differences between pneumonia images and normal images, resulting in recognition failure. Therefore, this study expands the normal images in the dataset through data cropping, rotation, and other methods and uses a 50-layer deep residual network to learn the shallow pneumonia features in chest X-rays. Then, through a two-layer dictionary, the pneumonia features learned by the residual network are further abstracted and learned, and subtle differences between different lung images are discovered. Finally, a pneumonia detection model is constructed by fusing the multi-level pneumonia features extracted from residual networks and dictionary learning. To validate the effectiveness of the algorithm, the performance of the pneumonia detection model is evaluated on the chest X-ray pneumonia dataset. According to the test results, the proposed model has a detection accuracy of 97.12%. In the indicator test, the score on the harmonic mean between accuracy and recall is 97.73%. Compared with existing methods, it has achieved higher recognition accuracy.