The field of railway detection and monitoring generates massive image data, image scene classification is of great value for subsequent analysis and management. In this study, a visual scene classification model that combines Deep Convolutional Neural Networks (DCNN) and Grad Class Activation Mapping (Grad-CAM) is proposed, DCNN extract feature of railway scene classification image dataset by transfer learning method, Grad-CAM improves the interpretability of the classification model by calculating the weighted thermogram and activation scores of the categories. In the experiment, the effects of different DCNN structures on the performance of railway image scene classification tasks are compared, and visual interpretation of scene classification model is realized. At the same time, based on visualization method, an optimization process is proposed to improve model classification ability by reducing internal deviation of dataset, which verifies the effectiveness of the deep learning technology for image scene classification task.