Computer vision is an important branch of machine learning at present, which requests much higher instantaneity and accuracy as the driverless and SI-Drive development. To optimize the current methods, the Faster Region-based Convolutional Neural Network (Faster R-CNN) is upgraded by adding SENet to it in this study. The upgraded Faster R-CNN model is applied in pedestrian detection. The new model does not only bring higher accuracy but also accomplish a better detection rate. To verify the new method, an examine was done in INRIA set and our set. The result shows that the upgraded model has a better detection performance on both accuracy and rate which can meet the related specifications of real-time pedestrian detection basically. Finally, the method was tested in the NVIDIA GTX1080Ti GPU. The results show that the mAP of upgraded model can achieve up to 92.7%, while the detection rate is up to 13.79 f/s under a relatively plain experimental condition. On the whole, the new model performs better than the traditional Faster R-CNN model.