Having studied the existing detection and classification algorithms, we design a scheme of fusion of improved Gaussian Mixture Model (GMM) and classification network (GoogLeNet) for vehicle detection and classification. In view of the inaccurate initialization and complex computation of GMM, we improve the algorithm of initialization models to increase the initialization efficiency. The five-frame difference method is used to execute the preliminary vehicle extraction. In the extracted vehicle area, GMM is used to get vehicle images, the five-frame difference method is combined with GMM to reduce the area of modeling and to increase the speed of vehicle detection and improve the real-time performance of the system. At last, we use GoogLeNet to execute the vehicle classification. The results show that the proposed methods have greatly improved the detection speed and recognition accuracy, and satisfy the real-time requirement of vehicle detection and recognition for surveillance video in real scenario.