Most of the current vehicle recognition methods rely on deep learning to directly input image data for training, thus obtaining a deep network. Due to the perspective distortion and scale change of an image, a large number of different types of data have to be used for training, without obtaining the vehicle-related physical information. To address the above problems, we propose a method of vehicle fine-grained recognition based on inverse projection space. First, the three-dimensional bounding boxes are constructed for vehicles under projection of a monocular camera by calibration information and geometric constraints. Second, the bounding boxes are unfolded to obtain normalized and standardized three-dimensional data in the inverse projection space. Finally, a deep convolutional network is introduced to obtain vehicle recognition results and its corresponding physical sizes of five common types of vehicles by training these standardized data. Experimental results show that, compared with traditional end-to-end vehicle recognition methods based on deep learning, the proposed method can effectively improve the accuracy of recognition while using less training data, and the three-dimensional physical sizes of vehicles can also be obtained simultaneously.