Currently, most of vehicle structured information was obtained through multiple steps, which caused the problems such as fussy training, limited training data in each step, and the accumulation of error in processing. Therefore, multi-task learning was applied to union the structured information extraction in a single neural network, and shared feature extraction structure can help to reduce the error accumulation in various processing. A loss function of multi-task learning was put forward for end-to-end network training. For solving the training data limitation problem, a new dataset augmentation and combination approach was advanced. The experimental results on the dataset KITTI show that the mAP (mean of the Average Precision) of VSENet achieves 93.82%, and the processing speed can satisfy the real-time request. Compared with multi-step vehicle structure information extraction method, the proposed approach reduces 60% of average processing time and achieves similar or better performance. These results show that the proposed method has certain advancement and is effective.