Traffic sign recognition equipment has low power consumption and hardware performance, while the existing convolutional neural network model has high memory footprint, slow training speed, and high computational overhead, which cannot be applied to the recognition equipment. To solve this problem, in order to reduce model storage and improve training speed, deep separation convolution and mixed wash grouping convolution are introduced and combined with the ultimate learning machine. Two lightweight convolutional neural network models are proposed:DSC-ELM model and SGC-ELM model. The proposed models use the lightweight convolutional neural network to extract the features, and then send the features to the extreme learning machine for classification, which solve the problem of slow parameter training in the full connection layer of the convolutional neural network. The new models combine the advantages of lightweight convolutional neural network model with low memory footprint, good feature extraction quality, good generalization of ELM, and fast training and classification. Experimental results show that compared with other models, the hybrid model can accomplish traffic sign recognition tasks more quickly and accurately.