For the problems of weak music feature mining, complex deep learning classification models, and long training time, a music genre classification model based on spectrogram enhancement and convolutional neural network-based broad learning system (CNNBLS) is designed. This model first enhances the Mel spectrogram by randomly masking part of frequency channels in SpecAugment and then uses the cut Mel spectrogram as the input of CNNBLS. At the same time, exponential linear unit functions (ELUs) are fused into the convolutional layer of CNNBLS to enhance its classification accuracy. Compared to other machine learning network frameworks, CNNBLS can achieve higher classification accuracy with less training time. In addition, CNNBLS can quickly learn incremental data. The experimental results show that the non-incremental model of CNNBLS can achieve a classification accuracy of 90.06% after training 400 pieces of music data, while the incremental model of Incremental-CNNBLS can achieve a classification accuracy of 91.53% after adding 400 pieces of training data.