In order to improve recognition accuracy of the speech emotion recognition system effectively, a speech emotion recognition algorithm based on SVM is proposed. In the proposed algorithm, some parameters extracted from speech signals, such as: energy, pitch frequency and formant, are used as emotional features. Furthermore, an emotion recognition model is established with SVM method. Simulation environment experiential results reveal that the recognition ratio of the proposed algorithm obtains the relative increasing of 7.06% and 7.21% compared with artificial neural networks such as ACON (All Class in one Network, “one to many”) and OCON (One class in one network, “one to one”) methods. The result of the experiment shows that the speech emotion recognition algorithm based on SVM can improve the performance of the emotion recognition system effectively.