Human-computer interaction is inseparable from emotion recognition. At present, there is a problem of low recognition rate and poor robustness both in single-modality emotion recognition and multi-physiological parameters fusion emotion recognition. Therefore, a fusion emotion recognition system based on two different types of signals is proposed, that is, a dual-modality emotion recognition system that integrates physiological parameters of skin electrical signals and text information. Firstly, by collecting and analyzing the characteristic parameters of the corresponding emotional skin electrical signals and the emotional keyword features of the textual information, the artificial neural network algorithm and the Gaussian mixture model algorithm are designed as a single mode emotion classifier, respectively. The Gaussian mixture model weights the decision layers. Experimental results show that this kind of fusion system has higher accuracy than multi-modality emotion recognition combined with single mode and multiple physiological parameters. Therefore, based on the two different types of emotional characteristics of skin electrical signals and text information, an emotion recognition system with high recognition rate and sound robustness can be constructed.