Pain is a personal experience that is generally divided into acute pain and chronic pain, which can result from injury, illness, surgery, or other health problems. If pain is not treated in time, it will cause great harm to the patient's physical and psychological health. Not all patients, e.g. dementia, are able to self-report pain. The continuity and objectivity of the assessment cannot be guaranteed when medical care personnel assessing patient's pain. Therefore, the demand for automatic pain recognition system is increasing. In the past decade, many researchers have made breakthroughs in this area. This paper reviews the automatic pain recognition system. On the one hand, it describes the structural composition of the automatic pain recognition system, including data acquisition, data preprocessing, feature extraction, and classification. On the other hand, it summarizes a great number of techniques from pain modal representation, i.e. behavior, speech, physiology, and multi-modal fusion. This paper also discusses the key technologies in automatic pain recognition system, and analyzes several challenges and directions in the field.