Effectively identifying the different group of human in an image or video is an important part of intelligent image analysis. It is how to obtain “effective features” in the image. Based on the convolution neural network model, this study proposes a multi-model fusion convolution neural network method. The model trained by ImageNet participates in the initialization of the weights of the neural network model, achieves more effective features on the premise of effectively saving time and resource calculating costs. Experiments prove that the model can maintain the recognition accuracy of adult males, adult females, and children in natural scenes at about 85%, which improves the accuracy and reliability of group classification.