Gesture recognition based on RGB images is widely used in the field of human-computer interaction because of its low requirements for equipment and convenient data collection. In the process of gesture recognition and interaction of RGB images, on the one hand, the efficiency of gesture segmentation based on skin color information is low due to the illumination influence of RGB gesture images during collection; on the other hand, the interactive gestures cognized by users are different from those designed by designers, which leads to poor feedback of users’ interaction experience. In this study, we systematically optimize the above two problems. Firstly, users’ cognition is linked with the interactive gesture design principles to establish a gesture consensus set. Secondly, the gesture image is subjected to color balancing, and an elliptical skin color model is used to segment the gesture area. Then, the binarized gesture images are input into a MobileNet-V2 lightweight convolutional neural network to calculate the gesture recognition rate. The combination of end-user subjective evaluation of gestures and gesture recognition technology can systematically design gestures for interactive tasks, reduce the cognitive deviation of users in the actual interaction process, and improve the usability and efficiency of interactive systems.