Two-dimensional (2D) face recognition is greatly affected by illumination, occlusion, and attitude. To overcome these shortcomings, this study proposes a 3D face recognition algorithm with multi-modal fusion based on deep learning. Firstly, the convolutional autoencoder fuses the color image and the depth map, and the fused image is input to the network for pre-training. In addition, a new loss function cluster loss is designed for pre-training in combination with the Softmax loss, so as to obtain a highly accurate model. Then, transfer learning is employed to fine-tune the pre-trained model, and thus a lightweight neural network model is obtained. The processed original dataset is used as the test set, and the identification accuracy of the test reaches 96.37%. Experimental results verify that the proposed method makes up for some shortcomings of 2D face recognition, and it is less affected by illumination and occlusion. Compared with 3D face recognition using high-precision 3D face images, the proposed algorithm is faster and more robust.