Since skin melanoma images are featured by large intraclass differences and small sample datasets, the deep residual network can effectively solve the problem of over-fitting during training and improve the recognition accuracy. However, the network model has many training parameters and high time complexity. To improve the training efficiency and the recognition accuracy, we theoretically analyze its structure. By modifying the network structure, we replace the convolutional and pooling layers in the residual network with the Inception structure to lower the number of training parameters and the time complexity of the model. On this basis, we propose an Inception Deep Residual Network (IDRN) based classification and recognition algorithm for skin melanoma, where the Inception structure and the SeLU activation function respectively replace the convolutional and pooling layers and the traditional ReLU function. Subsequently, experimental validation is carried out on the published ISIC2017 dataset of dermoscopic images of melanoma. The theoretical and experimental results show that compared with the traditional convolutional neural network ResNet50, the proposed algorithm reduces time complexity and improves recognition accuracy.