Many skin cancer diseases have obvious early symptoms. Currently, the diagnosis of skin cancer mainly relies on medical workers with professional knowledge, bringing the problems such as long time consumption and low reusability. In response to these problems, a lightweight skin disease recognition model based on improved MobileNetV3-Small is proposed in this study. Firstly, a CaCo attention module based on coordinate attention (CA) mechanism is proposed, Secondly, for the uneven distribution of the samples of skin-cancer datasets, a combination of multiple loss functions is proposed to enhance the learning ability of the model for cases with few samples. The improved MobileNetV3-CaCo model has an accuracy, balance accuracy, and model parameter quantity of 93.39%, 86.35%, and 2.29M, respectively, thus ideal recognition results are achieved.