The traditional CNN models have a poor weather classification effect for aerial video images and cannot satisfy the applications to mobile devices, and the existing weather image datasets are lacking, with single scenes. To address these problems, this study constructs four types of UAV aerial weather image datasets of sunny days, rainy days, snowy days, and foggy days for multiple scenes and proposes a weather scene classification model for UAV aerial video images based on lightweight transfer learning. The model uses a transfer learning approach to train two lightweight CNNs on the ImageNet dataset and designs three lightweight CNN branches for feature extraction. In feature extraction, EfficientNet-b0, a modification of the ECANet attention mechanism, is first used as the main branch to extract whole-image features, and two MobileNetv2 branches are employed to extract deep features unique to the sky and non-sky localities separately. Next, feature fusion is carried out for the three regions by Concatenate. Finally, a Softmax layer is used to classify the four classes of weather scenes. The experimental results indicate that the method achieves the accuracy of 97.3% in classifying weather scenes when applied to mobile and other computationally constrained devices, with good classification results.