Considering the outdoor monitoring system using the image to identify weather conditions, a novel classification method with bag of words model is proposed which combines SVM with random forest to identify the sunny or cloudy days. The bag of words model uses the SIFT feature to construct the dictionary by clustering, and uses the least squares method to solve the dictionary structure parameters of the optimal image. Finally, the multi-scale image bag of words model feature is obtained according to the pyramid matching. The construction of the classifier uses the support vector machine (SVM) as the first level classifier to classify the small confidence samples, and then uses the random forest as the second level classifier to judge. Through the test of the 10 000 images of two categories of weather images, the recognition accuracy verifies the effectiveness of the method.