Unknown malicious network traffic detection is one of the core problems to be solved in anomaly detection as the traffic data obtained from high-speed network data flow are often unbalanced and changeable. Although there have been many studies on feature processing and detection methods of unknown malicious network traffic detection, these methods have shortcomings in simultaneously solving data imbalance and variability as well as detection performance. Considering the difficulty in unknown malicious network traffic detection, this study proposes an unknown malicious traffic detection model based on integrated SVM and bagging. Firstly, in view of the imbalance of network traffic data, a traffic processing method based on Multi-SMOTE oversampling is put forward to improve the feature quality upon traffic processing. Secondly, considering the distribution diversity of network traffic data, an unknown traffic screening method based on semi-supervised spectral clustering is presented to screen unknown traffic from mixed traffic with a diverse distribution. Finally, with the idea of Bagging, an unknown malicious traffic detector based on integrated SVM is trained. The experimental results reveal that the proposed detection model is superior to the current similar methods in comprehensive evaluation (F1 value), and it also has good generalization ability on different data sets.