In case that public safety has already caused extensive social concern in recent years, how to use surveillance video to detect abnormal pedestrians and prevent dangerous events becomes a hot topic. Abnormal pedestrians are those who are distinctly different from ordinary pedestrians in appearance, for example, using helmet to cover the face or ducking from the camera. Considering that the characteristics of abnormal pedestrians are mainly concentrated in head and face, this study proposes a fast detection method for abnormal pedestrians based on multi-task Convolutional Neural Network (CNN) and one-class Support Vector Machine (SVM) for head-facial features. First, we detect head-facial regions in surveillance video, then we use the multi-task CNN to extract features of these regions, and then we use one-class SVM to judge whether it is a normal pedestrian or not. In addition, this study designs a convolution kernel splitting method for CNN to accelerate the feature extraction speed. Finally, the experiment shows that the algorithm proposed in this study can effectively and quickly detect abnormal pedestrians in surveillance video.