The abuse of Unmanned Aerial Vehicles (UAVs) brings great security risks to the low altitude area. Then the research on detection of UAVs’ illegal intrusion has become important for a low-altitude defense system. In this study, a multi-sensor information fusion technique based on radar and a RGB camera is designed to detect small objects in the low altitude range. After that, the Single Shot multibox Detector (SSD) for deep learning is introduced to train the UAV detection model and predict the category and location of objects captured by the RGB camera. An experimental platform is built to verify that the information fusion method can collect the location, speed, appearance of targets, and the deep learning model can determine the categories of suspicious targets.