Intelligent protection for rail vehicles involves the tasks of railway track intrusion detection and driving area segmentation. In the field of deep learning, there are algorithms for each task, but they cannot meet the needs of multi-task situations very well. This algorithm uses a lightweight convolution neural network (CNN) as an encoder to extract the feature map and then sends it to two decoders based on one-stage detection network to complete their respective tasks. Semantic features of different levels and scales are fused in the feature map output by the encoder, which performs pixel-level semantic prediction well and improves the detection and segmentation performance significantly. The equipment using this algorithm will master the recognition, detection, judgment, and tracking of new targets, ensuring the traveling safety of rail vehicles.