After a detailed analysis of the existing anti-intrusion system of gantry cranes, an anti-intrusion monitoring model based on machine vision and deep learning is built in view of the complex working environments and low intelligence levels of the system. In consideration of the advantages and disadvantages of each target detection algorithm and semantic segmentation algorithm, the semantic segmentation algorithm is adopted as the anti-intrusion model, and the ICNet is used as the main semantic segmentation network. Compared with other networks, ICNet displays the best accuracy, with a training accuracy of 99.37% and a training loss of 1.81%. The results prove the intelligence and feasibility of the anti-intrusion system based on semantic segmentation.