Considering the low detection rate of X-ray security inspection of contraband, an algorithm based on the improved Cascade RCNN is proposed. By this algorithm, a batch feature erasing (BFE) module is introduced into the network structure, which can enhance local feature learning by randomly erasing the same region and thus further enhance the learning expression of residual features. In addition, the weighted SD loss function is presented in this algorithm to solve the problem of low detection rates, which employs weight fusion to fuse Smooth L1 loss and DIoU loss, and by changing the proportion coefficient of weights, it can make the detection result more accurate. The experimental results show that the detection rate of the improved Cascade RCNN on an open contraband dataset for X-ray security inspection is increased by 3.11% compared with that of the original algorithm, and the accuracy of the improved algorithm is raised.