To improve the efficiency and accuracy of rail surface defect detection, a rail surface defect detection algorithm based on background difference and maximum entropy is proposed. Firstly, the background model of the rail images is built, and the original images are differentiated from the background images to avoid the influence of illumination change and uneven reflection and accurately highlight the defect area. Then, the improved genetic algorithm is combined with the maximum entropy method to seek the best segmentation threshold and binarize the difference graph. The operational speed of the maximum entropy method is accelerated by the improved genetic algorithm. Finally, the binary images are filtered to complete the segmentation of rail surface defects. The simulations indicate that this method can segment defects quickly and accurately, and the precision, recall, and accuracy are 88.6%, 93.4%, and 90.6%, respectively.