Given the limitations of the four-band images' color cast correction algorithm based on ternary linear polynomial regression, to better solve the RGBIR four-band image color cast problem of infrared crosstalk, three aspects, i.e., the sample, data type, and correction model of the polynomial regression algorithm, are worked on to improve the color cast correction effect of four-band image. The correction model was established by adding training samples of the algorithm and converting the data into signed floating-point pixel values to make the correction algorithm more robust. Due to the nonlinear gray-scale expression of RGB images, the ternary linear model was changed to the ternary quadratic one. Experiments show that the proposed optimization method in this study improves the color cast correction effect of the four-band images.