Binary networks have obvious advantages in terms of speed, energy consumption, and memory consumption, but they cause a great loss of accuracy for the deep network model. In order to solve the problems above, this study proposes a staged residual binarization optimization algorithm for binary networks to obtain a better binary neural network model. In this study, we combine the random quantification method with XNOR-net, and propose two improved algorithms, namely applying weights approximation factor and deterministic quantization networks, and a new staged residual binarization BNN training optimization algorithm, in order to obtain the recognition accuracy of the full-accuracy neural network. Experimental results show that staged residual binarization algorithm can effectively improve the training accuracy of binary model, and does not increase the computational complexity of the related network in the testing process, thus maintaining the advantages of high speed, low memory usage, and small energy consumption.