The diversity of crowd scale in reality is a great challenge to crowd counting algorithms. Therefore, a novel crowd counting algorithm based on scale fusion is proposed in this study. Firstly, the algorithm for density map generation is optimized. Multiple head detectors are used to obtain part of the head scales of the sparse crowd, and RBF interpolation is employed to complete this part of the density map. As to the dense part of crowd, the traditional distance self-adaptive algorithm is adopted to generate a more accurate density map. Secondly, the regression neural network of the density map is designed with a mobile inverted bottleneck convolution module, and a dilated convolution module is added to facilitate the extraction of head edge features. Finally, the loss function of the regression neural network is optimized by distinguishing the crowd area from the non-crowd area. In the experiment part, the algorithm is compared with other similar algorithms on multiple datasets, and the results show that the proposed method can significantly improve the accuracy of crowd counting.