Given the common problems of crowd counting with a complex background, occlusion, and uneven crowd distribution, a joint loss-based space-channel dual attention network (JL-SCDANet) is proposed. The front end of the network extracts coarse-grained features of an image, and the spatial attention mechanism and channel attention mechanism are added in the middle to highlight the key areas of the image, while the back end uses dilated convolution that can increase the receptive field without losing the image resolution to extract deep two-dimensional features. In addition, the model is trained with the joint loss function to enhance its robustness. Comparative experiments are carried out on three public data sets (i.e., ShanghaiTech Part B, mall, and UCF_CC_50) to verify the improvement effect of the model. In terms of the mean absolute error (MAE) and mean square error (MSE), the results on ShanghaiTech Part B, mall, and UCF_CC_50 reach 8.13 and 13.13, 1.78 and 2.28, and 182.12 and 210.24, respectively. The experimental results prove the effectiveness of the network in improving the accuracy of population statistics.