The access control management methods for important places such as military weapons warehouses are insufficient in security. In order to solve the defects, we design an access control system based on improved convolutional neural network. This paper first introduces the basic knowledge of convolutional neural networks, then introduces Particle Swarm Optimization (PSO) algorithm to design and optimize initial weights and thresholds of convolutional neural networks. After designing, experiment with the MNIST handwritten digital dataset is carried out. The results demonstrate that the modified convolutional neural network can make the convergence speed faster, and the loss is smaller, so the outcome is obviously better than the traditional convolutional neural network. On this basis, according to the actual working conditions of the troops, PSO is applied in the MTCNN and SIAM-ResNet face detection algorithm, the access control system based on improved convolutional neural network is designed, which makes the access control of important places in the army have higher security and reliability.