The deep neural network can better express features but results in difficult optimization, high training cost, and vanishing gradient. The surge in quantity of parameters leads to a too bloated model to be deployed on the platform with weak computing power and small storage, such as mobile terminal and industrial control equipment. Aiming at these problems, we construct a lightweight neural network combining atrous convolutions and multi-scale sparse structures to extract the features of images, and realize the end-to-end recognition for the captcha images with color pattern noise and seriously touched and distorted characters. The dataset containing one million images was divided into training sets, validation sets, and test sets in the ratio of 98:1:1 and trained in batches. Consequently, the lightweight neural network has a recognition rate of 98.9% on test sets with much fewer parameters.