In the future, the main development mode of pig breeding industry is information and intelligence. In order to monitor the behavior of pigs intelligently, so as to monitor the health and growth of pigs, this paper presents a system technology for contactless identification and monitoring pig behavior based on machine vision. The system collects pig behavior sequence images by CCD camera, then extracts the depth features of those images using convolution neural network. After that, the feature fusion method is used to fuse the depth features of the behavior sequence images. Finally, the pig’s behavior activities are identified according to the fusion depth feature. The system realizes the high-precision identification of pig’s motion behavior, claudication behavior, volt behavior, breathing behavior, eating behavior, and excretion behavior under natural scenes. The accuracy rates of recognizing all kinds of behavior are more than 94%, which are higher than the state-of-the-art methods.