The increasing popularity of smartphones brings not only convenience to people but also a lot of hidden dangers. Thus, it is necessary to monitor and restrict the use of phones in some specific situations. In this paper, we design a system to monitor phone usage. First, YOLOv3 is used to detect human bodies in an image. Then, the joints for each person are obtained by the OpenPose pose estimation algorithm. Furthermore, YOLOv3 is employed to judge whether there is a mobile phone in the hands. Finally, the current phone usage status is recognized by a neural network classifier. The experimental results show that the proposed scheme has good detection and recognition performance and can meet the application requirements in relevant scenarios.