A virtual sports interaction system based on real-time video perception is proposed in response to the problems that traditional sports are limited by venues and equipment in the context of ongoing COVID-19 response, and the related products in the market are expensive and not scalable. The system is designed with a video data acquisition module and a human joint point extraction module, which can acquire human joint point coordinates in combination with OpenPose and capture human gestures and body movements in real time. The action semantic understanding module includes motion action understanding and drawing action understanding. The former recognizes the motion action semantics depending on the relative position relationship of the limb joints in motion. The latter generates the drawing action trajectories of wrist joints as sketch images, uses AlexNet to recognize and classify them, and resolves them into the corresponding drawing action semantics. The classification accuracy of the model is 98.83% in edge-side devices. A Unity-based sketch game application is used as the visual interaction interface to realize motion interaction in a virtual scene. The system adopts the interaction mode of real-time video perception to achieve home exercise and fitness without other external devices, which is more participatory and interesting.