To rapidly and accurately detect and track multiple objects, this paper presents a method of multiple objects detection and tracking based on Omni-directional images. Firstly, getting 360-degree Omni-directional images by Omni-directional Vision Sensor in real time, which is an effective way to solve the problem of the multiple video source correlation. Secondly, the MHoEI algorithm that is created by combining Motion History Images algorithm with Motion Energy Images algorithm can efficiently detect and track multiple objects in this paper. Finally,the matching algorithm based on object oriented technology is proposed through fusing many properties of objects,which was used to identify different objects. The algorithm better solves the problem of data association of different properties. Experiment results show that the method in this paper can better track objects in complex background. The system has robustness, lightly computational load, high efficiency features.