The existing human body detecting algorithms often have the shortages, such as less precision and less adaptability of environments. In order to improve the detecting results, an optimized algorithm is presented based on the combination of the weight of RGB channel and self-adaptive threshold in sub-area. Firstly, a simply background subtraction is applied to detect the moving human body area cursorily; secondly, the human area is calculated and estimated by its characters, then the two areas are combined to get the final target area. The target area will be divided into several sub-areas with same size, and then the weight of RGB channel and the threshold of each sub-area can be calculated and normalized to the range of [0,255], respectively. By using the calculated threshold and the weight value of target area’s RGB channel, we can do a fine background subtraction to detect the final result. Experiment results show that the precision of our algorithm has improved by 10% as compared with the algorithm based on HIS, and has improved by 40% as compared with the traditional algorithms. It can wholly achieve the requirements of precision and adaptability by actually testing.