With the growing population of elderly people, the safety of the elders living alone becomes a rising issue for the society. Falling down is one of the most common and greatest risks and injuries occurring to the elders living at home. There have been many algorithms on elderly falling detection. However, the vast majority of the existing methods, which use foreground extraction to get human body silhouette are implemented on static cameras. It means that we should implement cameras for every independent region in the house to make sure that the elders is visible in the frame, which is impractical. This paper proposes a novel approach for detecting human body falls based on image semantic segmentation and convolutional neural network model(CNN), which can be implemented on portable cameras. First, the fully convolutional network(FCN) is used to segment human pixels in the frame. If the body shape meets the conditions of area ratio, aspect ratio is used to estimate whether it is a falling body or not. Otherwise, a combined CNN classification model is used. Regions of human body are classified in three cases (fall, stand, half-lying) and the results are used to estimate whether there is a falling body in the frame. From the experimental results we achieved, it was concluded that our method has a high recognition rate (91.32%) and low false alarm rate(1.66%).