Moving object detection algorithms are widely used in video surveillance and other fields. But due to noise, illumination changes and other interference, traditional algorithms are often ineffective. To get a better performance, we transform the problem into a pixel-wise segmentation problem, and propose a novel algorithm based on deep encoder-decoder neural networks. We train an encoder-decoder network offline to learn the differences between the background and the video frame. We firstly use the Gaussian Mixture Model (GMM) to generate a background, and then feed video frames and the background into the encoder-decoder network to get detection results. This method utilizes the advantages of deep convolution network in anti-noise and feature learning, and performs well without complicated parameter tuning. We experiment on the CDnet2014 dataset, and results show that the algorithm we propose performs much better than the original GMM algorithm, and even outperforms some top algorithms in some scenes. Due to the simple network architecture, our algorithm is capable of almost real-time processing using a GPU, which shows its great practicality.