In view of the difficulty of obtaining the ideal effect by the single feature in image matching, an improved image matching algorithm based on deep convolution network is proposed.First of all, the algorithm expands the convolution layers, and efficiently computes the matrix multiplication by using the BLAS (Basic Linear Algebra Subprograms) libraries.The algorithm can accelerate the running speed.Then, a screening method of matching points based on the POEM (Pattern of Oriented Edge Magnitudes) feature similarity of feature points is used as well.The method can remove some wrong matching points, make the estimated fundamental matrix more robust and improve the repeating texture and rotational image.The accuracy and instantaneity of the algorithm are proved by the experimental results.