One of the major problems of the object tracking algorithm is the imbalance of positive and negative samples, and the positive samples are of high similarity. Aiming at the problem of insufficient positive samples in the tracking update process, this study proposes an improved MDNet-based video object tracking algorithm based on the multi-domain convolutional neural network (MDNet) algorithm. First, the strategy of candidate selection is improved in the original algorithm, and a model update method is presented on the basis of the combination of the candidate confidence and the threshold judgment of coordinate variance. Second, the cross-entropy loss function of the original algorithm is altered to a focal loss function with better performance. The experimental results show that the algorithm has a significant improvement in tracking precision and success rate compared with the MDNet algorithm under the same experimental environment.