Motor imagery is a promising brain-computer interface paradigm. In the motor imagery classification tasks based on EEG, the equipment and the subjects will lead to the phenomenon of data distribution drift related to the subjects and time. This data distribution drift will reduce the classification accuracy of the classifier. Transfer learning can solve this distribution drift phenomenon very well. In this study, a new single source domain selection algorithm, multi-subdomain transferability estimation (MSTE) and a new transfer method, task-oriented subdomain adversarial transfer network (ToSAN), for the classification tasks of EEG signals are proposed. MSTE can evaluate the similarity in time and category between the source domain and the target domain. ToSAN can decompose features for classification tasks and perform multiple subdomain alignments on task-related features to overcome distribution differences. The experimental results on BCI Competition IV 2a and BCI Competition IV 2b show that compared with other methods, ToSAN improves the classification accuracy by at least 2.67% and 8.6%, respectively. The combination of MSTE and ToSAN achieve a classification accuracy of 81.73% and 88.73% on the BCI Competition IV 2a and BCI Competition IV 2b datasets, which is significantly better than all comparison methods.