Abstract:In a heterogeneous Hadoop cluster scenario, the hybrid use of erasure codes and replica storage modes, as well as the real-time computing capability difference of server nodes lead to the low efficiency of MapReduce job processing. To deal with this problem, this study implements a scheduling strategy that dynamically adjusts MapReduce job assignment in multi-concurrent scenarios according to data storage situations and the real-time load of nodes. This strategy dynamically controls the concurrent amount of tasks of each node by modifying data storage location strategies in the current Hadoop framework, so as to achieve more balanced resource allocation among jobs when multiple jobs are concurrent. The experimental results show that the scheduling mode proposed in this study can shorten the job completion time by about 17% and effectively avoid the starvation phenomenon faced by some jobs compared with the two default job scheduling strategies of Hadoop.