Number Adjustment Method of MapReduce Jobs Based on YARN Resource Scheduler
CSTR:
Author:
Affiliation:

Clc Number:

Fund Project:

  • Article
  • |
  • Figures
  • |
  • Metrics
  • |
  • Reference
  • |
  • Related
  • |
  • Cited by
  • |
  • Materials
  • |
  • Comments
    Abstract:

    YARN is a distributed resource management system of Hadoop. It can be used to improve the utilization of memory, I/O, network, disk and other resources of distributed cluster. However, there are many configuration parameters in YARN. Due to this reason, manual tuning of Hadoop performance to get the best performance is difficult and time-consuming. Based on the existing YARN resource scheduler, a successive approximation closed-loop feedback control method is proposed. This method can dynamically tune the parallel number of MapReduce (MR) jobs in the running state of the cluster, and eliminating the process of manual adjustment of parameters. Experiments show that the proposed approach reduces the MR operation time for 53% and 14% based on capacity scheduler and fair scheduler, respectively, compared with the default configuration.

    Reference
    Related
    Cited by
Get Citation

廉华,刘瑜.基于YARN资源调度器的MapReduce作业数调节方法.计算机系统应用,2020,29(3):218-222

Copy
Share
Article Metrics
  • Abstract:
  • PDF:
  • HTML:
  • Cited by:
History
  • Received:August 07,2019
  • Revised:September 05,2019
  • Adopted:
  • Online: March 02,2020
  • Published: March 15,2020
Article QR Code
You are the firstVisitors
Copyright: Institute of Software, Chinese Academy of Sciences Beijing ICP No. 05046678-3
Address:4# South Fourth Street, Zhongguancun,Haidian, Beijing,Postal Code:100190
Phone:010-62661041 Fax: Email:csa (a) iscas.ac.cn
Technical Support:Beijing Qinyun Technology Development Co., Ltd.

Beijing Public Network Security No. 11040202500063