Parallel Cleaning Strategy of Checkpoint Cache Based on Spark Utility Aware
CSTR:
Author:
Affiliation:

Clc Number:

Fund Project:

  • Article
  • |
  • Figures
  • |
  • Metrics
  • |
  • Reference
  • |
  • Related
  • |
  • Cited by
  • |
  • Materials
  • |
  • Comments
    Abstract:

    In view of the fact that the cache data cleaning of spark checkpoint needs to be cleaned by the programmer after the job is completed, which may lead to memory accumulation of the failure data. This study analyzes the execution mechanism of checkpoint, deduces that the checkpoint cache cleaning method is not extensible with the increase of the number of check points. The matching degree between checkpoint cache and memory slot is measured by using the utility entropy model of checkpoint cache. The optimal checkpoint cache cleaning time is derived by using the principle of best utility matching. The PCC strategy based on utility entropy optimizes memory resources by making the checkpoint cache clean-up time approximately equal to the time when the checkpoint is written to HDFS. The experimental results show that in the multi-job execution environment based on fair scheduling, with the increase of the number of checkpoints, the execution efficiency of the unoptimized program becomes worse. After using PCC strategy, the program execution time, power consumption and GC time can be reduced by 10.1%, 9.5% and 19.5%, respectively. Effectively improve the efficiency of multi-checkpoint program execution.

    Reference
    Related
    Cited by
Get Citation

宋一鑫,于俊洋,何欣,王锦江. Spark效用感知的检查点缓存并行清理策略.计算机系统应用,2022,31(4):253-259

Copy
Share
Article Metrics
  • Abstract:
  • PDF:
  • HTML:
  • Cited by:
History
  • Received:June 23,2021
  • Revised:July 14,2021
  • Adopted:
  • Online: March 22,2022
  • Published:
Article QR Code
You are the firstVisitors
Copyright: Institute of Software, Chinese Academy of Sciences Beijing ICP No. 05046678-3
Address:4# South Fourth Street, Zhongguancun,Haidian, Beijing,Postal Code:100190
Phone:010-62661041 Fax: Email:csa (a) iscas.ac.cn
Technical Support:Beijing Qinyun Technology Development Co., Ltd.

Beijing Public Network Security No. 11040202500063