###
计算机系统应用:2018,27(8):232-236
本文二维码信息
码上扫一扫!
基于关键阶段分析的Spark性能预测模型
葛庆宝1,2, 陶耀东2, 高岑2, 田月2, 孟祥茹2
(1.中国科学院大学, 北京 100049;2.中国科学院 沈阳计算技术研究所, 沈阳 110168)
Performance Prediction Model for Spark Based on Key Stages Analysis
GE Qing-Bao1,2, TAO Yao-Dong2, GAO Cen2, TIAN Yue2, MENG Xiang-Ru2
(1.University of Chinese Academy of Sciences, Beijing 100049, China;2.Shenyang Institute of Computing Technology, Chinese Academy of Sciences, Shenyang 110168, China)
摘要
图/表
参考文献
相似文献
本文已被:浏览 133次   下载 110
投稿时间:2017-12-08    修订日期:2018-01-04
中文摘要: Spark作为目前大数据处理领域广泛使用的计算平台,合理分配集群资源对Spark作业性能优化有着重要的作用.性能预测是集群资源分配优化的基础和关键,本文正是基于此提出了一种Spark性能预测模型.文中选取作业执行时间作为Spark性能衡量指标,提出了Spark作业关键阶段的概念,通过运行小批量数据集来获取关键阶段的运行时间和作业输入数据量之间关系,从而构建了Spark性能预测模型.实验结果表明该模型较为有效.
中文关键词: Spark  资源分配  性能预测  关键阶段
Abstract:Spark is widely used as a computing platform for large data processing, reasonable allocation of cluster resources plays an important role in the operation of Spark performance optimization. The performance prediction is the basis and key of cluster resource allocation optimization, thus we put forward a Spark performance prediction model in this paper. This paper selects the job execution time as a measure indicator of Spark performance, and put forward the concept of key Stage of Spark job. Finally, we built the model by analyzing relationships between the key Stages and the amount of input data through running a small quantity of data. The experimental results show that the model is effective
文章编号:     中图分类号:    文献标志码:
基金项目:
引用文本:
葛庆宝,陶耀东,高岑,田月,孟祥茹.基于关键阶段分析的Spark性能预测模型.计算机系统应用,2018,27(8):232-236
GE Qing-Bao,TAO Yao-Dong,GAO Cen,TIAN Yue,MENG Xiang-Ru.Performance Prediction Model for Spark Based on Key Stages Analysis.COMPUTER SYSTEMS APPLICATIONS,2018,27(8):232-236

用微信扫一扫

用微信扫一扫