###
:2019,28(6):189-197
←前一篇   |   后一篇→
本文二维码信息
码上扫一扫!
广播机制解决Shuffle过程数据倾斜的方法
(上海淇毓信息科技有限公司, 上海 200120)
Method Research to Solve Shuffle Data Skew Based on Broadcast
(Shanghai Qiyu Information Technology Co. Ltd., Shanghai 200120, China)
摘要
图/表
参考文献
相似文献
本文已被:浏览 68次   下载 102
投稿时间:2018-12-19    修订日期:2019-01-15
中文摘要: 在Spark计算平台中,数据倾斜往往导致某些节点承受更大的网络流量和计算压力,给集群的CPU、内存、磁盘和流量带来了巨大的负担,影响整个集群的计算性能.本文通过对Spark Shuffle设计和算法实现的研究,深入分析在大规模分布式环境下发生数据倾斜的本质原因.提出了广播机制避免Shuffle过程数据倾斜的方法,分析了广播变量分发逻辑过程,给出广播变量性能优势分析和该方法的算法实现.通过Broadcast Join实验验证了该方法在性能上有稳定的提升.
Abstract:In the Spark computing platform, data skew often causes some nodes to withstand greater network traffic and computing pressure, which imposes a huge burden on the cluster's CPU, memory, disk, and traffic, affecting the computing performance of the entire cluster. Through the research on Spark Shuffle design and algorithm implementation, and deep analyses on the essential reasons of data skew in large-scale distributed environment, this study proposes a method to avoid data skew in shuffle process through the broadcast mechanism, analyzes the process of broadcast variable distribution logic, and gives the algorithm implementation and performance advantage analysis of the method. The performance of the method is improved by the Broadcast Join experiment.
keywords: data skew  partition  shuffle  broadcast
文章编号:     中图分类号:    文献标志码:
基金项目:
引用文本:
吴恩慈.广播机制解决Shuffle过程数据倾斜的方法.计算机系统应用,2019,28(6):189-197
WU En-Ci.Method Research to Solve Shuffle Data Skew Based on Broadcast.COMPUTER SYSTEMS APPLICATIONS,2019,28(6):189-197

用微信扫一扫

用微信扫一扫