###
:2019,28(9):65-71
本文二维码信息
码上扫一扫!
基于Hive的高可用双引擎数据仓库
(1.中国科学院 计算机网络信息中心, 北京 100190;2.中国科学院大学, 北京 100190)
High Availability Dual Engine Data Warehouse Based on Hive
(1.Computer Network Information Center, Chinese Academy of Sciences, Beijing 100190, China;2.University of Chinese Academy of Sciences, Beijing 100190, China)
摘要
图/表
参考文献
相似文献
本文已被:浏览 94次   下载 63
投稿时间:2019-02-28    修订日期:2019-03-14
中文摘要: 打破信息孤岛,整合异构数据,汇聚共享交换,深度分析挖掘,提供行业领域辅助决策和态势分析具有深远的理论和应用价值.本文以中国科学院教育科研态势感知服务的实际需求为牵引,设计并实现了一套基于Hive的Hadoop/Spark双计算引擎大数据仓库,支持多种方式OLAP分析,进行了可用性、负载均衡、资源管理的优化设计,为后续进行全院数据汇聚挖掘、知识图谱构建、学科态势分析提供了平台支撑.实验表明,系统灵活高效,高可用可扩展,资源调度科学,负载均衡效果明显.
中文关键词: 数据仓库  Hive  高可用  OLAP  Hadoop
Abstract:Breaking isolated information island, integrating heterogeneous data, gathering and sharing exchanges, conducting in-depth analysis and mining, and providing industry-side decision-making and situation analysis have far-reaching theoretical and applied value. Based on the actual demand of the situational awareness service of the Chinese Academy of Sciences, this study designs and implements a Hive-based Hadoop/Spark dual computing engine big data warehouse supporting OLAP analysis in multiple ways, and carries out an optimization design of usability, load balancing, and resource management, which provides platform support for the subsequent data aggregation and mining, knowledge map construction and discipline situation analysis. Experimental results show that the system is flexible, efficient, available, and scalable, the resource scheduling is scientific, and the load balancing effect is obvious.
文章编号:     中图分类号:    文献标志码:
基金项目:中国科学院"十三五"信息化专项(XXH13504-03)
引用文本:
李翀,张彤彤,杜伟静,刘学敏.基于Hive的高可用双引擎数据仓库.计算机系统应用,2019,28(9):65-71
LI Chong,ZHANG Tong-Tong,DU Wei-Jing,LIU Xue-Min.High Availability Dual Engine Data Warehouse Based on Hive.COMPUTER SYSTEMS APPLICATIONS,2019,28(9):65-71

用微信扫一扫

用微信扫一扫