Dynamic Data Partition Based on Node Load
CSTR:
Author:
Affiliation:

Clc Number:

Fund Project:

  • Article
  • |
  • Figures
  • |
  • Metrics
  • |
  • Reference
  • |
  • Related
  • |
  • Cited by
  • |
  • Materials
  • |
  • Comments
    Abstract:

    A dynamic data partition system based on node load mainly considers the load of CPU, memory, and bandwidth of nodes. It first uses the quadratic smoothing method to predict the load of nodes, then combines AHP and entropy index weight to get the processing capacity of each node, and finally dynamically adjusts the load balance of the system for different application scenarios to improve the response speed of applications. It includes the modules of load monitoring and collection, prediction, data pre-partition, and data migration. Given the heterogeneity of node resources in a distributed environment, it aims to reduce the data transmission between nodes in the process of data analysis and calculation, make full use of node computing resources, and improve the parallel computing speed of application analysis through load balancing. Therefore, this study proposes a dynamic partition data mechanism based on node load to improve the system load balance and application response speed and assist the relevant staff in making the decision. This study combines data analysis application scenarios integrating Spark and Elasticsearch for testing.

    Reference
    Related
    Cited by
Get Citation

孟令伍,杨阳朝,黄晓明,练丽萍.基于节点负载的数据动态分区.计算机系统应用,2021,30(12):299-307

Copy
Share
Article Metrics
  • Abstract:
  • PDF:
  • HTML:
  • Cited by:
History
  • Received:March 02,2021
  • Revised:March 29,2021
  • Adopted:
  • Online: December 10,2021
  • Published:
Article QR Code
You are the firstVisitors
Copyright: Institute of Software, Chinese Academy of Sciences Beijing ICP No. 05046678-3
Address:4# South Fourth Street, Zhongguancun,Haidian, Beijing,Postal Code:100190
Phone:010-62661041 Fax: Email:csa (a) iscas.ac.cn
Technical Support:Beijing Qinyun Technology Development Co., Ltd.

Beijing Public Network Security No. 11040202500063