Fast Data Extraction for Numerical Weather Prediction Based on Decision Analysis
Author:
  • Article
  • | |
  • Metrics
  • |
  • Reference [12]
  • |
  • Related [20]
  • | | |
  • Comments
    Abstract:

    Traditional data extraction methods are usually inefficient. To address this problem, we first design an exact position addressing-based algorithm with multi-processing methods to achieve the accurate positioning of data blocks by taking the massive data generated from semi-structured numerical weather prediction (NWP) products as the research object. Then, an extraction algorithm is designed to extract data in the spatial range on demand, namely, to realize on-demand data extraction according to attribute dimensions as well as the latitude and longitude of data. As a result, the multi-process data reading under unified whole-process control is achieved on the basis of the above two algorithms. For testing, the time consumption of a single data plane is taken as the main assessment index, and the single-, quad-, octo-, and 16-core processes are employed for data processing. The test results reveal that the processing with 16-core processes is faster than that of a single-core process, and the time consumption is reduced from 257 ms to 37 ms. This method can effectively improve the efficiency of data extraction for non-structural NWP products and has been put into use in decision analysis for urban governance.

    Reference
    [1] 李永生, 曾沁, 杨玉红, 等. 基于大数据技术的气象算法并行化研究. 计算机技术与发展, 2016, 26(9): 47–49, 55
    [2] 李鸣野. 基于散列查找和多线程调度的快速提取GRIB数据方法. 山西师范大学学报(自然科学版), 2019, 33(2): 10–17
    [3] 曾沁, 李永生. 基于分布式计算框架的风暴三维追踪方法. 计算机应用, 2017, 37(4): 941–944. [doi: 10.11772/j.issn.1001-9081.2017.04.0941
    [4] 李永生, 曾沁, 徐美红, 等. 基于Hadoop的数值预报产品服务平台设计与实现. 应用气象学报, 2015, 26(1): 122–128. [doi: 10.11898/1001-7313.20150113
    [5] 但玻, 冯汉中, 罗可生. ECMWF0.25*0.25经纬网格模式资料处理及软件实现. 高原山地气象研究, 2013, 33(3): 92–96
    [6] 刘媛媛, 应显勋, 赵芳. GRIB2介绍及解码初探. 气象科技, 2006, 34(S1): 61–64
    [7] 赵芳, 薛蕾, 刘媛媛. 表格驱动码业务试验系统设计与实现. 气象科技, 2018, 46(4): 679–684
    [8] 肖华东, 孙婧, 孙朝阳, 等. 中国气象局S2S数据归档中心设计及关键技术. 应用气象学报, 2017, 28(5): 632–640. [doi: 10.11898/1001-7313.20170511
    [9] 孙周军, 乔文文, 侯灵, 等. 混合架构的可视化数据调度检索模型. 计算机系统应用, 2019, 28(12): 63–71. [doi: 10.15888/j.cnki.csa.007202
    [10] 王兵, 李杰. 基于通用模型的GRIB格式数据读取技术. 航空计算技术, 2018, 48(6): 96–101. [doi: 10.3969/j.issn.1671-654X.2018.06.023
    [11] 张瑞聪, 任鹏程, 房凯, 等. Hadoop环境下分布式物联网设备状态分析处理系统. 计算机系统应用, 2019, 28(12): 79–85. [doi: 10.15888/j.cnki.csa.007181
    [12] 胡洋. 基于深度学习的SDN虚拟蜜网路由优化. 计算机系统应用, 2020, 29(10): 274–279. [doi: 10.15888/j.cnki.csa.007626
    Cited by
    Comments
    Comments
    分享到微博
    Submit
Get Citation

李永生,张金标,张敏,陈冰怀.面向决策分析的海量气象数值预报数据快速提取.计算机系统应用,2022,31(9):319-323

Copy
Share
Article Metrics
  • Abstract:748
  • PDF: 1704
  • HTML: 1288
  • Cited by: 0
History
  • Received:December 14,2021
  • Revised:January 12,2022
  • Online: June 17,2022
Article QR Code
You are the first1015044Visitors
Copyright: Institute of Software, Chinese Academy of Sciences Beijing ICP No. 05046678-3
Address:4# South Fourth Street, Zhongguancun,Haidian, Beijing,Postal Code:100190
Phone:010-62661041 Fax: Email:csa (a) iscas.ac.cn
Technical Support:Beijing Qinyun Technology Development Co., Ltd.

Beijing Public Network Security No. 11040202500063