本文已被:浏览 1708次 下载 2257次
Received:January 29, 2018 Revised:February 27, 2018
Received:January 29, 2018 Revised:February 27, 2018
中文摘要: 为应对海量遥感影像快速计算的需求,通过对影像获取、算法和计算过程优化和改进,提出了一种基于Apache Spark并行计算框架的MODIS海表温度反演方法,实现了海量MODIS遥感影像的海表温度快速反演.应用四轮网络查询请求获取特定的时空范围影像数据,提高影像获取阶段的效率;应用简化算法参数、拟合过程变量改进海表温度劈窗算法,使之适合快速并行计算;应用弹性分布式数据集(RDD)窄依赖关系的优点,避免并行计算中的数据交换延迟.通过单机模式与集群模式对比实验,发现集成了并行计算框架的集群模式影像处理效率约为单机模式的10倍.研究结果表明了融合集群计算技术的海表温度反演过程有效提高了传统单机应用程序的处理效率.
中文关键词: Apache Spark MODIS 海表温度
Abstract:In response to computing problems of massive remote sensing images, a method based on Apache Spark is proposed and implemented in retrieving MODIS Sea Surface Temperature (SST) by optimizing and improving the image acquisition, algorithm, and computing process. It applied four bouts of network requests to acquire user-defined data of specific time and zones to improve the efficiency of image acquisition. For a parallelizable algorithm, improvements that reduce parameters and simplify intermediate models are added to the split window algorithm, thus to adapt to fast parallelized computing. Taking advantage of narrow dependence between Resilient Distributed Datasets (RDD), delays for partitions' interactions are evaded. With comparison between single mode and cluster mode, the latter incorporated with Apache Spark has an efficiency of ten times to the former. This study proves that, comparing with a single machine's, programs that retrieving MODIS SST with cluster computing techniques has a higher efficiency.
keywords: Apache Spark MODIS Sea Surface Temperature (SST)
文章编号: 中图分类号: 文献标志码:
基金项目:国家自然科学基金(41771422);湖北省自然科学基金(2017CFB616)
引用文本:
刘欢,陈能成,陈泽强.基于Apache Spark的MODIS海表温度反演方法.计算机系统应用,2018,27(9):112-117
LIU Huan,CHEN Neng-Cheng,CHEN Ze-Qiang.Retrieving Method for MODIS Sea Surface Temperature with Apache Spark.COMPUTER SYSTEMS APPLICATIONS,2018,27(9):112-117
刘欢,陈能成,陈泽强.基于Apache Spark的MODIS海表温度反演方法.计算机系统应用,2018,27(9):112-117
LIU Huan,CHEN Neng-Cheng,CHEN Ze-Qiang.Retrieving Method for MODIS Sea Surface Temperature with Apache Spark.COMPUTER SYSTEMS APPLICATIONS,2018,27(9):112-117