论文部分内容阅读
针对风云气象卫星地面应用系统中如何缩短大数据集产品的处理时延,提高系统资源利用率的问题,提出了一种基于性能预测的、计算节点作业执行概率的信息素动态加权并行算法(pdw),并构建了新的并行处理模式。该模式一方面能够根据轨道级遥感数据与目标产品之间的空间位置匹配关系进行网格处理粒度的合理划分;另一方面,采用pdw算法综合考虑节点本身的处理能力和实时负载信息进行分粒度作业的资源匹配调度。业务实测结果表明,较传统的处理模式,采用新的处理模式后,具有典型大数据集特征的风云三号(FY-3)中分辨率光谱成像仪(MERSI)250 m分辨率全球植被指数分层数据格式(HDF)产品的处理时效提高了10倍左右,所提并行处理模式可以有效缓解I/O资源占用瓶颈、均衡系统负载,提升大数据集产品的处理效率。
Aiming at the problem of how to shorten the processing delay of big data set products and improve the utilization of system resources in the meteorological satellite ground application system, a pheromone dynamic weighted parallel algorithm (pdw ), And built a new parallel processing model. On the one hand, this model can reasonably divide the granularity of the grid according to the spatial position matching between the orbital level remote sensing data and the target product. On the other hand, using the pdw algorithm to consider the processing capacity of the node itself and the real-time load information The job’s resource matching schedule. The results of business measurements show that the FY-3 Mid-Resolution Spectral Imager (MERSI) 250 m resolution Global Vegetation Index The processing time of HDF products is increased by about 10 times. The proposed parallel processing mode can effectively alleviate the bottleneck of I / O resources consumption, balance the system load and improve the processing efficiency of big data set products.