【摘 要】
：
电力用采大数据上的OLAP查询具有涉及数据量大、多表连接操作频繁、SQL结构复杂等特点,传统关系型数据库面对该类应用,暴露出可扩展性弱、数据写入吞吐量低与查询效率低等问题。针对上述问题,本文设计并实现了一套基于Spark/Shark的电力用采大数据OLAP分析系统,采用分布式文件系统HDFS保存电力用采大数据,通过Shark进行前端SQL解析, Spark进行查询计算。然而,原生Shark只支持粗
【机 构】
：
国网信息通信产业有限公司/中国科学院计算技术研究所/网家电网浙江省电力公司
论文部分内容阅读
电力用采大数据上的OLAP查询具有涉及数据量大、多表连接操作频繁、SQL结构复杂等特点,传统关系型数据库面对该类应用,暴露出可扩展性弱、数据写入吞吐量低与查询效率低等问题。针对上述问题,本文设计并实现了一套基于Spark/Shark的电力用采大数据OLAP分析系统,采用分布式文件系统HDFS保存电力用采大数据,通过Shark进行前端SQL解析, Spark进行查询计算。然而,原生Shark只支持粗粒度分区,不支持细粒度的索引技术,难以高效过滤无关数据,影响了查询性能。为克服这一不足,该系统实现了一种基于前缀树的细粒度索引结构TrieIndex,并通过数据重组织技术优化了数据在HDFS的分布,提升了Shark的数据过滤能力以及电力用采大数据OLAP分析的性能。采用真实用采数据与查询的实验结果表明,该系统比关系型数据库的写入速度提升了12倍,比原生Shark的查询效率提升了10倍以上。
其他文献
Amongst the available amorphous semiconductors that are suitable for large-area and flexible electronics,IGZO has shown the most potential due to its high electron mobility and low-temperature film de
Realization of magnetization reversal at a nanoscale by employing an electrical field is extremely important for the development of various spintronic devices.Several ways to control the magnetization
Low-dimensional oxide structures have remarkable properties that can be tuned by external fields.Particularly,based on the perovskite ABO3(111)surfaces,the so-called double perovskite A2BBO6 structure
In recent years,resistive switching(RS)effect has attracted considerable attention due to its scientific interest and important applications [1-3].It has been emphasized that a robust and predictive u
Electrolyte gating with ionic liquid has attracted significant attention as a powerful tool for greatly modulating the physical properties in a variety of materials.However,for electrolyte gating in t
Recently,atomic layer deposition(ALD)has been adopted in the surface modifications of nanoparticles with controllable dense and uniform nanoscale coating.Up to now,several kinds of equipment designs,i
Pulsed ultraviolet light from a XeF excimer laser was used to grow thin films of zinc oxide on(111)p-type silicon wafers within a versatile high vacuum laser deposition system.Pressure,target temperat
Resistive switching behaviour has been intensively studied in oxides[1,2].In oxides,it is well known that oxygen vacancies play an important role in the resistive switching behaviour[3].The oxygen vac
Surface plasmon polarization(SPP)waveguides consisting of dielectric claddings and metal thin films have been shown to transmit optical signals beyond the diffraction limit.It is appealing for nanosca
The sandwich films composed aluminum ions doped zinc oxide(AZO)/silver(Ag)/aluminum doped zinc oxide(AZO)films were prepared on polyimide(PI)and glass substrates by RF magnetron sputtering using the p