MapReduce optimization algorithm based on machine learning in heterogeneous cloud environment

• 作 者: LIN Wen-hui LEI Zhen-ming LIU Jun YANG Jie LIU Fang HE Gang WANG Qin
• 机 构: Beijing Key Laboratory of Network System Architecture and Convergence, Beijing University of Posts a
• 出 处: The Journal of China Universities of Posts and Telecommunica
• 发表日期: 2013年06期
• 关键词: cloud heterogeneous assignment execution dynamically running maximize optimize c

We present an approach to optimize the MapReduce architecture,which could make heterogeneous cloud environment more stable and efficient.Fundamentally different from previous methods,our approach introduces the machine learning technique into MapReduce framework,and dynamically improve MapReduce algorithm according to the statistics result of machine learning.There are three main aspects:learning machine performance,reduce task assignment algorithm based on learning result,and speculative execution optimization mechanism.Furthermore,there are two important features in our approach.First,the MapReduce framework can obtain nodes’performance values in the cluster through machine learning module.And machine learning module will daily calibrate nodes’performance values to make an accurate assessment of cluster performance.Second,with the optimization of tasks assignment algorithm,we can maximize the performance of heterogeneous clusters.According to our evaluation result,the cluster performance could have 19%improvement in current heterogeneous cloud environment,and the stability of cluster has greatly enhanced.

We present an approach to optimize the MapReduce architecture, which could make heterogeneous cloud environments more stable and efficient. Fundamentally different from previous methods, our approach introduces the machine learning technique into MapReduce framework, and dynamically improve MapReduce algorithm according to the statistics result of machine learning.There are three main aspects: learning machine performance, reduce task assignment algorithm based on learning result, and speculative execution optimization mechanism. Future, there are two important features in our approach. First, the MapReduce framework can obtain nodes’ performance values ​​in the cluster through machine learning module. And machine learning module will daily calibrate nodes’ performance values ​​to make an accurate assessment of cluster performance. Second, with the optimization of tasks assignment algorithm, we can maximize the performance of heterogeneous clusters. According to our evaluation result, the cluster per formance could have 19% improvement in current heterogeneous cloud environment, and the stability of cluster has greatly enhanced.