Efficient Auto Scaling and Cost-Effective Architecture in Apache Hadoop

Abstract

In the age of Big Data Analytics, Cloud Computing has been regarded as a feasible and applicable technology to address Big Data Challenges, from storage capacities to distributed processing computations. One of the keys of its success is its high scalability which refers to the ability of the system to increase its performance, resources and functionalities according to the workload. This flexibility has been seen as an appropriate way to decrease datacenters' energy consumption and thus assures cost-saving and efficiency without effecting performance of the system. In order to handle Big Data operations, Cloud Computing has implemented various platforms and tools such as Apache Hadoop and provides distributed processing of very large data sets across multiple clusters. This paper proposes an auto scaling architecture based on the framework of Hadoop; it adjusts automatically the computation resources depending on the workload. In order to validate the effectiveness of the proposed architecture, a case study about Twitter data analysis in a cloud simulated environment has been implemented to improve the cost-effectiveness and the efficiency of the system.