A Hadoop based Framework to Process Geo-distributed Big Data

Marco Cavallo, Lorenzo Cusma', Giuseppe Di Modica, Carmelo Polito, Orazio Tomarchio

2016

Abstract

In many application fields such as social networks, e-commerce and content delivery networks there is a constant production of big amounts of data in geographically distributed sites that need to be timely elaborated. Distributed computing frameworks such as Hadoop (based on the MapReduce paradigm) have been used to process big data by exploiting the computing power of many cluster nodes interconnected through high speed links. Unfortunately, Hadoop was proved to perform very poorly in the just mentioned scenario. We designed and developed a Hadoop framework that is capable of scheduling and distributing hadoop tasks among geographically distant sites in a way that optimizes the overall job performance. We propose a hierarchical approach where a top-level entity, by exploiting the information concerning the data location, is capable of producing a smart schedule of low-level, independent MapReduce sub-jobs. A software prototype of the framework was developed. Tests run on the prototype showed that the job scheduler makes good forecasts of the expected job’s execution time.

References

  1. Andrews, G. E. (1976). The Theory of Partitions, volume 2 of Encyclopedia of Mathematics and its Applications.
  2. Cavallo, M., Cusmá, L., Di Modica, G., Polito, C., and Tomarchio, O. (2015). A scheduling strategy to run Hadoop jobs on geodistributed data. In CLIOT 2015 - 3rd International Workshop on CLoud for IoT, in conjunction with the Fourth European Conference on Service-Oriented and Cloud Computing (ESOCC), Taormina (Italy).
  3. Dean, J. and Ghemawat, S. (2004). MapReduce: simplified data processing on large clusters. In OSDI04: Proceeding of the 6th Conference on Symposium on operating systems design and implementation. USENIX Association.
  4. Heintz, B., Chandra, A., Sitaraman, R., and Weissman, J. (2014). End-to-end Optimization for Geo-Distributed MapReduce. IEEE Transactions on Cloud Computing, PP(99):1-1.
  5. Jayalath, C., Stephen, J., and Eugster, P. (2014). From the Cloud to the Atmosphere: Running MapReduce across Data Centers. IEEE Transactions on Computers, 63(1):74-87.
  6. Kim, S., Won, J., Han, H., Eom, H., and Yeom, H. Y. (2011). Improving Hadoop Performance in Intercloud Environments. SIGMETRICS Perform. Eval. Rev., 39(3):107-109.
  7. Kreutz, D., Ramos, F., Esteves Verissimo, P., Esteve Rothenberg, C., Azodolmolky, S., and Uhlig, S. (2015). Software-Defined Networking: A Comprehensive Survey. Proceedings of the IEEE, 103(1):14- 76.
  8. Luo, Y., Guo, Z., Sun, Y., Plale, B., Qiu, J., and Li, W. W. (2011). A Hierarchical Framework for Cross-domain MapReduce Execution. In Proceedings of the Second International Workshop on Emerging Computational Methods for the Life Sciences, ECMLS 7811, pages 15- 22.
  9. Mattess, M., Calheiros, R. N., and Buyya, R. (2013). Scaling MapReduce Applications Across Hybrid Clouds to Meet Soft Deadlines. In Proceedings of the 2013 IEEE 27th International Conference on Advanced Information Networking and Applications, AINA 7813, pages 629-636.
  10. OSGi Alliance (2013). Open Service Gateway initiative (OSGi). Available at http://www.osgi.org/.
  11. Petri, I., Montes, J. D., Zou, M., Rana, O. F., Beach, T., Li, H., and Rezgui, Y. (2014). In-transit data analysis and distribution in a multi-cloud environment using cometcloud. In International Conference on Future Internet of Things and Cloud, FiCloud 2014, pages 471-476.
  12. Wright, P. and Manieri, A. (2014). Internet of things in the cloud - theory and practice. In CLOSER 2014 - Proceedings of the 4th International Conference on Cloud Computing and Services Science, pages 164-169.
Download


Paper Citation


in Harvard Style

Cavallo M., Cusma' L., Di Modica G., Polito C. and Tomarchio O. (2016). A Hadoop based Framework to Process Geo-distributed Big Data . In Proceedings of the 6th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER, ISBN 978-989-758-182-3, pages 178-185. DOI: 10.5220/0005806101780185


in Bibtex Style

@conference{closer16,
author={Marco Cavallo and Lorenzo Cusma' and Giuseppe Di Modica and Carmelo Polito and Orazio Tomarchio},
title={A Hadoop based Framework to Process Geo-distributed Big Data},
booktitle={Proceedings of the 6th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER,},
year={2016},
pages={178-185},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005806101780185},
isbn={978-989-758-182-3},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 6th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER,
TI - A Hadoop based Framework to Process Geo-distributed Big Data
SN - 978-989-758-182-3
AU - Cavallo M.
AU - Cusma' L.
AU - Di Modica G.
AU - Polito C.
AU - Tomarchio O.
PY - 2016
SP - 178
EP - 185
DO - 10.5220/0005806101780185