Adaptive Resource Management for Distributed Data Analytics based on Container-level Cluster Monitoring

Thomas Renner, Lauritz Thamsen, Odej Kao

2017

Abstract

Many distributed data analysis jobs are executed repeatedly in production clusters. Examples include daily executed batch jobs and iterative programs. These jobs present an opportunity to learn workload characteristics through continuous fine-grained cluster monitoring. Therefore, based on detailed profiles of resource utilization, data placement, and job runtimes, resource management can in fact adapt to actual workloads. In this paper, we present a system architecture that contains four mechanisms for an adaptive resource management, encompassing data placement, resource allocation, and container as well as job scheduling. In particular, we extended Apache Hadoop's scheduling and data placement to improve resource utilization and job runtimes for recurring analytics jobs. Furthermore, we developed a Hadoop submission tool that allows users to reserve resources for specific target runtimes and which uses historical data available from cluster monitoring for predictions.

Paper Citation

in Harvard Style

Renner T., Thamsen L. and Kao O. (2017). Adaptive Resource Management for Distributed Data Analytics based on Container-level Cluster Monitoring . In Proceedings of the 6th International Conference on Data Science, Technology and Applications - Volume 1: DATA, ISBN 978-989-758-255-4, pages 38-47. DOI: 10.5220/0006420100380047

in Bibtex Style

@conference{data17,
author={Thomas Renner and Lauritz Thamsen and Odej Kao},
title={Adaptive Resource Management for Distributed Data Analytics based on Container-level Cluster Monitoring},
booktitle={Proceedings of the 6th International Conference on Data Science, Technology and Applications - Volume 1: DATA,},
year={2017},
pages={38-47},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006420100380047},
isbn={978-989-758-255-4},
}

in EndNote Style

TY - CONF
JO - Proceedings of the 6th International Conference on Data Science, Technology and Applications - Volume 1: DATA,
TI - Adaptive Resource Management for Distributed Data Analytics based on Container-level Cluster Monitoring
SN - 978-989-758-255-4
AU - Renner T.
AU - Thamsen L.
AU - Kao O.
PY - 2017
SP - 38
EP - 47
DO - 10.5220/0006420100380047