loading
Papers Papers/2022 Papers Papers/2022

Research.Publish.Connect.

Paper

Paper Unlock

Authors: Thomas Renner ; Lauritz Thamsen and Odej Kao

Affiliation: Complex and Distributed IT Systems and Technische Universität Berlin, Germany

Keyword(s): Cluster Resource Management, Distributed Data Analytics, Distributed Dataflow Systems, Adaptive Resource Management.

Abstract: Many distributed data analysis jobs are executed repeatedly in production clusters. Examples include daily executed batch jobs and iterative programs. These jobs present an opportunity to learn workload characteristics through continuous fine-grained cluster monitoring. Therefore, based on detailed profiles of resource utilization, data placement, and job runtimes, resource management can in fact adapt to actual workloads. In this paper, we present a system architecture that contains four mechanisms for an adaptive resource management, encompassing data placement, resource allocation, and container as well as job scheduling. In particular, we extended Apache Hadoop's scheduling and data placement to improve resource utilization and job runtimes for recurring analytics jobs. Furthermore, we developed a Hadoop submission tool that allows users to reserve resources for specific target runtimes and which uses historical data available from cluster monitoring for predictions.

CC BY-NC-ND 4.0

Sign In Guest: Register as new SciTePress user now for free.

Sign In SciTePress user: please login.

PDF ImageMy Papers

You are not signed in, therefore limits apply to your IP address 3.230.1.23

In the current month:
Recent papers: 100 available of 100 total
2+ years older papers: 200 available of 200 total

Paper citation in several formats:
Renner, T.; Thamsen, L. and Kao, O. (2017). Adaptive Resource Management for Distributed Data Analytics based on Container-level Cluster Monitoring. In Proceedings of the 6th International Conference on Data Science, Technology and Applications - DATA; ISBN 978-989-758-255-4; ISSN 2184-285X, SciTePress, pages 38-47. DOI: 10.5220/0006420100380047

@conference{data17,
author={Thomas Renner. and Lauritz Thamsen. and Odej Kao.},
title={Adaptive Resource Management for Distributed Data Analytics based on Container-level Cluster Monitoring},
booktitle={Proceedings of the 6th International Conference on Data Science, Technology and Applications - DATA},
year={2017},
pages={38-47},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006420100380047},
isbn={978-989-758-255-4},
issn={2184-285X},
}

TY - CONF

JO - Proceedings of the 6th International Conference on Data Science, Technology and Applications - DATA
TI - Adaptive Resource Management for Distributed Data Analytics based on Container-level Cluster Monitoring
SN - 978-989-758-255-4
IS - 2184-285X
AU - Renner, T.
AU - Thamsen, L.
AU - Kao, O.
PY - 2017
SP - 38
EP - 47
DO - 10.5220/0006420100380047
PB - SciTePress