A Platform for Interactive Data Science with Apache Spark for On-premises Infrastructure

Rafal Lokuciejewski, Dominik Schüssele, Florian Wilhelm, Sven Groppe

2021

Abstract

Various cloud providers offer integrated platforms for interactive development in notebooks for processing and analysis of Big Data on large compute clusters. Such platforms enable users to easily leverage frameworks like Apache Spark as well as to manage cluster resources. However, Data Scientists and Engineers are facing the lack of a similar holistic solution when working with on-premises infrastructure. Especially a central point of administration to access a notebooks’ UI, manage notebook kernels, allocate resources for frameworks like Apache Spark or monitor cluster workloads, in general, is currently missing for on-premises infrastructure. To overcome these issues and provide on-premises users with a platform for interactive development, we propose a cross-cluster architecture resulting from an extensive requirements engineering process. Based on open-source components, the designed platform provides an intuitive Web-UI that enables users to easily access notebooks, manage custom kernel-environments as well as monitor cluster resources and current workloads. Besides an admin panel for user restrictions, the platform provides isolation of user workloads and scalability by design. The designed platform is evaluated against prior solutions for on-premises as well as from a user perspective by utilizing the User Experience Questionnaire, an independent benchmark tool for interactive products.

Paper Citation

in Harvard Style

Lokuciejewski R., Schüssele D., Wilhelm F. and Groppe S. (2021). A Platform for Interactive Data Science with Apache Spark for On-premises Infrastructure. In Proceedings of the 11th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER, ISBN 978-989-758-510-4, pages 65-76. DOI: 10.5220/0010447500650076

in Bibtex Style

@conference{closer21,
author={Rafal Lokuciejewski and Dominik Schüssele and Florian Wilhelm and Sven Groppe},
title={A Platform for Interactive Data Science with Apache Spark for On-premises Infrastructure},
booktitle={Proceedings of the 11th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER,},
year={2021},
pages={65-76},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010447500650076},
isbn={978-989-758-510-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 11th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER,
TI - A Platform for Interactive Data Science with Apache Spark for On-premises Infrastructure
SN - 978-989-758-510-4
AU - Lokuciejewski R.
AU - Schüssele D.
AU - Wilhelm F.
AU - Groppe S.
PY - 2021
SP - 65
EP - 76
DO - 10.5220/0010447500650076