A NUMA Aware SparkTM on Many-cores and Large Memory Servers

François Waeselynck, Benoit Pelletier

2019

Abstract

Within the scope of the CloudDBAppliance project, we investigate how Apache SparkTM can leverage a many cores and large memory platform, with a scale up approach as opposed to the commonly used scale out one: that is, the approach is to deploy a spark cluster to few large servers with many cores (up to several hundreds) and large memory (up to several tera-byte), rather than spreading it on many vanilla servers, and to stack several Spark executor processes per cluster node when running a job. It requires to cope with the non-uniform memory access within such servers, so we inculcate NUMA awareness to Spark, that provides a smart and application transparent placement of executor processes. We experiment it on a BullSequanaTM S series platform with the Intel HiBench suite benchmark and compare performance where NUMA awareness is off or on.

Paper Citation

in Harvard Style

Waeselynck F. and Pelletier B. (2019). A NUMA Aware SparkTM on Many-cores and Large Memory Servers.In Proceedings of the 9th International Conference on Cloud Computing and Services Science - Volume 1: ADITCA, ISBN 978-989-758-365-0, pages 648-653. DOI: 10.5220/0007905506480653

in Bibtex Style

@conference{aditca19,
author={François Waeselynck and Benoit Pelletier},
title={A NUMA Aware SparkTM on Many-cores and Large Memory Servers},
booktitle={Proceedings of the 9th International Conference on Cloud Computing and Services Science - Volume 1: ADITCA,},
year={2019},
pages={648-653},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0007905506480653},
isbn={978-989-758-365-0},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 9th International Conference on Cloud Computing and Services Science - Volume 1: ADITCA,
TI - A NUMA Aware SparkTM on Many-cores and Large Memory Servers
SN - 978-989-758-365-0
AU - Waeselynck F.
AU - Pelletier B.
PY - 2019
SP - 648
EP - 653
DO - 10.5220/0007905506480653