Tackling the Six Fundamental Challenges of Big Data in Research Projects by Utilizing a Scalable and Modular Architecture

Andreas Freymann, Florian Maier, Kristian Schaefer, Tom Böhnel

2020

Abstract

Over the last decades the necessity for processing and storing huge amounts of data has increased enormously, especially in the fundamental research area. Beside the management of large volumes of data, research projects are facing additional fundamental challenges in terms of data velocity, data variety and data veracity to create meaningful data value. In order to cope with these challenges solutions exist. However, they often show shortcomings in adaptability, usability or have high licence fees. Thus, this paper proposes a scalable and modular architecture based on open source technologies using micro-services which are deployed using Docker. The proposed architecture has been adopted, deployed and tested within a current research project. In addition, the deployment and handling is compared with another technology. The results show an overcoming of the fundamental challenges of processing huge amounts of data and the handling of Big Data in research projects.

Paper Citation

in Harvard Style

Freymann A., Maier F., Schaefer K. and Böhnel T. (2020). Tackling the Six Fundamental Challenges of Big Data in Research Projects by Utilizing a Scalable and Modular Architecture.In Proceedings of the 5th International Conference on Internet of Things, Big Data and Security - Volume 1: IoTBDS, ISBN 978-989-758-426-8, pages 249-256. DOI: 10.5220/0009388602490256

in Bibtex Style

@conference{iotbds20,
author={Andreas Freymann and Florian Maier and Kristian Schaefer and Tom Böhnel},
title={Tackling the Six Fundamental Challenges of Big Data in Research Projects by Utilizing a Scalable and Modular Architecture},
booktitle={Proceedings of the 5th International Conference on Internet of Things, Big Data and Security - Volume 1: IoTBDS,},
year={2020},
pages={249-256},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0009388602490256},
isbn={978-989-758-426-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 5th International Conference on Internet of Things, Big Data and Security - Volume 1: IoTBDS,
TI - Tackling the Six Fundamental Challenges of Big Data in Research Projects by Utilizing a Scalable and Modular Architecture
SN - 978-989-758-426-8
AU - Freymann A.
AU - Maier F.
AU - Schaefer K.
AU - Böhnel T.
PY - 2020
SP - 249
EP - 256
DO - 10.5220/0009388602490256