loading
Papers Papers/2022 Papers Papers/2022

Research.Publish.Connect.

Paper

Paper Unlock

Authors: Andreas Freymann ; Florian Maier ; Kristian Schaefer and Tom Böhnel

Affiliation: Anwendungszentrum KEIM, Fraunhofer Institute for Industrial Engineering IAO, Esslingen am Neckar, Germany

Keyword(s): Big Data Fundamentals, Scalability, Modular Architecture, Research Projects, Data Lake, Real Time, Open Source, Docker Swarm, Micro Services.

Abstract: Over the last decades the necessity for processing and storing huge amounts of data has increased enormously, especially in the fundamental research area. Beside the management of large volumes of data, research projects are facing additional fundamental challenges in terms of data velocity, data variety and data veracity to create meaningful data value. In order to cope with these challenges solutions exist. However, they often show shortcomings in adaptability, usability or have high licence fees. Thus, this paper proposes a scalable and modular architecture based on open source technologies using micro-services which are deployed using Docker. The proposed architecture has been adopted, deployed and tested within a current research project. In addition, the deployment and handling is compared with another technology. The results show an overcoming of the fundamental challenges of processing huge amounts of data and the handling of Big Data in research projects.

CC BY-NC-ND 4.0

Sign In Guest: Register as new SciTePress user now for free.

Sign In SciTePress user: please login.

PDF ImageMy Papers

You are not signed in, therefore limits apply to your IP address 107.21.137.184

In the current month:
Recent papers: 100 available of 100 total
2+ years older papers: 200 available of 200 total

Paper citation in several formats:
Freymann, A.; Maier, F.; Schaefer, K. and Böhnel, T. (2020). Tackling the Six Fundamental Challenges of Big Data in Research Projects by Utilizing a Scalable and Modular Architecture. In Proceedings of the 5th International Conference on Internet of Things, Big Data and Security - IoTBDS; ISBN 978-989-758-426-8; ISSN 2184-4976, SciTePress, pages 249-256. DOI: 10.5220/0009388602490256

@conference{iotbds20,
author={Andreas Freymann. and Florian Maier. and Kristian Schaefer. and Tom Böhnel.},
title={Tackling the Six Fundamental Challenges of Big Data in Research Projects by Utilizing a Scalable and Modular Architecture},
booktitle={Proceedings of the 5th International Conference on Internet of Things, Big Data and Security - IoTBDS},
year={2020},
pages={249-256},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0009388602490256},
isbn={978-989-758-426-8},
issn={2184-4976},
}

TY - CONF

JO - Proceedings of the 5th International Conference on Internet of Things, Big Data and Security - IoTBDS
TI - Tackling the Six Fundamental Challenges of Big Data in Research Projects by Utilizing a Scalable and Modular Architecture
SN - 978-989-758-426-8
IS - 2184-4976
AU - Freymann, A.
AU - Maier, F.
AU - Schaefer, K.
AU - Böhnel, T.
PY - 2020
SP - 249
EP - 256
DO - 10.5220/0009388602490256
PB - SciTePress