Approach to Deploying Batch File Data Products in a Big Data Environment

Richard de Arruda Felix, Patricia Della Méa Plentz, Jean Hauck

2025

Abstract

Data science has become essential across industries such as government, healthcare, and finance, driving decision-making through large-scale data analysis. Deploying batch data products, like the periodic calculation of credit scores for millions, presents significant challenges, including integration with existing big data architectures and ensuring scalability and efficiency. This study proposes an optimized approach that leverages software engineering and agile methodologies to streamline the deployment of such products. Validated through action research conducted at a Brazilian credit bureau, the approach demonstrated a substantial reduction in deployment time by improving documentation, development, and testing processes, offering a scalable solution to modern batch data processing challenges.

Paper Citation

in Harvard Style

Felix R., Plentz P. and Hauck J. (2025). Approach to Deploying Batch File Data Products in a Big Data Environment. In Proceedings of the 10th International Conference on Internet of Things, Big Data and Security - Volume 1: IoTBDS; ISBN 978-989-758-750-4, SciTePress, pages 93-104. DOI: 10.5220/0013293100003944

in Bibtex Style

@conference{iotbds25,
author={Richard Felix and Patricia Plentz and Jean Hauck},
title={Approach to Deploying Batch File Data Products in a Big Data Environment},
booktitle={Proceedings of the 10th International Conference on Internet of Things, Big Data and Security - Volume 1: IoTBDS},
year={2025},
pages={93-104},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013293100003944},
isbn={978-989-758-750-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 10th International Conference on Internet of Things, Big Data and Security - Volume 1: IoTBDS
TI - Approach to Deploying Batch File Data Products in a Big Data Environment
SN - 978-989-758-750-4
AU - Felix R.
AU - Plentz P.
AU - Hauck J.
PY - 2025
SP - 93
EP - 104
DO - 10.5220/0013293100003944
PB - SciTePress