A Comparison of the Efficiencies of Various Structured and Semi- Structured Data Formats in Data Analysis and Big Data Analytic Development

Heather Graham, Taoxin Peng

2024

Abstract

As data volumes grow, so too does our need and ability to analyse it. Cloud computing technologies offer a wide variety of options for analysing big data and make this ability available to anyone. However, the monetary implications for doing this in an inefficient fashion could surprise those who may be used to an on-premises solution to big data analysis, as they move from a model where storage is limited and processing power has little cost implications, to a model where storage is cheap but compute is expensive. This paper investigates the efficiencies gained or lost by using each of five data formats, CSV, JSON, Parquet, ORC and Avro, on Amazon Athena, which uses SQL as a query language over data at rest in Amazon S3, and on Amazon EMR, using the Pig language over a distributed Hadoop architecture. Experiment results suggest that ORC is the most efficient data format to use on the platforms tested against, based on time and monetary costs.

Paper Citation

in Harvard Style

Graham H. and Peng T. (2024). A Comparison of the Efficiencies of Various Structured and Semi- Structured Data Formats in Data Analysis and Big Data Analytic Development. In Proceedings of the 13th International Conference on Data Science, Technology and Applications - Volume 1: DATA; ISBN 978-989-758-707-8, SciTePress, pages 301-308. DOI: 10.5220/0012756300003756

in Bibtex Style

@conference{data24,
author={Heather Graham and Taoxin Peng},
title={A Comparison of the Efficiencies of Various Structured and Semi- Structured Data Formats in Data Analysis and Big Data Analytic Development},
booktitle={Proceedings of the 13th International Conference on Data Science, Technology and Applications - Volume 1: DATA},
year={2024},
pages={301-308},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012756300003756},
isbn={978-989-758-707-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 13th International Conference on Data Science, Technology and Applications - Volume 1: DATA
TI - A Comparison of the Efficiencies of Various Structured and Semi- Structured Data Formats in Data Analysis and Big Data Analytic Development
SN - 978-989-758-707-8
AU - Graham H.
AU - Peng T.
PY - 2024
SP - 301
EP - 308
DO - 10.5220/0012756300003756
PB - SciTePress