Low Level Big Data Processing

Jaime Salvador-Meneses, Zoila Ruiz-Chavez, Jose Garcia-Rodriguez

2018

Abstract

The machine learning algorithms, prior to their application, require that the information be stored in memory. Reducing the amount of memory used for data representation clearly reduces the number of operations required to process it. Many of the current libraries represent the information in the traditional way, which forces you to iterate the whole set of data to obtain the desired result. In this paper we propose a technique to process categorical information previously encoded using the bit-level schema, the method proposes a block processing which reduces the number of iterations on the original data and, at the same time, maintains a processing performance similar to the processing of the original data. The method requires the information to be stored in memory, which allows you to optimize the volume of memory consumed for representation as well as the operations required to process it. The results of the experiments carried out show a slightly lower time processing than the obtained with traditional implementations, which allows us to obtain a good performance.

Paper Citation

in Harvard Style

Salvador-Meneses J., Ruiz-Chavez Z. and Garcia-Rodriguez J. (2018). Low Level Big Data Processing. In Proceedings of the 10th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2018) - Volume 1: KDIR; ISBN 978-989-758-330-8, SciTePress, pages 347-352. DOI: 10.5220/0007227103470352

in Bibtex Style

@conference{kdir18,
author={Jaime Salvador-Meneses and Zoila Ruiz-Chavez and Jose Garcia-Rodriguez},
title={Low Level Big Data Processing},
booktitle={Proceedings of the 10th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2018) - Volume 1: KDIR},
year={2018},
pages={347-352},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0007227103470352},
isbn={978-989-758-330-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 10th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2018) - Volume 1: KDIR
TI - Low Level Big Data Processing
SN - 978-989-758-330-8
AU - Salvador-Meneses J.
AU - Ruiz-Chavez Z.
AU - Garcia-Rodriguez J.
PY - 2018
SP - 347
EP - 352
DO - 10.5220/0007227103470352
PB - SciTePress