Fast Deduplication Data Transmission Scheme on a Big Data Real-Time Platform

Sheng-Tzong Cheng, Jian-Ting Chen, Yin-Chun Chen

2017

Abstract

In this information era, it is difficult to exploit and compute high-amount data efficiently. Today, it is inadequate to use MapReduce to handle more data in less time let alone real time. Hence, In-memory Computing (IMC) was introduced to solve the problem of Hadoop MapReduce. IMC, as its literal meaning, exploits computing in memory to tackle the cost problem which Hadoop undue access data to disk caused and can be distributed to perform iterative operations. However, IMC distributed computing still cannot get rid of a bottleneck, that is, network bandwidth. It restricts the speed of receiving the information from the source and dispersing information to each node. According to observation, some data from sensor devices might be duplicate due to time or space dependence. Therefore, deduplication technology would be a good solution. The technique for eliminating duplicated data is capable of improving data utilization. This study presents a distributed real-time IMC platform -- “Spark Streaming” optimization. It uses deduplication technology to eliminate the possible duplicate blocks from source. It is expected to reduce redundant data transmission and improve the throughput of Spark Streaming.

Paper Citation

in Harvard Style

Cheng S., Chen J. and Chen Y. (2017). Fast Deduplication Data Transmission Scheme on a Big Data Real-Time Platform. In Proceedings of the Seventh International Symposium on Business Modeling and Software Design - Volume 1: BMSD, ISBN 978-989-758-238-7, pages 155-164. DOI: 10.5220/0006528401550164

in Bibtex Style

@conference{bmsd17,
author={Sheng-Tzong Cheng and Jian-Ting Chen and Yin-Chun Chen},
title={Fast Deduplication Data Transmission Scheme on a Big Data Real-Time Platform},
booktitle={Proceedings of the Seventh International Symposium on Business Modeling and Software Design - Volume 1: BMSD,},
year={2017},
pages={155-164},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006528401550164},
isbn={978-989-758-238-7},
}

in EndNote Style

TY - CONF

JO - Proceedings of the Seventh International Symposium on Business Modeling and Software Design - Volume 1: BMSD,
TI - Fast Deduplication Data Transmission Scheme on a Big Data Real-Time Platform
SN - 978-989-758-238-7
AU - Cheng S.
AU - Chen J.
AU - Chen Y.
PY - 2017
SP - 155
EP - 164
DO - 10.5220/0006528401550164