loading
Papers

Research.Publish.Connect.

Paper

Authors: Sheng-Tzong Cheng ; Jian-Ting Chen and Yin-Chun Chen

Affiliation: Department of Computer Science and Information Engineering, National Cheng Kung University, Taiwan

ISBN: 978-989-758-238-7

Keyword(s): Big Data, Deduplication, In-Memory Computing, Spark.

Abstract: In this information era, it is difficult to exploit and compute high-amount data efficiently. Today, it is inadequate to use MapReduce to handle more data in less time let alone real time. Hence, In-memory Computing (IMC) was introduced to solve the problem of Hadoop MapReduce. IMC, as its literal meaning, exploits computing in memory to tackle the cost problem which Hadoop undue access data to disk caused and can be distributed to perform iterative operations. However, IMC distributed computing still cannot get rid of a bottleneck, that is, network bandwidth. It restricts the speed of receiving the information from the source and dispersing information to each node. According to observation, some data from sensor devices might be duplicate due to time or space dependence. Therefore, deduplication technology would be a good solution. The technique for eliminating duplicated data is capable of improving data utilization. This study presents a distributed real-time IMC platform -- “Spar k Streaming” optimization. It uses deduplication technology to eliminate the possible duplicate blocks from source. It is expected to reduce redundant data transmission and improve the throughput of Spark Streaming. (More)

PDF ImageFull Text

Download
CC BY-NC-ND 4.0

Sign In Guest: Register as new SciTePress user now for free.

Sign In SciTePress user: please login.

PDF ImageMy Papers

You are not signed in, therefore limits apply to your IP address 3.209.80.87

In the current month:
Recent papers: 100 available of 100 total
2+ years older papers: 200 available of 200 total

Paper citation in several formats:
Cheng S.; Chen J.; Chen Y. and (2017). Fast Deduplication Data Transmission Scheme on a Big Data Real-Time Platform.In Proceedings of the Seventh International Symposium on Business Modeling and Software Design - Volume 1: BMSD, ISBN 978-989-758-238-7, pages 155-164. DOI: 10.5220/0006528401550164

@conference{bmsd17,
author={Sheng{-}Tzong Cheng and Jian{-}Ting Chen and Yin{-}Chun Chen},
title={Fast Deduplication Data Transmission Scheme on a Big Data Real-Time Platform},
booktitle={Proceedings of the Seventh International Symposium on Business Modeling and Software Design - Volume 1: BMSD,},
year={2017},
pages={155-164},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006528401550164},
isbn={978-989-758-238-7},
}

TY - CONF

JO - Proceedings of the Seventh International Symposium on Business Modeling and Software Design - Volume 1: BMSD,
TI - Fast Deduplication Data Transmission Scheme on a Big Data Real-Time Platform
SN - 978-989-758-238-7
AU - Cheng, S.
AU - Chen, J.
AU - Chen, Y.
PY - 2017
SP - 155
EP - 164
DO - 10.5220/0006528401550164

Login or register to post comments.

Comments on this Paper: Be the first to review this paper.