Distributed File System Based on Erasure Coding for I/O-Intensive Applications

Dimitri Pertin, Sylvain David, Pierre Evenou, Benoît Parrein, Nicolas Normand

2014

Abstract

Distributed storage systems take advantage of the network, storage and computational resources to provide a scalable infrastructure. But in such large system, failures are frequent and expected. Data replication is the common technique to provide fault-tolerance but suffers from its important storage consumption. Erasure coding is an alternative that offers the same data protection but reduces significantly the storage consumption. As it entails additional workload, current storage providers limit its use for longterm storage. We present the Mojette Transform (MT), an erasure code whose computations rely on fast XOR operations. The MT is part of RozoFS, a distributed file system that provides a global namespace relying on a cluster of storage nodes. This work is part of our ongoing effort to prove that erasure coding is not necessarily a bottleneck for intense I/O applications. In order to validate our approach, we consider a case study involving a storage cluster of RozoFS that supports video editing as an I/O intensive application.

References

  1. Fan, B., Tantisiriroj, W., Xiao, L., and Gibson, G. (2009). Diskreduce: Raid for data-intensive scalable computing. In Proc. PDSW 2009.
  2. Guédon, J. P., Parrein, B., and Normand, N. (2001). Internet distributed image information system. Integr. Comput.-Aided Eng., 8(3):205-214.
  3. Normand, N., Guédon, J., Philippé, O., and Barba, D. Controlled redundancy for image coding and high-speed transmission. In Proc. VCIP 1996, Orlando, FL.
  4. Parrein, B., Normand, N., and Guédon, J. P. (2001). Multiple description coding using exact discrete Radon transform. In Proceedings of the Data Compression Conference, DCC 7801, pages 508-.
  5. Plank, J. S. (2007). Jerasure: A library in C/C++ facilitating erasure coding for storage applications. Technical Report CS-07-603, University of Tennessee.
  6. Weil, S. A., Brandt, S. A., Miller, E. L., Long, D. D. E., and Maltzahn, C. (2006). Ceph: A scalable, highperformance distributed file system. In Proc. OSDI 2006.
Download


Paper Citation


in Harvard Style

Pertin D., David S., Evenou P., Parrein B. and Normand N. (2014). Distributed File System Based on Erasure Coding for I/O-Intensive Applications . In Proceedings of the 4th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER, ISBN 978-989-758-019-2, pages 451-456. DOI: 10.5220/0004960604510456


in Bibtex Style

@conference{closer14,
author={Dimitri Pertin and Sylvain David and Pierre Evenou and Benoît Parrein and Nicolas Normand},
title={Distributed File System Based on Erasure Coding for I/O-Intensive Applications},
booktitle={Proceedings of the 4th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER,},
year={2014},
pages={451-456},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0004960604510456},
isbn={978-989-758-019-2},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 4th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER,
TI - Distributed File System Based on Erasure Coding for I/O-Intensive Applications
SN - 978-989-758-019-2
AU - Pertin D.
AU - David S.
AU - Evenou P.
AU - Parrein B.
AU - Normand N.
PY - 2014
SP - 451
EP - 456
DO - 10.5220/0004960604510456