Non-functional Requirements for Real World Big Data Systems - An Investigation of Big Data Architectures at Facebook, Twitter and Netflix

Thalita Vergilio, Muthu Ramachandran

2018

Abstract

This research represents a unique contribution to the field of Software Engineering for Big Data in the form of an investigation of the big data architectures of three well-known real-world companies: Facebook, Twitter and Netflix. The purpose of this investigation is to gather significant non-functional requirements for real-world big data systems, with an aim to addressing these requirements in the design of our own unique architecture for big data processing in the cloud: MC-BDP (Multi-Cloud Big Data Processing). MC-BDP represents an evolution of the PaaS-BDP architectural pattern, previously developed by the authors. However, its presentation is not within the scope of this paper. The scope of this comparative study is limited to the examination of academic papers, technical blogs, presentations, source code and documentation officially published by the companies under investigation. Ten non-functional requirements are identified and discussed in the context of these companies’ architectures: batch data, stream data, late and out-of-order data, processing guarantees, integration and extensibility, distribution and scalability, cloud support and elasticity, fault-tolerance, flow control, and flexibility and technology agnosticism. They are followed by the conclusion and considerations for future work.

Paper Citation

in Harvard Style

Vergilio T. and Ramachandran M. (2018). Non-functional Requirements for Real World Big Data Systems - An Investigation of Big Data Architectures at Facebook, Twitter and Netflix.In Proceedings of the 13th International Conference on Software Technologies - Volume 1: SE-CLOUD, ISBN 978-989-758-320-9, pages 833-840. DOI: 10.5220/0006825408330840

in Bibtex Style

@conference{se-cloud18,
author={Thalita Vergilio and Muthu Ramachandran},
title={Non-functional Requirements for Real World Big Data Systems - An Investigation of Big Data Architectures at Facebook, Twitter and Netflix},
booktitle={Proceedings of the 13th International Conference on Software Technologies - Volume 1: SE-CLOUD,},
year={2018},
pages={833-840},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006825408330840},
isbn={978-989-758-320-9},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 13th International Conference on Software Technologies - Volume 1: SE-CLOUD,
TI - Non-functional Requirements for Real World Big Data Systems - An Investigation of Big Data Architectures at Facebook, Twitter and Netflix
SN - 978-989-758-320-9
AU - Vergilio T.
AU - Ramachandran M.
PY - 2018
SP - 833
EP - 840
DO - 10.5220/0006825408330840