Scheduling of Streaming Data Processing with Overload of Resources using Genetic Algorithm

Mikhail Melnik, Denis Nasonov, Nikolay Butakov

2018

Abstract

The growing demand for processing of streaming data contributes to the development of distributed streaming platforms, such as Apache Storm or Flink. However, the volume of data and complexity of their processing is growing extremely fast, which poses new challenges and tasks for developing new tools and methods for improving the efficiency of streaming data processing. One of the main ways to improve a system performance is an effective scheduling and a proper configuration of the computing platform. Running large-scale streaming applications, especially in the clouds, requires a high cost of computing resources and additional efforts to deploy and support an application itself. Thus, there is a need for an opportunity to estimate the performance of the system and its behaviour before real calculations are made. Therefore, in this work we propose a model for distributed data stream processing, stream scheduling problem statement and a developed simulator of the streaming platform, immediately allowing to explore the behaviour of the system under various conditions. In addition, we propose a genetic algorithm for efficient stream scheduling and conducting experimental studies.

Paper Citation

in Harvard Style

Melnik M., Nasonov D. and Butakov N. (2018). Scheduling of Streaming Data Processing with Overload of Resources using Genetic Algorithm. In Proceedings of the 10th International Joint Conference on Computational Intelligence (IJCCI 2018) - Volume 1: IJCCI; ISBN 978-989-758-327-8, SciTePress, pages 232-241. DOI: 10.5220/0006951902320241

in Bibtex Style

@conference{ijcci18,
author={Mikhail Melnik and Denis Nasonov and Nikolay Butakov},
title={Scheduling of Streaming Data Processing with Overload of Resources using Genetic Algorithm},
booktitle={Proceedings of the 10th International Joint Conference on Computational Intelligence (IJCCI 2018) - Volume 1: IJCCI},
year={2018},
pages={232-241},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006951902320241},
isbn={978-989-758-327-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 10th International Joint Conference on Computational Intelligence (IJCCI 2018) - Volume 1: IJCCI
TI - Scheduling of Streaming Data Processing with Overload of Resources using Genetic Algorithm
SN - 978-989-758-327-8
AU - Melnik M.
AU - Nasonov D.
AU - Butakov N.
PY - 2018
SP - 232
EP - 241
DO - 10.5220/0006951902320241
PB - SciTePress