A Data-aware MultiWorkflow Scheduler for Clusters on WorkflowSim

César Acevedo, Porfidio Hernández, Antonio Espinosa, Victor Mendez


Most scientific workflows are defined as Direct Acyclic Graphs. Despite DAGs are very expressive to reflect dependencies relationships, current approaches are not aware of the storage physiognomy in terms of performance and capacity. Provide information about temporal storage allocation on data intensive applications helps to avoid performance issues. Nevertheless, we need to evaluate several combinations of data file locations and application scheduling. Simulation is one of the most popular evaluation methods in scientific workflow execution to develop new storage-aware scheduling techniques or improve existing ones, to test scalability and repetitiveness. This paper presents a multiworkflow store-aware scheduler policy as an extension of WorkflowSim, enabling its combination with other WorkflowSim scheduling policies and the possibility of evaluating a wide range of storage and file allocation possibilities. This paper also presents a proof of concept of a real world implementation of a storage-aware scheduler to validate the accuracy of the WorkflowSim extension and the scalability of our scheduler technique. The evaluation on several environments shows promising results up to 69% of makespan improvement on simulated large scale clusters with an error of the WorflowSim extension between 0,9% and 3% comparing with the real infrastructure implementation.


  1. Acevedo, C., Hernandez, P., Espinosa, A., and Mendez, V. (2016). A data-aware multiworkflow cluster scheduler. In Proceedings of the 1st International Conference on Complex Information Systems, pages 95-102. SCITEPRESS.
  2. Ananthanarayanan, G., Ghodsi, A., Wang, A., Borthakur, D., Kandula, S., Shenker, S., and Stoica, I. (2012). Pacman: coordinated memory caching for parallel jobs. In Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation, pages 20-20. USENIX Association.
  3. Barbosa, J. and Monteiro, A. (2008). A list scheduling algorithm for scheduling multi-user jobs on clusters. High Performance Computing for Computational ScienceVECPAR 2008, pages 123--136.
  4. Bolze, R., Desprez, F., and Isnard, B. (2009). Evaluation of Online Multi-Workflow Heuristics based on ListScheduling Algorithms. Gwendia report L.
  5. Bryk, P., Malawski, M., Juve, G., and Deelman, E. (2016). Storage-aware algorithms for scheduling of workflow ensembles in clouds. Journal of Grid Computing, 14(2):359-378.
  6. Calheiros, R. N., Ranjan, R., Beloglazov, A., De Rose, C. A., and Buyya, R. (2011). Cloudsim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Software: Practice and Experience, 41(1):23- 50.
  7. Chen, W. and Deelman, E. (2012). WorkflowSim: A toolkit for simulating scientific workflows in distributed environments. In 2012 IEEE 8th International Conference on E-Science, e-Science 2012.
  8. Costa, L. B., Yang, H., Vairavanathan, E., Barros, A., Maheshwari, K., Fedak, G., Katz, D., Wilde, M., Ripeanu, M., and Al-Kiswany, S. (2015). The case for workflow-aware storage: An opportunity study. Journal of Grid Computing, 13(1):95-113.
  9. Dean, J. and Ghemawat, S. (2008). Mapreduce: Simplified data processing on large clusters. Communications of the ACM, 51(1):107-113.
  10. Delgado Peris, A., Hernández, J. M., and Huedo, E. (2016). Distributed late-binding scheduling and cooperative data caching. Journal of Grid Computing, pages 1- 22.
  11. Goecks, J., Nekrutenko, A., Taylor, J., and Team, T. G. (2010). Galaxy : a comprehensive approach for supporting accessible , reproducible , and transparent computational research in the life sciences. Genome biology.
  12. Hirales-Carbajal, A., Tchernykh, A., Röblitz, T., and Yahyapour, R. (2010). A grid simulation framework to study advance scheduling strategies for complex workflow applications. In Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW), 2010 IEEE International Symposium on, pages 1-8. IEEE.
  13. Hönig, U. and Schiffmann, W. (2006). A meta-algorithm for scheduling multiple dags in homogeneous system environments. In Proceedings of the eighteenth IASTED International Conference on Parallel and Distributed Computing and Systems (PDCS'06).
  14. Ilavarasan, E. and Thambidurai, P. (2007). Low complexity performance effective task scheduling algorithm for heterogeneous computing environments. Journal of Computer sciences.
  15. Merdan, M., Moser, T., Wahyudin, D., Biffl, S., and Vrba, P. (2008). Simulation of workflow scheduling strategies using the mast test management system. In Control, Automation, Robotics and Vision, 2008. ICARCV 2008. 10th International Conference on, pages 1172- 1177. IEEE.
  16. Ousterhout, J., Agrawal, P., Erickson, D., Kozyrakis, C., Leverich, J., Mazières, D., Mitra, S., Narayanan, A., Ongaro, D., Parulkar, G., et al. (2011). The case for ramcloud. Communications of the ACM, 54(7):121- 130.
  17. Ramakrishnan, A., Singh, G., Zhao, H., Deelman, E., Sakellariou, R., Vahi, K., Blackburn, K., Meyers, D., and Samidi, M. (2007). Scheduling data-intensive workflows onto storage-constrained distributed resources. In Seventh IEEE International Symposium on Cluster Computing and the Grid (CCGrid'07), pages 401-409. IEEE.
  18. Shankar, S. and DeWitt, D. J. (2007). Data driven workflow planning in cluster management systems. In Proceedings of the 16th international symposium on High performance distributed computing, pages 127-136. ACM.
  19. Topcuoglu, H., Hariri, S., and Wu, M.-Y. (1999). Task scheduling algorithms for heterogeneous processors. In Heterogeneous Computing Workshop, 1999.(HCW'99) Proceedings. Eighth, pages 3-14. IEEE.
  20. Vairavanathan, E., Al-Kiswany, S., Costa, L. B., Zhang, Z., Katz, D. S., Wilde, M., and Ripeanu, M. (2012). A workflow-aware storage system: An opportunity study. In Proceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012), pages 326-334. IEEE Computer Society.
  21. Wang, X., Olston, C., Sarma, A. D., and Burns, R. (2011). Coscan: cooperative scan sharing in the cloud. In Proceedings of the 2nd ACM Symposium on Cloud Computing, page 11. ACM.
  22. Wickberg, T. and Carothers, C. (2012). The ramdisk storage accelerator: a method of accelerating i/o performance on hpc systems using ramdisks. In Proceedings of the 2nd International Workshop on Runtime and Operating Systems for Supercomputers, page 5. ACM.
  23. Zhang, Y.-F., Tian, Y.-C., Fidge, C., and Kelly, W. (2016). Data-aware task scheduling for all-to-all comparison problems in heterogeneous distributed systems. Journal of Parallel and Distributed Computing, 93-94:87 - 101.
  24. Zhao, H. and Sakellariou, R. (2006). Scheduling multiple dags onto heterogeneous systems. In Proceedings 20th IEEE International Parallel & Distributed Processing Symposium, pages 14-pp. IEEE.

Paper Citation

in Harvard Style

Acevedo C., Hernández P., Espinosa A. and Mendez V. (2017). A Data-aware MultiWorkflow Scheduler for Clusters on WorkflowSim . In Proceedings of the 2nd International Conference on Complexity, Future Information Systems and Risk - Volume 1: COMPLEXIS, ISBN 978-989-758-244-8, pages 79-86. DOI: 10.5220/0006303500790086

in Bibtex Style

author={César Acevedo and Porfidio Hernández and Antonio Espinosa and Victor Mendez},
title={A Data-aware MultiWorkflow Scheduler for Clusters on WorkflowSim},
booktitle={Proceedings of the 2nd International Conference on Complexity, Future Information Systems and Risk - Volume 1: COMPLEXIS,},

in EndNote Style

JO - Proceedings of the 2nd International Conference on Complexity, Future Information Systems and Risk - Volume 1: COMPLEXIS,
TI - A Data-aware MultiWorkflow Scheduler for Clusters on WorkflowSim
SN - 978-989-758-244-8
AU - Acevedo C.
AU - Hernández P.
AU - Espinosa A.
AU - Mendez V.
PY - 2017
SP - 79
EP - 86
DO - 10.5220/0006303500790086