A Data-Aware MultiWorkflow Cluster Scheduler

César Acevedo, Porfidio Hernandez, Antonio Espinosa, Víctor Méndez

2016

Abstract

Previous scheduling research work is based on the analysis of the computational time of application workflows. Current use of clusters deals with the execution of multiworkflows that may share applications and input files. In order to reduce the makespan of such multiworkflows adequate data allocation policies should be applied to reduce input data latency. We propose a scheduling strategy for multiworkflows that considers the data location of shared input files in different locations of the storage system of the cluster. For that, we first merge all workflows in a study and evaluate the global design pattern obtained. Then, we apply a classic list scheduling heuristic considering the location of the input files in the storage system to reduce the communication overhead of the applications. We have evaluated our proposal with an initial set of experimental environments showing promising results of up to 20% makespan improvement.

References

  1. Afrati, F., Papadimitriou, C. H., and Papageorgiou, G. (1988). Scheduling dags to minimize time and communication. In VLSI Algorithms and Architectures, pages 134-138. Springer.
  2. Ananthanarayanan, G., Ghodsi, A., Wang, A., Borthakur, D., Kandula, S., Shenker, S., and Stoica, I. (2012). Pacman: coordinated memory caching for parallel jobs. In Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation, pages 20-20. USENIX Association.
  3. Barbosa, J. and Monteiro, A. P. (2008). A list scheduling algorithm for scheduling multi-user jobs on clusters. In High Performance Computing for Computational Science-VECPAR 2008, pages 123-136. Springer.
  4. Bittencourt, L. F. and Madeira, E. R. (2010). Towards the scheduling of multiple workflows on computational grids. Journal of grid computing, 8(3):419-441.
  5. Bolze, R., Desprez, F., and Insard, B. Evaluation of online multi-workflow heuristics based on list scheduling methods. Technical report, Gwendia ANR-06- MDCA-009.
  6. Cerezo, N., Montagnat, J., and Blay-Fornarino, M. (2013). Computer-assisted scientific workflow design. Journal of grid computing, 11(3):585-612.
  7. Costa, L. B., Yang, H., Vairavanathan, E., Barros, A., Maheshwari, K., Fedak, G., Katz, D., Wilde, M., Ripeanu, M., and Al-Kiswany, S. (2015). The case for workflow-aware storage: An opportunity study. Journal of Grid Computing, 13(1):95-113.
  8. Goecks, J., Nekrutenko, A., Taylor, J., et al. (2010). Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences. Genome Biol, 11(8):R86.
  9. Gu, Y. and Wu, Q. (2010). Optimizing distributed computing workflows in heterogeneous network environments. In Distributed Computing and Networking, pages 142-154. Springer.
  10. Hönig, U. and Schiffmann, W. (2006). A meta-algorithm for scheduling multiple dags in homogeneous system environments. In Proceedings of the eighteenth IASTED International Conference on Parallel and Distributed Computing and Systems (PDCS'06).
  11. Ilavarasan, E. and Thambidurai, P. (2007). Low complexity performance effective task scheduling algorithm for heterogeneous computing environments. Journal of Computer sciences, 3(2):94-103.
  12. Kwok, Y.-K. and Ahmad, I. (1999). Static scheduling algorithms for allocating directed task graphs to multiprocessors. ACM Computing Surveys (CSUR), 31(4):406-471.
  13. Mandal, A., Kennedy, K., Koelbel, C., Marin, G., MellorCrummey, J., Liu, B., and Johnsson, L. (2005). Scheduling strategies for mapping application workflows onto the grid. In High Performance Distributed Computing, 2005. HPDC-14. Proceedings. 14th IEEE International Symposium on, pages 125-134. IEEE.
  14. Takpé, T. and Suter, F. (2009). Concurrent scheduling of parallel task graphs on multi-clusters using constrained resource allocations. In Parallel & Distributed Processing, 2009. IPDPS 2009. IEEE International Symposium on, pages 1-8. IEEE.
  15. Ousterhout, J., Agrawal, P., Erickson, D., Kozyrakis, C., Leverich, J., Mazières, D., Mitra, S., Narayanan, A., Ongaro, D., Parulkar, G., et al. (2011). The case for ramcloud. Communications of the ACM, 54(7):121- 130.
  16. Park, G.-L., Shirazi, B., and Marquis, J. (1997). Dfrn: A new approach for duplication based scheduling for distributed memory multiprocessor systems. In Parallel Processing Symposium, 1997. Proceedings., 11th International, pages 157-166. IEEE.
  17. Pinedo, M. L. (2012). Scheduling: theory, algorithms, and systems. Springer Science & Business Media.
  18. Rahman, M., Venugopal, S., and Buyya, R. (2007). A dynamic critical path algorithm for scheduling scientific workflow applications on global grids. In e-Science and Grid Computing, IEEE International Conference on, pages 35-42. IEEE.
  19. Stavrinides, G. L. and Karatza, H. D. (2011). Scheduling multiple task graphs in heterogeneous distributed realtime systems by exploiting schedule holes with bin packing techniques. Simulation Modelling Practice and Theory, 19(1):540-552.
  20. Topcuoglu, H., Hariri, S., and Wu, M.-Y. (1999). Task scheduling algorithms for heterogeneous processors. In Heterogeneous Computing Workshop, 1999.(HCW'99) Proceedings. Eighth, pages 3-14. IEEE.
  21. Wang, X., Olston, C., Sarma, A. D., and Burns, R. (2011). Coscan: cooperative scan sharing in the cloud. In Proceedings of the 2nd ACM Symposium on Cloud Computing, page 11. ACM.
  22. Wickberg, T. and Carothers, C. (2012). The ramdisk storage accelerator: a method of accelerating i/o performance on hpc systems using ramdisks. In Proceedings of the 2nd International Workshop on Runtime and Operating Systems for Supercomputers, page 5. ACM.
  23. Yang, T. and Gerasoulis, A. (1994). Dsc: Scheduling parallel tasks on an unbounded number of processors. Parallel and Distributed Systems, IEEE Transactions on, 5(9):951-967.
  24. Yu, J. and Buyya, R. (2005). A taxonomy of scientific workflow systems for grid computing. ACM Sigmod Record, 34(3):44-49.
  25. Yu, Z. and Shi, W. (2008). A planner-guided scheduling strategy for multiple workflow applications. In Parallel Processing-Workshops, 2008. ICPP-W'08. International Conference on, pages 1-8. IEEE.
  26. Zhao, H. and Sakellariou, R. (2006). Scheduling multiple dags onto heterogeneous systems. In Parallel and Distributed Processing Symposium, 2006. IPDPS 2006. 20th International, pages 14-pp. IEEE.
  27. Zhao, T. and Hu, J. (2010). Performance evaluation of parallel file system based on lustre and grey theory. In Grid and Cooperative Computing (GCC), 2010 9th International Conference on, pages 118-123. IEEE.
Download


Paper Citation


in Harvard Style

Acevedo C., Hernandez P., Espinosa A. and Méndez V. (2016). A Data-Aware MultiWorkflow Cluster Scheduler . In Proceedings of the 1st International Conference on Complex Information Systems - Volume 1: COMPLEXIS, ISBN 978-989-758-181-6, pages 95-102. DOI: 10.5220/0005932000950102


in Bibtex Style

@conference{complexis16,
author={César Acevedo and Porfidio Hernandez and Antonio Espinosa and Víctor Méndez},
title={A Data-Aware MultiWorkflow Cluster Scheduler},
booktitle={Proceedings of the 1st International Conference on Complex Information Systems - Volume 1: COMPLEXIS,},
year={2016},
pages={95-102},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005932000950102},
isbn={978-989-758-181-6},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 1st International Conference on Complex Information Systems - Volume 1: COMPLEXIS,
TI - A Data-Aware MultiWorkflow Cluster Scheduler
SN - 978-989-758-181-6
AU - Acevedo C.
AU - Hernandez P.
AU - Espinosa A.
AU - Méndez V.
PY - 2016
SP - 95
EP - 102
DO - 10.5220/0005932000950102