![](bg8.png)
5 CONCLUSION
We have examined and compared several DH and ML
delay predictors on data from a real multi-skill call
center. We found that the ML predictors are much
more accurate than the DH predictors. Within the ML
predictors, ANN was more accurate than RS and LR,
but the latter can be trained much faster than ANN,
and could be more accurate when the amount of avail-
able data is smaller. We saw the negative impact of
leaving out relevant input variables on the accuracy
of the ML predictors, and illustrated how well Boruta
can identify the most relevant input variables. In on-
going work, we want to develop effective methods to
predict and announce not only a point estimate of the
waiting time, but an estimate of the entire conditional
distribution of the delay.
ACKNOWLEDGEMENTS
This work has been supported by grants from
NSERC-Canada and Hydro-Qu
´
ebec, and a Canada
Research Chair to P. L’Ecuyer. We thank Ger Koole,
from VU Amsterdam, who provided the data.
REFERENCES
Ang, E., Kwasnick, S., Bayati, M., Plambeck, E., and Ara-
tow, M. (2016). Accurate emergency department wait
time prediction. Manufacturing & Service Operations
Management, 18(1):141–156.
Armony, M., Shimkin, N., and Whitt, W. (2009). The im-
pact of delay announcements in many-server queues
with abandonments. Operations Research, 57:66–81.
Bengio, Y. (2012). Practical recommendations for gradient-
based training of deep architectures. Neural Net-
works: Tricks of the Trade, 7700:437–478.
Bengio, Y., Courville, A. C., and Vincent, P. (2012). Unsu-
pervised feature learning and deep learning: A review
and new perspectives. CoRR, abs/1206.5538:1–30.
Bergstra, J. and Bengio, Y. (2012). Random search for
hyper-parameter optimization. Journal of Machine
Learning Research, 13:281–305.
Breiman, L. (2001). Random forests. Machine learning,
45(1):5–32.
de Boor, C. (1978). A Practical Guide to Splines. Num-
ber 27 in Applied Mathematical Sciences Series.
Springer-Verlag, New York.
Degenhardt, F., Seifert, S., and Szymczak, S. (2017). Evalu-
ation of variable selection methods for random forests
and omics data sets. Briefings in Bioinformatics,
20(2):492–503.
Dong, J., Yom Tov, E., and Yom Tov, G. (2018). The im-
pact of delay announcements on hospital network co-
ordination and waiting times. Management Science,
65(5):1949–2443.
Friedman, J., Hastie, T., and Tibshirani, R. (2010). Regular-
ization paths for generalized linear models via coordi-
nate descent. Journal of Statistical Software, 33(1):1–
22.
Friedman, J., Hastie, T., Tibshirani, R., Narasimhan,
B., Simon, N., and Qian, J. (2019). R Pack-
age glmnet: Lasso and Elastic-Net Regularized
Generalized Linear Models. https://CRAN.R-
project.org/package=glmnet.
Glorot, X., Bordes, A., and Bengio, Y. (2011). Deep sparse
rectifier neural networks. In Gordon, G., Dunson,
D., and Miroslav, editors, Proceedings of the Four-
teenth International Conference on Artificial Intelli-
gence and Statistics, volume 15 of Proceedings of Ma-
chine Learning Research, pages 315–323, Fort Laud-
erdale, FL, USA. PMLR.
Goodfellow, I., Bengio, Y., and Courville, A.
(2016). Deep Learning. MIT Press.
http://www.deeplearningbook.org.
Goodfellow, I., Warde-Farley, D., Lamblin, P., Dumoulin,
V., Mirza, M., Pascanu, R., Bergstra, J., Bastien, F.,
and Bengio, Y. (2013). Pylearn2: A machine learning
research library.
Gulcehre, C. and Bengio, Y. (2016). Knowledge mat-
ters: Importance of prior information for optimization.
Journal of Machine Learning Research, 17:1–32.
Ibrahim, R., L’Ecuyer, P., Shen, H., and Thiongane, M.
(2016). Inter-dependent, heterogeneous, and time-
varying service-time distributions in call centers. Eu-
ropean Journal of Operational Research, 250:480–
492.
Ibrahim, R. and Whitt, W. (2009a). Real-time delay es-
timation based on delay history. Manufacturing and
Services Operations Management, 11:397–415.
Ibrahim, R. and Whitt, W. (2009b). Real-time delay estima-
tion in overloaded multiserver queues with abandon-
ments. Management Science, 55(10):1729–1742.
Ibrahim, R. and Whitt, W. (2010). Delay predictors for cus-
tomer service systems with time-varying parameters.
In Proceedings of the 2010 Winter Simulation Confer-
ence, pages 2375–2386. IEEE Press.
Ibrahim, R. and Whitt, W. (2011). Real-time delay esti-
mation based on delay history in many-server service
systems with time-varying arrivals. Production and
Operations Management, 20(5):654–667.
James, G., Witten, D., Hastie, T., and Tibshirani, R. (2013).
An Introduction to Statistical Learning, with Applica-
tions in R. Springer-Verlag, New York.
Kursa, M. B. and Rudnicki, W. R. (2010). Feature selec-
tion with the boruta package. Journal of Statistical
Software, 36:1–13.
LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep learn-
ing. Nature, 521:436–444.
L’Ecuyer, P. (2016). SSJ: Stochastic simulation in Java.
http://simul.iro.umontreal.ca/ssj/.
L’Ecuyer, P., Meliani, L., and Vaucher, J. (2002). SSJ:
A framework for stochastic simulation in Java. In
Y
¨
ucesan, E., Chen, C.-H., Snowdon, J. L., and
Charnes, J. M., editors, Proceedings of the 2002
Delay Predictors in Multi-skill Call Centers: An Empirical Comparison with Real Data
107