Gradient Descent Analysis: On Visualizing the Training of Deep Neural Networks

Martin Becker, Jens Lippel, Thomas Zielke

2019

Abstract

We present an approach to visualizing gradient descent methods and discuss its application in the context of deep neural network (DNN) training. The result is a novel type of training error curve (a) that allows for an exploration of each individual gradient descent iteration at line search level; (b) that reflects how a DNN’s training error varies along each of the descent directions considered; (c) that is consistent with the traditional training error versus training iteration view commonly used to monitor a DNN’s training progress. We show how these three levels of detail can be easily realized as the three stages of Shneiderman’s Visual Information Seeking Mantra. This suggests the design and development of a new interactive visualization tool for the exploration of DNN learning processes. We present an example that showcases a conceivable interactive workflow when working with such a tool. Moreover, we give a first impression of a possible DNN hyperparameter analysis.

Paper Citation

in Harvard Style

Becker M., Lippel J. and Zielke T. (2019). Gradient Descent Analysis: On Visualizing the Training of Deep Neural Networks. In Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019) - Volume 3: IVAPP; ISBN 978-989-758-354-4, SciTePress, pages 338-345. DOI: 10.5220/0007583403380345

in Bibtex Style

@conference{ivapp19,
author={Martin Becker and Jens Lippel and Thomas Zielke},
title={Gradient Descent Analysis: On Visualizing the Training of Deep Neural Networks},
booktitle={Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019) - Volume 3: IVAPP},
year={2019},
pages={338-345},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0007583403380345},
isbn={978-989-758-354-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019) - Volume 3: IVAPP
TI - Gradient Descent Analysis: On Visualizing the Training of Deep Neural Networks
SN - 978-989-758-354-4
AU - Becker M.
AU - Lippel J.
AU - Zielke T.
PY - 2019
SP - 338
EP - 345
DO - 10.5220/0007583403380345
PB - SciTePress