Decoupling the Backward Pass Using Abstracted Gradients

Kyle Rogers, Hao Yu, Seong-Eun Cho, Nancy Fulda, Jordan Yorgason, Tyler Jarvis

2024

Abstract

In this work we introduce a novel method for decoupling the backward pass of backpropagation using mathematical and biological abstractions to approximate the error gradient. Inspired by recent findings in neuroscience, our algorithm allows gradient information to skip groups of layers during the backward pass, such that weight updates at multiple depth levels can be calculated independently. We explore both gradient abstractions using the identity matrix as well as an abstraction that we derive mathematically for network regions that consist of piecewise-linear layers (including layers with ReLU and leaky ReLU activations). We validate the derived abstraction calculation method on a fully connected network with ReLU activations. We then test both the derived and identity methods on the transformer architecture and show the capabilities of each method on larger model architectures. We demonstrate empirically that a network trained using an appropriately chosen abstraction matrix can match the loss and test accuracy of an unmodified network, and we provide a roadmap for the application of this method toward depth-wise parallelized models and discuss the potential of network modularization by this method.

Paper Citation

in Harvard Style

Rogers K., Yu H., Cho S., Fulda N., Yorgason J. and Jarvis T. (2024). Decoupling the Backward Pass Using Abstracted Gradients. In Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART; ISBN 978-989-758-680-4, SciTePress, pages 507-518. DOI: 10.5220/0012362800003636

in Bibtex Style

@conference{icaart24,
author={Kyle Rogers and Hao Yu and Seong-Eun Cho and Nancy Fulda and Jordan Yorgason and Tyler Jarvis},
title={Decoupling the Backward Pass Using Abstracted Gradients},
booktitle={Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART},
year={2024},
pages={507-518},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012362800003636},
isbn={978-989-758-680-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART
TI - Decoupling the Backward Pass Using Abstracted Gradients
SN - 978-989-758-680-4
AU - Rogers K.
AU - Yu H.
AU - Cho S.
AU - Fulda N.
AU - Yorgason J.
AU - Jarvis T.
PY - 2024
SP - 507
EP - 518
DO - 10.5220/0012362800003636
PB - SciTePress