A Survey of Deep Learning: From Activations to Transformers

Johannes Schneider, Michalis Vlachos

2024

Abstract

Deep learning has made tremendous progress in the last decade. A key success factor is the large amount of architectures, layers, objectives, and optimization techniques. They include a myriad of variants related to attention, normalization, skip connections, transformers and self-supervised learning schemes – to name a few. We provide a comprehensive overview of the most important, recent works in these areas to those who already have a basic understanding of deep learning. We hope that a holistic and unified treatment of influential, recent works helps researchers to form new connections between diverse areas of deep learning. We identify and discuss multiple patterns that summarize the key strategies for many of the successful innovations over the last decade as well as works that can be seen as rising stars. We also include a discussion on recent commercially built, closed-source models such as OpenAI’s GPT-4 and Google’s PaLM 2.

Paper Citation

in Harvard Style

Schneider J. and Vlachos M. (2024). A Survey of Deep Learning: From Activations to Transformers. In Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART; ISBN 978-989-758-680-4, SciTePress, pages 419-430. DOI: 10.5220/0012404300003636

in Bibtex Style

@conference{icaart24,
author={Johannes Schneider and Michalis Vlachos},
title={A Survey of Deep Learning: From Activations to Transformers},
booktitle={Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART},
year={2024},
pages={419-430},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012404300003636},
isbn={978-989-758-680-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART
TI - A Survey of Deep Learning: From Activations to Transformers
SN - 978-989-758-680-4
AU - Schneider J.
AU - Vlachos M.
PY - 2024
SP - 419
EP - 430
DO - 10.5220/0012404300003636
PB - SciTePress