A Review of Contextualized Word Embeddings and Pre-Trained Language Models, with a Focus on GPT and BERT

Maya Thapa, Puneet Kapoor, Sakshi Kaushal, Ishani Sharma

2024

Abstract

Word meanings are attempted to be encapsulated by word embeddings, which are n-dimensional distributed representations of text. Multiple computational layers are used by deep learning models to obtain hierarchical data representations. Word embedding as a deep learning approach has attracted a lot of attention and is used in many Natural Language Processing (NLP) applications such as text classification, sentiment analysis, named entity recognition, and topic modeling. Thus, adopting suitable deep-learning models and word embeddings is essential for getting greater results. Highlighting the substantial impact of word embeddings on NLP over the past decade, the study transitions into a detailed examination of contextualized embeddings. It addresses the limitations of conventional static embeddings and introduces the revolutionary nature of contextualized embeddings, particularly in capturing both syntactic and semantic subtleties. The paper serves as a comprehensive review of pre-trained language models, emphasizing advancements in NLP applications. Models like GPT-3 and BERT take center stage in the analysis, showcasing their strengths and limitations.

Paper Citation

in Harvard Style

Thapa M., Kapoor P., Kaushal S. and Sharma I. (2024). A Review of Contextualized Word Embeddings and Pre-Trained Language Models, with a Focus on GPT and BERT. In Proceedings of the 1st International Conference on Cognitive & Cloud Computing - Volume 1: IC3Com; ISBN 978-989-758-739-9, SciTePress, pages 205-214. DOI: 10.5220/0013305900004646

in Bibtex Style

@conference{ic3com24,
author={Maya Thapa and Puneet Kapoor and Sakshi Kaushal and Ishani Sharma},
title={A Review of Contextualized Word Embeddings and Pre-Trained Language Models, with a Focus on GPT and BERT},
booktitle={Proceedings of the 1st International Conference on Cognitive & Cloud Computing - Volume 1: IC3Com},
year={2024},
pages={205-214},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013305900004646},
isbn={978-989-758-739-9},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 1st International Conference on Cognitive & Cloud Computing - Volume 1: IC3Com
TI - A Review of Contextualized Word Embeddings and Pre-Trained Language Models, with a Focus on GPT and BERT
SN - 978-989-758-739-9
AU - Thapa M.
AU - Kapoor P.
AU - Kaushal S.
AU - Sharma I.
PY - 2024
SP - 205
EP - 214
DO - 10.5220/0013305900004646
PB - SciTePress