How Fine Tuning Affects Contextual Embeddings: A Negative Result Explanation

Ha-Thanh Nguyen, Vu Tran, Minh-Phuong Nguyen, Le-Minh Nguyen, Ken Satoh

2023

Abstract

Recently, deep learning models trained on large amounts of data have achieved impressive results in the field of legal document processing. However, being seen as black boxes, these models lack explainability. This paper aims to shed light on the inner behavior of legal learning models by analyzing the effect of fine-tuning on legal contextual embeddings. This paper provides pieces of evidence to explain the relationship between the moving of contextual embeddings and the effectiveness of a model when fine-tuned on legal tasks. It can help further explain the effect of finetuning on language models. To this end, we use multilingual transformer models, fine-tune them on the lawfulness classification task, and record the changes in the embeddings. The experimental results reveal interesting phenomena. The method in this paper can be used to confirm whether a deep learning model truly gains the knowledge in a legal problem to make the predictions or simply memorize the training examples, or worse, predict randomly.

Paper Citation

in Harvard Style

Nguyen H., Tran V., Nguyen M., Nguyen L. and Satoh K. (2023). How Fine Tuning Affects Contextual Embeddings: A Negative Result Explanation. In Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART, ISBN 978-989-758-623-1, pages 586-591. DOI: 10.5220/0011714200003393

in Bibtex Style

@conference{icaart23,
author={Ha-Thanh Nguyen and Vu Tran and Minh-Phuong Nguyen and Le-Minh Nguyen and Ken Satoh},
title={How Fine Tuning Affects Contextual Embeddings: A Negative Result Explanation},
booktitle={Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART,},
year={2023},
pages={586-591},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011714200003393},
isbn={978-989-758-623-1},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART,
TI - How Fine Tuning Affects Contextual Embeddings: A Negative Result Explanation
SN - 978-989-758-623-1
AU - Nguyen H.
AU - Tran V.
AU - Nguyen M.
AU - Nguyen L.
AU - Satoh K.
PY - 2023
SP - 586
EP - 591
DO - 10.5220/0011714200003393