Can We Use Probing to Better Understand Fine-Tuning and Knowledge Distillation of the BERT NLU?

Jakub Hościłowicz, Jakub Hościłowicz, Marcin Sowański, Marcin Sowański, Piotr Czubowski, Artur Janicki

2023

Abstract

In this article, we use probing to investigate phenomena that occur during fine-tuning and knowledge distillation of a BERT-based natural language understanding (NLU) model. Our ultimate purpose was to use probing to better understand practical production problems and consequently to build better NLU models. We designed experiments to see how fine-tuning changes the linguistic capabilities of BERT, what the optimal size of the fine-tuning dataset is, and what amount of information is contained in a distilled NLU based on a tiny Transformer. The results of the experiments show that the probing paradigm in its current form is not well suited to answer such questions. Structural, Edge and Conditional probes do not take into account how easy it is to decode probed information. Consequently, we conclude that quantification of information decodability is critical for many practical applications of the probing paradigm.

Paper Citation

in Harvard Style

Hościłowicz J., Sowański M., Czubowski P. and Janicki A. (2023). Can We Use Probing to Better Understand Fine-Tuning and Knowledge Distillation of the BERT NLU?. In Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART, ISBN 978-989-758-623-1, pages 625-632. DOI: 10.5220/0011724900003393

in Bibtex Style

@conference{icaart23,
author={Jakub Hościłowicz and Marcin Sowański and Piotr Czubowski and Artur Janicki},
title={Can We Use Probing to Better Understand Fine-Tuning and Knowledge Distillation of the BERT NLU?},
booktitle={Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART,},
year={2023},
pages={625-632},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011724900003393},
isbn={978-989-758-623-1},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART,
TI - Can We Use Probing to Better Understand Fine-Tuning and Knowledge Distillation of the BERT NLU?
SN - 978-989-758-623-1
AU - Hościłowicz J.
AU - Sowański M.
AU - Czubowski P.
AU - Janicki A.
PY - 2023
SP - 625
EP - 632
DO - 10.5220/0011724900003393