Pre-Trained Prompt-Tuning Based on Adversarial Regularization for Text Classification

Xiaoying Huang, Baihui Tang, Sanxing Cao

2022

Abstract

The advent of large-scale pre-trained models has greatly promoted the development of natural language processing. Many natural language processing tasks choose to fit the gap between downstream tasks and pre-training tasks through fine-tuning. However, the existing pre-trained model has a large number of parameters, and it also needs a lot of data to fine-tuning. To adapt to the training of large-scale pre-trained models, researchers proposed to replace fine-tuning with prompt-tuning to reduce the demand for supervised data. However, the performance of prompt-tuning is not stable enough. This paper proposes a method of adding adversarial regularization training based on prompt-tuning, adding disturbance in word embedding, and continuously updating the disturbance in a small range, to increase the robustness of the model and make the model obtain higher accuracy under less supervised data.

Paper Citation

in Harvard Style

Huang X., Tang B. and Cao S. (2022). Pre-Trained Prompt-Tuning Based on Adversarial Regularization for Text Classification. In Proceedings of the 3rd International Symposium on Automation, Information and Computing - Volume 1: ISAIC; ISBN 978-989-758-622-4, SciTePress, pages 286-291. DOI: 10.5220/0011922200003612

in Bibtex Style

@conference{isaic22,
author={Xiaoying Huang and Baihui Tang and Sanxing Cao},
title={Pre-Trained Prompt-Tuning Based on Adversarial Regularization for Text Classification},
booktitle={Proceedings of the 3rd International Symposium on Automation, Information and Computing - Volume 1: ISAIC},
year={2022},
pages={286-291},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011922200003612},
isbn={978-989-758-622-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 3rd International Symposium on Automation, Information and Computing - Volume 1: ISAIC
TI - Pre-Trained Prompt-Tuning Based on Adversarial Regularization for Text Classification
SN - 978-989-758-622-4
AU - Huang X.
AU - Tang B.
AU - Cao S.
PY - 2022
SP - 286
EP - 291
DO - 10.5220/0011922200003612
PB - SciTePress