So Can We Use Intrinsic Bias Measures or Not?

Sarah Schröder, Alexander Schulz, Philip Kenneweg, Barbara Hammer

2023

Abstract

While text embeddings have become the state-of-the-art in many natural language processing applications, the presence of bias that such models often learn from training data can become a serious problem. As a reaction, a large variety of measures for detecting bias has been proposed. However, an extensive comparison between them does not exists so far. We aim to close this gap for the class of intrinsic bias measures in the context of pretrained language models and propose an experimental setup which allows a fair comparison by using a large set of templates for each bias measure. Our setup is based on the idea of simulating pretraining on a set of differently biased corpora, thereby obtaining a ground truth for the present bias. This allows us to evaluate in how far bias is detected by different measures and also enables to judge the robustness of bias scores.

Paper Citation

in Harvard Style

Schröder S., Schulz A., Kenneweg P. and Hammer B. (2023). So Can We Use Intrinsic Bias Measures or Not?. In Proceedings of the 12th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM, ISBN 978-989-758-626-2, pages 403-410. DOI: 10.5220/0011693700003411

in Bibtex Style

@conference{icpram23,
author={Sarah Schröder and Alexander Schulz and Philip Kenneweg and Barbara Hammer},
title={So Can We Use Intrinsic Bias Measures or Not?},
booktitle={Proceedings of the 12th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,},
year={2023},
pages={403-410},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011693700003411},
isbn={978-989-758-626-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 12th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,
TI - So Can We Use Intrinsic Bias Measures or Not?
SN - 978-989-758-626-2
AU - Schröder S.
AU - Schulz A.
AU - Kenneweg P.
AU - Hammer B.
PY - 2023
SP - 403
EP - 410
DO - 10.5220/0011693700003411