Semantic Properties of Cosine Based Bias Scores for Word Embeddings

Sarah Schröder, Alexander Schulz, Fabian Hinder, Barbara Hammer

2024

Abstract

Plenty of works have brought social biases in language models to attention and proposed methods to detect such biases. As a result, the literature contains a great deal of different bias tests and scores, each introduced with the premise to uncover yet more biases that other scores fail to detect. What severely lacks in the literature, however, are comparative studies that analyse such bias scores and help researchers to understand the benefits or limitations of the existing methods. In this work, we aim to close this gap for cosine based bias scores. By building on a geometric definition of bias, we propose requirements for bias scores to be considered meaningful for quantifying biases. Furthermore, we formally analyze cosine based scores from the literature with regard to these requirements. We underline these findings with experiments to show that the bias scores’ limitations have an impact in the application case.

Paper Citation

in Harvard Style

Schröder S., Schulz A., Hinder F. and Hammer B. (2024). Semantic Properties of Cosine Based Bias Scores for Word Embeddings. In Proceedings of the 13th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM; ISBN 978-989-758-684-2, SciTePress, pages 160-168. DOI: 10.5220/0012577200003654

in Bibtex Style

@conference{icpram24,
author={Sarah Schröder and Alexander Schulz and Fabian Hinder and Barbara Hammer},
title={Semantic Properties of Cosine Based Bias Scores for Word Embeddings},
booktitle={Proceedings of the 13th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM},
year={2024},
pages={160-168},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012577200003654},
isbn={978-989-758-684-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 13th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM
TI - Semantic Properties of Cosine Based Bias Scores for Word Embeddings
SN - 978-989-758-684-2
AU - Schröder S.
AU - Schulz A.
AU - Hinder F.
AU - Hammer B.
PY - 2024
SP - 160
EP - 168
DO - 10.5220/0012577200003654
PB - SciTePress