Double Trouble? Impact and Detection of Duplicates in Face Image Datasets

Torsten Schlett, Christian Rathgeb, Juan Tapia, Christoph Busch

2024

Abstract

Various face image datasets intended for facial biometrics research were created via web-scraping, i.e. the collection of images publicly available on the internet. This work presents an approach to detect both exactly and nearly identical face image duplicates, using file and image hashes. The approach is extended through the use of face image preprocessing. Additional steps based on face recognition and face image quality assessment models reduce false positives, and facilitate the deduplication of the face images both for intra- and inter-subject duplicate sets. The presented approach is applied to five datasets, namely LFW, TinyFace, Adience, CASIA-WebFace, and C-MS-Celeb (a cleaned MS-Celeb-1M variant). Duplicates are detected within every dataset, with hundreds to hundreds of thousands of duplicates for all except LFW. Face recognition and quality assessment experiments indicate a minor impact on the results through the duplicate removal. The final deduplication data is made available at https://github.com/dasec/dataset-duplicates.

Paper Citation

in Harvard Style

Schlett T., Rathgeb C., Tapia J. and Busch C. (2024). Double Trouble? Impact and Detection of Duplicates in Face Image Datasets. In Proceedings of the 13th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM; ISBN 978-989-758-684-2, SciTePress, pages 801-808. DOI: 10.5220/0012422500003654

in Bibtex Style

@conference{icpram24,
author={Torsten Schlett and Christian Rathgeb and Juan Tapia and Christoph Busch},
title={Double Trouble? Impact and Detection of Duplicates in Face Image Datasets},
booktitle={Proceedings of the 13th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM},
year={2024},
pages={801-808},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012422500003654},
isbn={978-989-758-684-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 13th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM
TI - Double Trouble? Impact and Detection of Duplicates in Face Image Datasets
SN - 978-989-758-684-2
AU - Schlett T.
AU - Rathgeb C.
AU - Tapia J.
AU - Busch C.
PY - 2024
SP - 801
EP - 808
DO - 10.5220/0012422500003654
PB - SciTePress