Clustering for Explainability: Extracting and Visualising Concepts from Activation

Alexandre Lambert, Alexandre Lambert, Alexandre Lambert, Aakash Soni, Assia Soukane, Amar Ramdane Cherif, Arnaud Rabat

2024

Abstract

Despite significant advances in computer vision with deep learning models (e.g. classification, detection, and segmentation), these models remain complex, making it challenging to assess their reliability, interpretability, and consistency under diverse. There is growing interest in methods for extracting human-understandable concepts from these models, but significant challenges persist. These challenges include difficulties in extracting concepts relevant to both model parameters and inference while ensuring the concepts are meaningful to individuals with varying expertise levels without requiring a panel of evaluators to validate the extracted concepts. To tackle these challenges, we propose concept extraction by clustering activations. Activations represent a model’s internal state based on its training, and can be grouped to represent learned concepts. We propose two clustering methods for concept extraction, a metric for evaluating their importance, and a concept visualization technique for concept interpretation. This approach can help identify biases in models and datasets.

Paper Citation

in Harvard Style

Lambert A., Soni A., Soukane A., Ramdane Cherif A. and Rabat A. (2024). Clustering for Explainability: Extracting and Visualising Concepts from Activation. In Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 2: KEOD; ISBN 978-989-758-716-0, SciTePress, pages 151-158. DOI: 10.5220/0012927900003838

in Bibtex Style

@conference{keod24,
author={Alexandre Lambert and Aakash Soni and Assia Soukane and Amar Ramdane Cherif and Arnaud Rabat},
title={Clustering for Explainability: Extracting and Visualising Concepts from Activation},
booktitle={Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 2: KEOD},
year={2024},
pages={151-158},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012927900003838},
isbn={978-989-758-716-0},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 2: KEOD
TI - Clustering for Explainability: Extracting and Visualising Concepts from Activation
SN - 978-989-758-716-0
AU - Lambert A.
AU - Soni A.
AU - Soukane A.
AU - Ramdane Cherif A.
AU - Rabat A.
PY - 2024
SP - 151
EP - 158
DO - 10.5220/0012927900003838
PB - SciTePress