TeTIm-Eval: A Novel Curated Evaluation Data Set for Comparing Text-to-Image Models

Federico Galatolo, Mario Cimino, Edoardo Cogotti

2023

Abstract

Evaluating and comparing text-to-image models is a challenging problem. Significant advances in the field have recently been made, piquing interest of various industrial sectors. As a consequence, a gold standard in the field should cover a variety of tasks and application contexts. In this paper a novel evaluation approach is experimented, on the basis of: (i) a curated data set, made by high-quality royalty-free image-text pairs, divided into ten categories; (ii) a quantitative metric, the CLIP-score, (iii) a human evaluation task to distinguish, for a given text, the real and the generated images. The proposed method has been applied to the most recent models, i.e., DALLE2, Latent Diffusion, Stable Diffusion, GLIDE and Craiyon. Early experimental results show that the accuracy of the human judgement is fully coherent with the CLIP-score. The dataset has been made available to the public.

Paper Citation

in Harvard Style

Galatolo F., Cimino M. and Cogotti E. (2023). TeTIm-Eval: A Novel Curated Evaluation Data Set for Comparing Text-to-Image Models. In Proceedings of the 12th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM, ISBN 978-989-758-626-2, pages 590-596. DOI: 10.5220/0011885800003411

in Bibtex Style

@conference{icpram23,
author={Federico Galatolo and Mario Cimino and Edoardo Cogotti},
title={TeTIm-Eval: A Novel Curated Evaluation Data Set for Comparing Text-to-Image Models},
booktitle={Proceedings of the 12th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,},
year={2023},
pages={590-596},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011885800003411},
isbn={978-989-758-626-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 12th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,
TI - TeTIm-Eval: A Novel Curated Evaluation Data Set for Comparing Text-to-Image Models
SN - 978-989-758-626-2
AU - Galatolo F.
AU - Cimino M.
AU - Cogotti E.
PY - 2023
SP - 590
EP - 596
DO - 10.5220/0011885800003411