Authors:
Malick Ebiele
1
;
Malika Bendechache
2
;
Eamonn Clinton
3
and
Rob Brennan
1
Affiliations:
1
ADAPT, School of Computer Science, University College Dublin, Belfield, Dublin, Ireland
;
2
School of Computer Science, University of Galway, Galway, Ireland
;
3
Tailte Éireann, Phoenix Park, Dublin, Ireland
Keyword(s):
Data Valuation, Data Value, Personalized Data Value, Dataset Retrieval, Information Retrieval, Quantitative Data Valuation.
Abstract:
In this paper, we propose a data valuation method that is used for Dataset Retrieval (DR) results re-ranking.
Dataset retrieval is a specialization of Information Retrieval (IR) where instead of retrieving relevant documents, the information retrieval system returns a list of relevant datasets. To the best of our knowledge,
data valuation has not yet been applied to dataset retrieval. By leveraging metadata and users’ preferences,
we estimate the personal value of each dataset to facilitate dataset ranking and filtering. With two real users
(stakeholders) and four simulated users (users’ preferences generated using a uniform weight distribution), we
studied the user satisfaction rate. We define users’ satisfaction rate as the probability that users find the datasets
they seek in the top k = {5,10} of the retrieval results. Previous studies of fairness in rankings (position bias)
have shown that the probability or the exposure rate of a document drops exponentially from the top
1 to the
top 10, from 100% to about 20%. Therefore, we calculated the Jaccard score@5 and Jaccard score@10 between our approach and other re-ranking options. It was found that there is a 42.24% and a 56.52% chance on
average that users will find the dataset they are seeking in the top 5 and top 10, respectively. The lowest chance
is 0% for the top 5 and 33.33% for the top 10; while the highest chance is 100% in both cases. The dataset
used in our experiments is a real-world dataset and the result of a query sent to a National mapping agency data
catalog. In the future, we are planning to extend the experiments performed in this paper to publicly available
data catalogs.
(More)