Model Transparency: Why Do We Care?

Ioannis Papantonis and Vaishak Belle

University of Edinburgh, U.K.

Keywords:

Explainable AI.

Abstract:

Artiﬁcial intelligence (AI) and especially machine learning (ML) has been increasingly incorporated into a

wide range of critical applications, such as healthcare, justice, credit risk assessment, and loan approval.

In this paper, we survey the motivations for caring about model transparency, especially as AI systems are

becoming increasingly complex leviathans with many moving parts. We then brieﬂy outline the challenges in

providing computational solutions to transparency.

1 INTRODUCTION

Artiﬁcial intelligence (AI) and especially machine

learning (ML) has been increasingly incorporated into

a wide range of critical applications, such as health-

care Kononenko (2001); Loftus et al. (2019), jus-

tice Chouldechova (2017); Christin (2017); Kleinberg

et al. (2017), credit risk assessment Chen et al. (2016);

Finlay (2011), and loan approval Finlay (2011); Wu

et al. (2019a). At the same time, automated sys-

tems have an effect on casual everyday decisions,

by recommending news articles Alvarado and Waern

(2018), movies Bennett et al. (2007), and music

Mehrotra et al. (2018). In the core of this ML pre-

dominance lies the expectation that models can be

more accurate than humans Poursabzi-Sangdeh et al.

(2021a), something that has already been demon-

strated in various cases Culverhouse et al. (2003);

Goh et al. (2020); Hilder et al. (2009); Grove et al.

(2000). Having said that, employing algorithms

even as recommendation systems for cultural prod-

ucts (like movies) makes them part of the human cul-

ture, since they not only handle cultural products, but

they also inﬂuence peoples’ decisions and perceptions

Gillespie (2016). This also means that they should not

be viewed as mere tools Bozdag (2013), but rather

as entities that hold their own values Alvarado and

Waern (2018).

As such, it is paramount to make sure that their

values align with those of human’s, thus enabling

ML’s responsible integration to society Russell et al.

(2015); Gabriel (2020); Christian (2020). This need

is further magniﬁed by several recent instances of au-

tomated systems perpetuating undesired historical hu-

man biases, such as Amazon’s recruitment algorithm

exhibiting misogynistic behaviour Meyer (2018), or

commercial systems utilized by the US criminal jus-

tice system being extremely biased against black de-

fendants Angwin et al. (2016); Dressel and Farid

(2018). Apart from that, ML failures can arise, for

example, due to misuse, as in the case of an individ-

ual who spent an extra year in prison due to a typo-

graphical error in one of the inputs that was given to

the ML system Wexler (2017). Of course, poor model

design is another major source of catastrophic failures

with far-reaching implications, such as putting people

in danger due to inaccurate air quality assessment Mc-

Gough (2018), or by providing life-threatening cancer

treatment recommendations Strickland (2019); Ross

and Swetlitz (2018). These and other similar pit-

falls, along with the consequences and confusion that

come with them Galanos (2019); Aleksander (2017),

have led to some extreme arguments about ML po-

tentially eroding the social fabric and even posing a

threat to society’s democratic foundation Bozdag and

Van Den Hoven (2015).

In light of such concerns, it is becoming increas-

ingly clear that proactive actions need to be taken on

a large scale in order to avoid bleak future situations.

The urgency of this matter is reﬂected, for example,

in the Declaration of Cooperation on Artiﬁcial Intelli-

gence signed by the members of the European Union

(EU).

This development was followed by the forma-

tion of an expert group on AI,

with the goal of an-

choring the development of AI that is both success-

ful and ethically sound. Transparency was a central

notion highlighted in this call, and especially its re-

lationship with trustworthiness, which is one of the

https://digital-strategy.ec.europa.eu/en/news/eu-

member-states-sign-cooperate-artiﬁcial-intelligence

https://ec.europa.eu/commission/presscorner/detail/

en/IP\ 18\ 1381

650

Papantonis, I. and Belle, V.

Model Transparency: Why Do We Care?.

DOI: 10.5220/0011726300003393

In Proceedings of the 15th International Conference on Agents and Artiﬁcial Intelligence (ICAART 2023) - Volume 3, pages 650-657

ISBN: 978-989-758-623-1; ISSN: 2184-433X

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

end goals of this initiative. As the Commission Vice-

President for the Digital Single Market, Andrus An-

sip, noted: “As always with the use of technologies,

trust is a must”. Subsequent research outputs of the

resulting group have further emphasized the impor-

tance of transparency,

listing it as one of the seven

key requirements in order to achieve trustworthy AI.

At the same time, several additional large scale

initiatives regarding the responsible integration of AI

have been taken, such as:

• The Asilomar AI Principles, with the support of

the Future of Life Institute.

• The Montreal Declaration for Responsible AI,

with the support of the University of Montreal.

• The General Principles, with contributions from

250 thought leaders throughout the world.

• The Tenets of the Partnership on AI, with con-

tributions from stakeholders coming from diverse

ﬁelds that make use of AI (academia, industry,

etc).

It is worth noting that a comparative meta-analysis

found the above sets of principles to greatly overlap,

ensuring that the scientiﬁc/regulatory/investing com-

munities have reached a satisfying level of consensus

Floridi et al. (2018). Again, transparency was recog-

nized as a core component that should drive the in-

tegration of automated systems, present in all initia-

tives, although the terminology was somehow incon-

sistent

. Finally, an additional survey that analyzed

84 different ethical guidelines for AI found that trans-

parency was the most common principle among them,

called for in 73 of them Jobin et al. (2019).

2 DIMENSIONS OF

TRANSPARENCY

Having established the need for transparency, as the

ability to understand AI/ML, a natural next step is to

https://ec.europa.eu/ info /publications/white- paper-

artiﬁcial- intelligenceeuropean- approach- excellence- and-

trust\ en

https : / / ec.europa.eu / digital - singlemarket /en / news /

ethics-guidelines-trustworthy-ai

https://futureoﬁfe.org/ai-principles

https://www.montrealdeclaration- responsibleai.com/

the-declaration

http : / / standards.ieee.org / develop / indconn / ec /

autonomous\ systems.html

https://www.partnershiponai.org/$\sim$tenets/

The authors in Floridi et al. (2018) introduce the term

Explicability, to overcome this inconsistency, however they

deﬁne it in terms of transparency.

deﬁne all the notions it should encompass. At this

point, it should be noted that transparency is by no

means a new concept by itself, rather it has a long his-

tory in an array of disciplines Margetts (2011); Hood

and Heald (2006). Despite that, AI/ML adds a unique

dimension to it, since other disciplines rarely face

the issue of employing tools with black-box design

where the decision process itself is elusive Floridi

et al. (2018). This challenge has been a contributing

factor to the surge in related publications, which in-

crease by about 100% every other year Larsson et al.

(2019).

While this is an impressive rate, stakeholders out-

side the AI community have indicated that future de-

velopments should be predicated on further mutual

engagement between the two parties (non-AI and AI)

Bhatt et al. (2020b,a). This is a reasonable con-

cern, since although the AI community has produced

a vast literature during the last decade, the major-

ity of the scientiﬁc output addresses only the techni-

cal side of transparency, through the lens of model

transparency and model explainability Arrieta et al.

(2020). The former paradigm advocates in favour of

utilizing “transparent” (or white-box) models, mean-

ing that their design allows for readily inspecting their

inner workings Linardatos et al. (2020), such as rule-

based classiﬁers or regression analysis. On the other

hand, the latter approach, which is also known as ex-

plainability in AI (XAI), develops post-hoc techniques

that can provide explanations and information about

the decision process of black-box models, i.e. models

with an overly complex design that does not allow for

gaining any meaningful insights, such as neural net-

works or random forests Guidotti et al. (2018). These

are both essential research directions, however, com-

paring them to the notion of transparency discussed so

far, they seems rather narrow-scoped, focusing only

on the technical side of achieving a transparent AI in-

tegration.

This observation has motivated a series of works

that advocate in favour of expanding the scope of

transparency, as used in the AI/ML community, to

encompass a wider range of goals Mittelstadt et al.

(2019), in line with the calls mentioned earlier in this

section. More speciﬁcally, some important directions

that need to be incorporated into the AI community’s

agenda are related to:

• Providing guidelines regarding the appropriate

way to utilize and explain AI systems, referred to

as competence.

• Building an environment of trust,

which “can

https : / / digital - strategy.ec.europa.eu / en / news / eu -

member-states-sign-cooperate-artiﬁcial-intelligence

Model Transparency: Why Do We Care?

651

only be achieved by ensuring an appropriate in-

volvement by human beings in relation to high-

risk AI applications”.

This is not an exhaustive list, as, for example, ad-

ditional dimensions that incorporate legal aspects can

potentially be fostered under this expanded notion of

transparency Larsson (2019). However, both of these

aspects can have an immediate positive impact, con-

sidering that AI/ML systems are already deployed, so

their correct and responsible use should be top prior-

ity.

3 DIRECTIONS

Given the dimensions identiﬁed above, we brieﬂy

discuss some research directions in both the techni-

cal camp (transparent models) as well as the socio-

technical camp (stakeholder engagement), and outline

challenges in both.

3.1 Transparent Models

Clearly, the use of transparent models – i.e., models

that are transparent by design, have interpretable fea-

tures, are human-readable, etc – is motivated by their

ability to allow users to understand their inner work-

ings. Let us examine a candidate, for concreteness,

but also mention challenges that arise with it.

• Probabilistic Models. For the sake of concrete-

ness, consider Bayesian networks, and other types

of graphical models Pearl (1988). Bayesian net-

works (BNs) are a class of probabilistic mod-

els that represent relationships between variables

by using direct (usually acyclic) graphs Darwiche

(2009). This has the very appealing advantage of

clearly expressing dependencies in the data, by

only drawing arrows between variables. Further-

more, once the BN is speciﬁed, graphical tests can

accurately recover all conditional independencies,

without the need to perform any algebraic manip-

ulations Geiger et al. (1990). Due to these prop-

erties BNs are arguably one of the most transpar-

ent model classes, since their internal representa-

tion (and its implications) can be easily inspected,

by construction. It is this strength that has turned

BNs into the backbone of causal inference, too;

causal relationships are represented through a BN,

while graphical criteria identify which causal ef-

fects can be estimated using observational data

https://ec.europa.eu/ info /publications/white- paper-

artiﬁcial- intelligenceeuropean- approach- excellence- and-

trust\ en

Pearl (2009). Naturally, BNs have found nu-

merous applications in many integral applications

Kalet et al. (2015); Castelletti and Soncini-Sessa

(2007); Shenton et al. (2014); Uusitalo (2007);

Stewart-Koster et al. (2010); Friis-Hansen (2000).

• Expert Knowledge. In addition to the above,

BNs allow for incorporating various forms of a

priori constraints, such as temporal ones Dechter

et al. (1991). Of course, probabilistic indepen-

dence constraints can be encoded as well dur-

ing model design, by directly adjusting the topol-

ogy of the directed graph. The combination of

all these properties as well as the ability to infer

causal relationships, instead of correlations, offers

a powerful alternative to black-box models, espe-

cially when considering high-stakes applications

Rudin (2019); Rudin et al. (2022).

• Computational Hurdles. While BNs come with

signiﬁcant advantages, a downside is that in-

ference using them is intractable, in the sense

that computing marginal probabilities is NP-hard

Cooper (1990). On top of that, specialized

routines are required to perform the inferential

step. This is the main motivation behind the re-

cent emergence of so-called tractable probabilis-

tic models (TPMs) Poon and Domingos (2011),

as an alternative approach that generalizes tradi-

tional BNs. TPMs directly encode the joint distri-

bution of a set of variables, in a way that allows

for a simple mechanism for performing inference.

Furthermore, they can potentially lead to expo-

nential savings in both inference time and stor-

ing space Darwiche (2003). Consequently, TPMs

have gathered signiﬁcant attention in many appli-

cations Bekker et al. (2014).

A downside, however, is that TPMs are repre-

sented as computational graphs that do not allow

for directly inspecting the relationships between

the variables. Furthermore, incorporating proba-

bilistic constraints is not immediate, as in the BN

case. In fact, it is unclear whether it is at all feasi-

ble. These challenges effectively turn TPMs into

black-box models, despite them being closely re-

lated to one of the most transparent model classes.

The same kind of computation vs transparency di-

chotomy can be observed in so-called variational

approaches Srivastava and Sutton (2017). These

models avoid the explicit computation of proba-

bilities at run-time by training, say, neural net-

works on the distribution encoded in the model.

But the downside is that neural networks are not

interpretable, and although there is considerable

work on unwrapping the functionings of neural

networks Sharma et al. (2019); Belle and Papanto-

ICAART 2023 - 15th International Conference on Agents and Artiﬁcial Intelligence

652

nis (2020), either by inspecting the internal nodes

or doing post-hoc analysis, transparency as a ﬁrst-

class object is ultimately lost.

• Balancing Transparency and Computation.

Perhaps a midpoint in between these extremes is

the emerging work on statistical relational learn-

ing and neuro-symbolic AI Gutmann et al. (2011);

Hu et al. (2016). The idea is to empower prob-

abilistic and deep learning models with logical

templates, either as a speciﬁcation language, a

training function, or a classiﬁcation target so that

(respectively) experts can encode their knowledge

using logic, perform data-efﬁcient learning using

domain-speciﬁc logical rules, or extract logical

rules for post-hoc inspection. It remains to be seen

whether this line of work will bear more fruit in

the long run.

We refer readers interested in learning about other

types of transparent models to Arrieta et al. (2019);

Mehrabi et al. (2021); Belle and Papantonis (2020)

3.2 User Competence and

Trustworthiness

Model transparency is an essential component for im-

portant applications, but, as argued earlier, it is rather

narrow scoped, since it does not address the perplex-

ing complexity of incorporating AI into society. En-

gaging with parties that either use or are affected by

AI can lead to signiﬁcant advances in terms of ensur-

ing its proper use as well as establishing a trusting

human-AI relationship. In fact, there seems to be a

strong link between these two desiderata, supported

by evidence suggesting that users’ understanding and

competence have a great inﬂuence on the amount of

trust placed upon an automated system Balfe et al.

(2018); Sheridan and Telerobotics (1992); Merritt and

Ilgen (2008). Fostering trust then, may have direct

implications on the adoption of AI in practical appli-

cations Linegang et al. (2006); Wright et al. (2019).

Having said that, trust is not a binary distinction

where one can only trust or distrust a model. Rather

than that, a user might trust a model’s outcomes for a

certain sub-population, while being suspicious of de-

cisions concerning another sub-population (perhaps

one that is under-represented in the dataset). This

showcases that trust is a thing to be adjusted or cal-

ibrated, so models are employed appropriately Zhang

et al. (2020). Failing to do so, can potentially lead to

an over-reliance on the model’s decisions Cummings

(2004) or model aversion, where users entirely dis-

miss a model after a few mistakes are made Dietvorst

et al. (2015).

There is already a considerable body of work that

explores the use of XAI generated explanations as

a means to enhance trust Mahbooba et al. (2021);

Guo (2020); Gunning et al. (2019); Gunning and Aha

(2019). This is a fairly reasonable approach, resem-

bling the way that humans justify their decisions by

providing information that is relevant to their decision

making process. Despite that, recent studies provide

evidence both in favour Lai et al. (2020); Lai and Tan

(2019) and against Poursabzi-Sangdeh et al. (2021b);

Chu et al. (2020); Carton et al. (2020) the utility of

explanations in making a model’s internal reasoning

clear to users. This has raised concerns about the way

users perceive XAI explanations overall, calling for

additional surveys to shed some light on this topic

Doshi-Velez and Kim (2017); Vaughan and Wallach

(2020).

Along this line, there is also alarming evidence

suggesting that practitioners utilize XAI techniques in

a wrong way Kaur et al. (2020). An important obser-

vation here is that misuse may arise both due to an in-

complete technical understanding of XAI, as well as

due to misunderstandings regarding XAI’s intended

use. This situation clearly impedes trust calibration,

and thus achieving transparency in AI’s social integra-

tion. Here are some avenues by means of which we,

as a community, can contribute to the understanding

and embedding of trust:

• Clarify Use. We need to establish frameworks

that establish the proper use of XAI, while also

developing a framework that can be used in order

to calibrate trust between human users and AI.

• Explicate Limitations. While there is a plethora

of technical XAI contributions, studying the ad-

vantages and limitations of each explanation type,

as well as ways they can be combined to convey a

more complete picture of a model’s decision mak-

ing process, has not received as much attention

by the AI community. We need to identify the

most prominent explanation types and techniques,

discuss the kind of insights each one offers, and

suggest conceptual frameworks to further empha-

size their distinctions. Finally, we need to propose

ways to combine multiple explanations together

in order to gain a more well-versed understanding

of a model. Some recent surveys such as Arri-

eta et al. (2019); Mehrabi et al. (2021); Wu et al.

(2019b); Belle and Papantonis (2020) are starting

to paint such a picture.

• Education in XAI. As mentioned earlier, practi-

tioners face various kinds of challenges when ap-

plying XAI techniques, most of them stemming

from their incomplete understanding of the ﬁeld.

Model Transparency: Why Do We Care?

653

A natural step to address this issue would be to of-

fer the affected parties sufﬁcient education to ap-

propriately understand and apply the right tech-

niques. However, there is a stark lack of aca-

demic resources on XAI, such as university level

courses. Of course, there are online articles dis-

cussing related things, but this is not a holistic,

systematic approach. In fact, there is only a sin-

gle academic course on XAI, offered by Harvard

University Lakkaraju and Lage (2019), as well

as some tutorials Samek and Montavon (2020);

Camburu and Akata (2021), but they are usu-

ally intended for researchers. We need to pro-

vide guidelines for implementing and delivering

courses, including coding assignments with con-

crete feedback.

• Trust Calibration and Model Comprehension

One of the ultimate goals of XAI is to facilitate

building trusting relationships between users and

AI. While educating people on the technical de-

tails and underlying principles is a step towards

this goal, there are additional factors to be consid-

ered to ensure proper use. A concerning ﬁnding

is that data scientists might understand explana-

tions, but instead of using them in order to fur-

ther inspect a model, they use them to construct

narratives to convince themselves that the model

performs as it should Kaur et al. (2020).

4 CONCLUSIONS

We have brieﬂy surveyed the importance of model

transparency and suggested some directions for fu-

ture work. We consider both technical directions, dis-

cussing the computation vs expressiveness tradeoff,

and socio-technical ones. We hope we convey the ur-

gency of the matter to readers, and that they are en-

couraged to come up with novel solutions that pro-

mote the responsible and safe integration of AI into

critical social applications.

ACKNOWLEDGEMENTS

This research was partly supported by a Royal Soci-

ety University Research Fellowship, UK, and partly

supported by a grant from the UKRI Strategic Priori-

ties Fund, UK to the UKRI Research Node on Trust-

worthy Autonomous Systems Governance and Regu-

lation (EP/V026607/1, 2020–2024).

REFERENCES

Aleksander, I. (2017). Partners of humans: a realistic as-

sessment of the role of robots in the foreseeable future.

Journal of Information Technology, 32(1):1–9.

Alvarado, O. and Waern, A. (2018). Towards algorithmic

experience: Initial efforts for social media contexts.

In Proceedings of the 2018 CHI Conference on Hu-

man Factors in Computing Systems, CHI ’18, pages

1–12, New York, NY, USA. Association for Comput-

ing Machinery.

Angwin, J., Larson, J., Mattu, S., and Kirchner, L. (2016).

Machine bias. In Ethics of Data and Analytics, pages

254–264. Auerbach Publications.

Arrieta, A. B., D

ıaz-Rodr

ıguez, N., Del Ser, J., Bennetot,

A., Tabik, S., Barbado, A., Garc

ıa, S., Gil-L

opez, S.,

Molina, D., Benjamins, R., et al. (2020). Explainable

artiﬁcial intelligence (xai): Concepts, taxonomies, op-

portunities and challenges toward responsible ai. In-

formation fusion, 58:82–115.

Arrieta, A. B., Rodr

ıguez, N. D., Ser, J. D., Bennetot, A.,

Tabik, S., Barbado, A., Garc

ıa, S., Gil-Lopez, S.,

Molina, D., Benjamins, R., Chatila, R., and Herrera, F.

(2019). Explainable artiﬁcial intelligence (XAI): con-

cepts, taxonomies, opportunities and challenges to-

ward responsible AI. CoRR, abs/1910.10045.

Balfe, N., Sharples, S., and Wilson, J. R. (2018). Under-

standing is key: An analysis of factors pertaining to

trust in a real-world automation system. Human fac-

tors, 60(4):477–495.

Bekker, J., Davis Leuven, J. K., Choi, A., Darwiche, A.,

and Van den Broeck, G. (2014). Tractable Learning

for Complex Probability Queries.

Belle, V. and Papantonis, I. (2020). Principles and practice

of explainable machine learning.

Bennett, J., Lanning, S., and Netﬂix, N. (2007). The netﬂix

prize. In In KDD Cup and Workshop in conjunction

with KDD.

Bhatt, U., Andrus, M., Weller, A., and Xiang, A. (2020a).

Machine learning explainability for external stake-

holders. arXiv preprint arXiv:2007.05408.

Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A.,

Jia, Y., Ghosh, J., Puri, R., Moura, J. M., and Eck-

ersley, P. (2020b). Explainable machine learning in

deployment. In Proceedings of the 2020 conference

on fairness, accountability, and transparency, pages

648–657.

Bozdag, E. (2013). Bias in algorithmic ﬁltering and per-

sonalization. Ethics and information technology,

15(3):209–227.

Bozdag, E. and Van Den Hoven, J. (2015). Breaking the

ﬁlter bubble: democracy and design. Ethics and in-

formation technology, 17(4):249–265.

Camburu, O.-M. and Akata, Z. (2021). Natural-xai: Ex-

plainable ai with natural language explanations. Inter-

national Conference on Machine Learning.

Carton, S., Mei, Q., and Resnick, P. (2020). Feature-based

explanations don’t help people detect misclassiﬁca-

tions of online toxicity. In Proceedings of the Inter-

ICAART 2023 - 15th International Conference on Agents and Artiﬁcial Intelligence

654

national AAAI Conference on Web and Social Media,

volume 14, pages 95–106.

Castelletti, A. and Soncini-Sessa, R. (2007). Bayesian net-

works and participatory modelling in water resource

management. Environmental Modelling & Software,

22(8):1075–1088.

Chen, N., Ribeiro, B., and Chen, A. (2016). Financial credit

risk assessment: A recent review. Artif. Intell. Rev.,

45(1):1–23.

Chouldechova, A. (2017). Fair prediction with disparate im-

pact: A study of bias in recidivism prediction instru-

ments. Big Data, 5(2):153–163. PMID: 28632438.

Christian, B. (2020). The alignment problem: Machine

learning and human values. WW Norton & Company.

Christin, A. (2017). Algorithms in practice: Comparing

web journalism and criminal justice. Big Data & So-

ciety, 4(2):2053951717718855.

Chu, E., Roy, D., and Andreas, J. (2020). Are visual ex-

planations useful? a case study in model-in-the-loop

prediction. arXiv preprint arXiv:2007.12248.

Cooper, G. F. (1990). The computational complexity

of probabilistic inference using bayesian belief net-

works. Artiﬁcial intelligence, 42(2-3):393–405.

Culverhouse, P. F., Williams, R., Reguera, B., Herry, V.,

and Gonz

alez-Gil, S. (2003). Do experts make mis-

takes? a comparison of human and machine indenti-

ﬁcation of dinoﬂagellates. Marine ecology progress

series, 247:17–25.

Cummings, M. (2004). Automation bias in intelligent time

critical decision support systems. In AIAA 1st intelli-

gent systems technical conference, page 6313.

Darwiche, A. (2003). A differential approach to inference

in bayesian networks. Journal of the ACM (JACM),

50(3):280–305.

Darwiche, A. (2009). Modeling and reasoning with

Bayesian networks. Cambridge university press.

Dechter, R., Meiri, I., and Pearl, J. (1991). Temporal con-

straint networks. Artiﬁcial intelligence, 49(1-3):61–

95.

Dietvorst, B. J., Simmons, J. P., and Massey, C. (2015).

Algorithm aversion: people erroneously avoid algo-

rithms after seeing them err. Journal of Experimental

Psychology: General, 144(1):114.

Doshi-Velez, F. and Kim, B. (2017). Towards a rigorous sci-

ence of interpretable machine learning. arXiv preprint

arXiv:1702.08608.

Dressel, J. and Farid, H. (2018). The accuracy, fairness,

and limits of predicting recidivism. Science advances,

4(1):eaao5580.

Finlay, S. (2011). Multiple classiﬁer architectures and their

application to credit risk assessment. European Jour-

nal of Operational Research, 210(2):368–378.

Floridi, L., Cowls, J., Beltrametti, M., Chatila, R.,

Chazerand, P., Dignum, V., Luetge, C., Madelin, R.,

Pagallo, U., Rossi, F., et al. (2018). Ai4people—an

ethical framework for a good ai society: Opportuni-

ties, risks, principles, and recommendations. Minds

and Machines, 28(4):689–707.

Friis-Hansen, A. (2000). Bayesian networks as a decision

support tool in marine applications.

Gabriel, I. (2020). Artiﬁcial intelligence, values, and align-

ment. Minds and machines, 30(3):411–437.

Galanos, V. (2019). Exploring expanding expertise: arti-

ﬁcial intelligence as an existential threat and the role

of prestigious commentators, 2014–2018. Technology

Analysis & Strategic Management, 31(4):421–432.

Geiger, D., Verma, T., and Pearl, J. (1990). d-separation:

From theorems to algorithms. In Machine Intelligence

and Pattern Recognition, volume 10, pages 139–148.

Elsevier.

Gillespie, T. (2016). #Trendingistrending: When Algo-

rithms Become Culture. Routledge.

Goh, Y., Cai, X., Theseira, W., Ko, G., and Khor, K.

(2020). Evaluating human versus machine learning

performance in classifying research abstracts. Scien-

tometrics, 125.

Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., and

Nelson, C. (2000). Clinical versus mechanical pre-

diction: a meta-analysis. Psychological assessment,

12(1):19.

Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Gian-

notti, F., and Pedreschi, D. (2018). A survey of meth-

ods for explaining black box models. ACM computing

surveys (CSUR), 51(5):1–42.

Gunning, D. and Aha, D. (2019). Darpa’s explainable

artiﬁcial intelligence (xai) program. AI magazine,

40(2):44–58.

Gunning, D., Steﬁk, M., Choi, J., Miller, T., Stumpf, S.,

and Yang, G.-Z. (2019). Xai—explainable artiﬁcial

intelligence. Science Robotics, 4(37):eaay7120.

Guo, W. (2020). Explainable artiﬁcial intelligence for 6g:

Improving trust between human and machine. IEEE

Communications Magazine, 58(6):39–45.

Gutmann, B., Thon, I., Kimmig, A., Bruynooghe, M., and

De Raedt, L. (2011). The magic of logical inference

in probabilistic programming. TPLP, 11:663–680.

Hilder, S., Harvey, R. W., and Theobald, B.-J. (2009). Com-

parison of human and machine-based lip-reading. In

AVSP, pages 86–89.

Hood, C. and Heald, D. (2006). Transparency in historical

perspective. Number 135. Oxford University Press.

Hu, Z., Ma, X., Liu, Z., Hovy, E. H., and Xing, E. P.

(2016). Harnessing deep neural networks with logic

rules. CoRR, abs/1603.06318.

Jobin, A., Ienca, M., and Vayena, E. (2019). The global

landscape of ai ethics guidelines. Nature Machine In-

telligence, 1(9):389–399.

Kalet, A. M., Gennari, J. H., Ford, E. C., and Phillips, M. H.

(2015). Bayesian network models for error detection

in radiotherapy plans. Physics in Medicine & Biology,

60(7):2735.

Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H.,

and Wortman Vaughan, J. (2020). Interpreting inter-

pretability: understanding data scientists’ use of inter-

pretability tools for machine learning. In Proceedings

of the 2020 CHI conference on human factors in com-

puting systems, pages 1–14.

Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., and

Mullainathan, S. (2017). Human Decisions and Ma-

Model Transparency: Why Do We Care?

655

chine Predictions*. The Quarterly Journal of Eco-

nomics, 133(1):237–293.

Kononenko, I. (2001). Machine learning for medical diag-

nosis: history, state of the art and perspective. Artif.

Intell. Medicine, 23(1):89–109.

Lai, V., Liu, H., and Tan, C. (2020). ” why is’

chicago’deceptive?” towards building model-driven

tutorials for humans. In Proceedings of the 2020 CHI

Conference on Human Factors in Computing Systems,

pages 1–13.

Lai, V. and Tan, C. (2019). On human predictions with ex-

planations and predictions of machine learning mod-

els: A case study on deception detection. In Proceed-

ings of the conference on fairness, accountability, and

transparency, pages 29–38.

Lakkaraju, H. and Lage, I. (2019). Interpretability and ex-

plainability in machine learning.

Larsson, S. (2019). The socio-legal relevance of artiﬁcial

intelligence. Droit et societe, (3):573–593.

Larsson, S., Anneroth, M., Fell

ander, A., Fell

ander-Tsai, L.,

Heintz, F., and

Angstr

om, R. C. (2019). Sustainable

ai: An inventory of the state of knowledge of ethical,

social, and legal challenges related to artiﬁcial intelli-

gence.

Linardatos, P., Papastefanopoulos, V., and Kotsiantis, S.

(2020). Explainable ai: A review of machine learn-

ing interpretability methods. Entropy, 23(1):18.

Linegang, M. P., Stoner, H. A., Patterson, M. J., Seppelt,

B. D., Hoffman, J. D., Crittendon, Z. B., and Lee, J. D.

(2006). Human-automation collaboration in dynamic

mission planning: A challenge requiring an ecolog-

ical approach. In Proceedings of the human factors

and ergonomics society annual meeting, volume 50,

pages 2482–2486. SAGE Publications Sage CA: Los

Angeles, CA.

Loftus, T., Tighe, P., Filiberto, A., Efron, P., Brakenridge,

S., Mohr, A., Rashidi, P., Upchurch, G., and Biho-

rac, A. (2019). Artiﬁcial intelligence and surgical

decision-making. JAMA Surgery, 155.

Mahbooba, B., Timilsina, M., Sahal, R., and Serrano, M.

(2021). Explainable artiﬁcial intelligence (xai) to en-

hance trust management in intrusion detection sys-

tems using decision tree model. Complexity, 2021.

Margetts, H. (2011). The internet and transparency. The

Political Quarterly, 82(4):518–521.

McGough, M. (2018). How Bad is Sacramento’s

Air, Exactly? Google Results Appear at Odds

with Reality, some say, chapter [online] Available:

https://www.sacbee.com/news/california/ﬁres/article216227775.html.

Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., and

Galstyan, A. (2021). A survey on bias and fairness in

machine learning. ACM Comput. Surv., 54(6).

Mehrotra, R., McInerney, J., Bouchard, H., Lalmas, M., and

Diaz, F. (2018). Towards a fair marketplace: Counter-

factual evaluation of the trade-off between relevance,

fairness & satisfaction in recommendation systems.

In Proceedings of the 27th ACM International Con-

ference on Information and Knowledge Management,

CIKM ’18, pages 2243–2251, New York, NY, USA.

Association for Computing Machinery.

Merritt, S. M. and Ilgen, D. R. (2008). Not all trust is

created equal: Dispositional and history-based trust

in human-automation interactions. Human factors,

50(2):194–210.

Meyer, D. (2018). Amazon Reportedly Killed an AI Re-

cruitment System Because It Couldn’t Stop the Tool

from Discriminating Against Women, chapter [online]

Available: https://fortune.com/2018/10/10/amazon-

ai-recruitment-bias-women-sexist/.

Mittelstadt, B., Russell, C., and Wachter, S. (2019). Ex-

plaining explanations in ai. In Proceedings of the con-

ference on fairness, accountability, and transparency,

pages 279–288.

Pearl, J. (1988). Probabilistic Reasoning in Intelligent Sys-

tems: Networks of Plausible Inference. Morgan Kauf-

mann.

Pearl, J. (2009). Causality: Models, Reasoning and Infer-

ence. Cambridge University Press, USA, 2nd edition.

Poon, H. and Domingos, P. (2011). Sum-product net-

works: A new deep architecture. In 2011 IEEE Inter-

national Conference on Computer Vision Workshops

(ICCV Workshops), pages 689–690.

Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M.,

Wortman Vaughan, J. W., and Wallach, H. (2021a).

Manipulating and measuring model interpretability. In

Proceedings of the 2021 CHI Conference on Human

Factors in Computing Systems, CHI ’21, New York,

NY, USA. Association for Computing Machinery.

Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M.,

Wortman Vaughan, J. W., and Wallach, H. (2021b).

Manipulating and measuring model interpretability. In

Proceedings of the 2021 CHI conference on human

factors in computing systems, pages 1–52.

Ross, C. and Swetlitz, I. (2018). Ibm’s watson super-

computer recommended ‘unsafe and incorrect’cancer

treatments, internal documents show. Stat, 25.

Rudin, C. (2019). Stop explaining black box machine learn-

ing models for high stakes decisions and use inter-

pretable models instead. Nature Machine Intelligence,

1(5):206–215.

Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L.,

and Zhong, C. (2022). Interpretable machine learn-

ing: Fundamental principles and 10 grand challenges.

Statistics Surveys, 16:1–85.

Russell, S., Dewey, D., and Tegmark, M. (2015). Re-

search priorities for robust and beneﬁcial artiﬁcial in-

telligence. Ai Magazine, 36(4):105–114.

Samek, W. and Montavon, G. (2020). Explainable ai for

deep networks. European Conference on Machine

Learning and Principles and Practice of Knowledge

Discovery in Databases.

Sharma, S., Henderson, J., and Ghosh, J. (2019). CERTI-

FAI: counterfactual explanations for robustness, trans-

parency, interpretability, and fairness of artiﬁcial intel-

ligence models. CoRR, abs/1905.07857.

Shenton, W., Hart, B. T., and Chan, T. U. (2014). A

bayesian network approach to support environmental

ﬂow restoration decisions in the yarra river, australia.

Stochastic Environmental Research and Risk Assess-

ment, 28(1):57–65.

ICAART 2023 - 15th International Conference on Agents and Artiﬁcial Intelligence

656

Sheridan, T. B. and Telerobotics, A. (1992). Human super-

visory control. Cambridge, MA: MIT Press.

Srivastava, A. and Sutton, C. (2017). AUTOENCODING

VARIATIONAL INFERENCE FOR TOPIC MOD-

ELS.

Stewart-Koster, B., Bunn, S., Mackay, S., Poff, N., Naiman,

R. J., and Lake, P. S. (2010). The use of bayesian

networks to guide investments in ﬂow and catchment

restoration for impaired river ecosystems. Freshwater

Biology, 55(1):243–260.

Strickland, E. (2019). Ibm watson, heal thyself: How ibm

overpromised and underdelivered on ai health care.

IEEE Spectrum, 56(4):24–31.

Uusitalo, L. (2007). Advantages and challenges of bayesian

networks in environmental modelling. Ecological

modelling, 203(3-4):312–318.

Vaughan, J. W. and Wallach, H. (2020). A human-centered

agenda for intelligible machine learning. Machines

We Trust: Getting Along with Artiﬁcial Intelligence.

Wexler, R. (2017). When a computer program keeps you in

jail. New York Times.

Wright, J. L., Chen, J. Y., and Lakhmani, S. G. (2019).

Agent transparency and reliability in human–robot in-

teraction: the inﬂuence on user conﬁdence and per-

ceived reliability. IEEE Transactions on Human-

Machine Systems, 50(3):254–263.

Wu, M., Huang, Y., and Duan, J. (2019a). Investigations

on classiﬁcation methods for loan application based

on machine learning. In 2019 International Confer-

ence on Machine Learning and Cybernetics (ICMLC),

pages 1–6.

Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., and Yu,

P. S. (2019b). A comprehensive survey on graph neu-

ral networks. CoRR, abs/1901.00596.

Zhang, Y., Liao, Q. V., and Bellamy, R. K. (2020). Effect of

conﬁdence and explanation on accuracy and trust cali-

bration in ai-assisted decision making. In Proceedings

of the 2020 Conference on Fairness, Accountability,

and Transparency, pages 295–305.

Model Transparency: Why Do We Care?

657