Managing the tension between opposing effects of explainability of artificial intelligence: a contingency theory perspective

2.3.1 Overview of theoretical perspectives in AI explainability

Despite large and valuable bodies of research in philosophy, psychology and the cognitive science of human decision-making, most work in explainable AI relies on the researchers' intuition of what constitutes a “good” explanation (Miller, 2019). Yet, drawing from social science and other relevant theories is key to making explainable AI useable (Miller, 2017). While most studies, particularly technical papers in AI and data analytics, lack theoretical social and psychological implications and inputs (e.g. Felzmann et al., 2019), scholars in other fields are increasingly making contributions that reference the theoretical behavioral underpinning of AI explainability.

For instance, Wang et al. (2019) draw from the fields of philosophy and psychology to offer some perspectives on designing theory-driven user-centric explainable AI. Hoffman and Klein (2017) discuss several theoretical foundations of what explanation entails and how people understand explanations, discuss causation versus causal reasoning and demonstrate theoretical implications of the close relationship between explanation and abductive inference. However, their theoretical discussion does not outline the implications of XAI outcomes for stakeholders and their reasoning goals. Guzman and Lewis (2020) address the disconnect between communication theory and emerging technologies and argue that the interactions between AI systems and humans do not fit neatly into traditional communication theory paradigms. They draw on a human-machine communication framework and call for more research into (1) the functional aspects through which users make sense of AI systems as communicators, (2) the relational dynamics through which users associate with these systems and, in turn, relate to themselves and peers and (3) the metaphysical implications called up by unclear boundaries between humans, AI systems and communication. Mohamed et al. (2020) explore the role of postcolonial and decolonial theories in understanding AI and establishing ethical principles for protecting vulnerable peoples. They argue that a decolonial critical approach can help AI communities develop insights and tactics to promote and guide transparent and accountable AI systems development.

However, much more needs to be done in theorizing human–AI interactions, particularly in understanding the contradictory effects of AI system explainability and transparency. While business and information systems scholars have been contemplating the theoretical foundations of AI systems since the late 1990s (e.g. Gregor and Benbasat, 1999), their primary focus has been on the technical evolution of AI. Future researchers have been invited to challenge existing theories and utilize their potential for addressing complex questions in AI and data analytics (Berente et al., 2019). In line with this invitation, we draw on contingency theory as a theoretical perspective that acknowledges multiple pathways and discuss how this perspective can encourage future research into AI explainability while considering its unwanted effects. As outlined in the next section, we chose this theory over other perspectives since it explicitly assumes there is no best solution to a problem. This fits the complex problem of explainability in AI systems as the theory suggests taking the circumstances of the problem into consideration and encourages problem solvers to engage with the context in finding a suitable solution.

2.3.2 Contingency theory

Contingency theory is a management theory or model that originated in organizational theory about leadership effectiveness (Feidler, 1964). It has become increasingly accepted because it opposes traditional management theory's contention that there is one best way of doing things (Csaszar and Ostler, 2020). Contingency theory offers two key principles for task-performance fitness: that there is no best way to do things or manage organizations and that a task may be conducted differently in different organizations depending on environmental and contextual factors (Galbraith, 1973). It emphasizes the uncertainty of the environment in which the organization operates and describes effectiveness as organizations coming to terms with their environment(s) to achieve desirable performance for given tasks (Tosi and Slocum, 1984). While the theory does not explicitly identify the underlying concepts of effectiveness, Tosi and Slocum (1984) point out that a balance between profitability (or service delivery for non-profit), satisfaction and social responsibility is key.

Contingency theory has been widely used in other disciplines, including information systems (IS) and IT management literature. While in the IS literature, it was initially used to study systems design and implementation, it was later applied to studying the influence of the environment and other variables on task-performance fitness (Reinking, 2012). Contingency theory suggests there is no single or best way an IS can be used in all situations; rather, contingent factors (i.e. those that cannot be influenced by the organization) need to be considered in engagement with users and other stakeholders to effectively design and use information systems (Shao et al., 2016). Some scholars have criticized early IS research for ill-defined performance expectations and recommended that the theory be used in a less deterministic manner and focus on particular contexts and applications (Weill and Olson, 1989).

This paper uses contingency theory as a theoretical lens for guiding organizations and decision-makers to manage AI explainability effects. The theory's focus on the contexts in which systems operate has been employed in earlier research to subject the explainability of AI to its domain (Lecue, 2019; Bellotti and Edwards, 2001). We draw on the four key principles of contingency theory (Feidler, 1964; Weill and Olson, 1989) to develop perspectives for managing the opposing effects of AI explainability. The four principles are: there is no universal or best way of doing things, the design of organizations and subsystems need to fit their environment, effective organizations need to fit their environments as well as their subsystems, and the needs and goals of an organization are better satisfied when its management style fits both the task (i.e. AI explanation) and the nature of the work (i.e. opposing effects) within the environment (i.e. the context in which the AI system operates).

4.1.1 Comprehensibility

Although, articles on explainability in AI have been abundant in the academic and business press in recent years, it is often surprisingly difficult to comprehend what the term explainability or its alternative terms mean in the context of AI. This ambiguity in the definition is inherited from the lack of clarity about what AI itself means; for instance, Kaplan and Haenlein (2019) describe interpretations of AI as being “as white as snow, as red as blood, as black as ebony” (p. 17). The absence of a precise definition has fueled conflict over expectations of explainability (Berkelaar, 2014). Miller (2019) stresses that despite considerable research in philosophy, psychology and cognitive science into how people define, generate, select, evaluate and present explanations, most work on explainable AI uses only the researchers' intuition about what constitutes a “good” explanation.

Contemporary explainability researchers have started to ask the fundamental question What is an explanation? (Adadi and Berrada, 2018). Scholars have begun to challenge or reject usage of the terms “explanation” or “transparency”, stressing that while they may be useful for developers or domain expert, they might be not comprehensible for laypeople, especially because (for instance) philosophers have an entirely different interpretation of what explanations are (Ntoutsi et al., 2020). What is implied by explanation has to do with how AI systems are “intelligent” in ways that are similar to human intelligence and thus need to compensate for their intelligence in the same way as a person; they need to justify their actions as, for example, a child might when rebuked (Harper, 2019). Páez (2019) argues that the quest for explainability in AI should be rewritten in terms of the wider goal of providing a realistic and naturalistic interpretation. Intuitively, the objective of presenting an interpretation of a concept or judgment is to make it comprehensible to its stakeholders. Without a prior understanding of what it means to suggest that an agent recognizes a concept or a judgment, explanatory strategies lack a well-defined objective.

There appear to be two schools of thoughts about defining explainability in AI, which overlap and, to some extent, contradict. One represents scholars who argue for unbiased explanations by machines in a way that idealizes a precise explanation that is comprehensible by humans. For instance, Guidotti et al. (2018) associate the notion of explanation to an interface between humans and a decision-maker, simultaneously, both an accurate proxy of the decision-maker and comprehensible to humans. Arrieta et al. (2020) distinguish between explainability, understandability, transparency and interpretability and define explainability in AI as “given a certain audience, explainability refers to the details and reasons a model gives to make its functioning clear or easy to understand” (p. 84). In this definition, they note that machines may have different audiences (e.g. users, decision-makers, policymakers) and that a concise reason for AI needs to be provide and customized for each.

On the other hand, Miller (2019) argues that people use particular prejudices and societal perceptions when creating and evaluating explanations that can enhance human experiences with explanatory AI. Since people assign human-like traits to artificial agents, they will expect explanations using the same conceptual framework used for human behaviors (De Graaf and Malle, 2017). While in an ideal sphere, AI and human systems would be equal and correspond with ground truth, this might not occur in the real world due to two problems (Holzinger et al., 2019). Ground truth is often hard or sometimes impossible to define, especially in contexts like medicine; man-made systems and models often look for causal reasons and justifications for understanding fundamental mechanisms. In comparison, AI systems are based on models that often provide only a probability analysis for further establishing causal argumentations. Thus, some scholars offer an alternative view of explainability and argue that the real aim is rather “explicability” of the result of the process—not the explainability of the process itself (Robbins, 2019b). This call for explicability places emphasis on requiring stakeholders to provide explanations when needed rather than requiring them for every decision. The same approach can apply to AI systems.

4.1.2 Conduct

There is an ongoing and seemingly natural tension between the conduct of AI systems (i.e. performance) and their explainability (Gunning and Aha, 2019; Silver et al., 2016). This is due to considerable research on deep learning and black box models for making faster and more accurate AI systems working—intentionally or unintentionally—against making these systems more explainable (Gunning et al., 2019). This problem requires considering trade-offs, including precision and fidelity, to forge a balance between performance and explainability.

It is often assumed that top-performing AI systems get the lowest marks for explainability, on the basis that approximation involves complexity because models require vast amounts of data to be collected and complex multi-variable models to be used for interpretation, learning and analysis (Diez-Olivan et al., 2019). However, more recently, scholars have begun to question this assumption (Robbins, 2019b) and argue that more complex models are not necessarily more accurate. Diez-Olivan et al. (2019) claim that the assumption is incorrect when the data are well structured, and features at our disposal are of great quality and value. However, it is true that more complex models are likely to be more flexible than their simpler counterparts and thus suitable for more complicated tasks. Diez-Olivan et al. stress that using inappropriate complex predictive models falls into the trap of overcomplicating problems, especially when insufficient data diversity (variance) exists. Thus, the added complexity of the model will only work against the explainability of the system.

D'Acquisto (2020) emphasizes the value of a balanced perspective in the battle between performance and explainability and points out that while a “certain level of transparency” (as opposed to a “certain level of autonomy”) of black box AI helps reduce mistrust in the system, the quest for explainability and transparency should not destabilize other principles and logical constraints (p. 899). Unreasonable and unjust AI explainability requirements can be a disincentive for innovation, especially as innovations are often protected by specific intellectual property laws with limited openness to external stakeholders. Setting a regime of AI explainability while stimulating performance and innovation can protect the interests of system owners and domain expert users.

4.1.3 Confidentiality

Holzinger et al. (2019) question the assumption that humans are always able to provide an explanation for their decisions and stress that experts are often unable to offer reasons for their decisions due to reasons such as heterogeneous and vast sources of information, as well as confidentiality, safety, security, privacy and ethical reasons. For instance, Arrieta et al. (2020) point to adversarial attacks. External parties try to manipulate an algorithm after learning how it operates, as an important security threat as a result of explainability. Attacks on a supervised machine learning classification model, for example, would reveal the minimum changes needed to be applied to the inputs to create a different classification output.

A challenge in dealing with algorithmic decision-making and analytics is that their underlying codes are often trade secrets and thus hidden from the public eye (Tóth, 2019); this maintains their competitive advantage, as well preventing the system from being “played”. For some organizations, protecting sensitive and novel algorithmic systems is a top priority, but this work against the system's transparency and explainability to outsiders and makes compliance challenging. The resulting non-explainable system can end in organizations overprotecting their AI machines, leading to dereliction of responsibility and abuse of power.

The contradiction between confidentiality and explainability will be a major challenge as AI use and adoption, as well as competition between AI developers and owners, rise in coming years (Arrieta et al., 2020), even though cutting-edge AI technology offers new promise for algorithmic copyright enforcement (Tóth, 2019). Imagine a domain-specific algorithm that a company has developed over several years of research and investment. The company may consider the knowledge synthesized in the algorithm confidential, and as such, may rightfully find it compromised even by providing basic information about its input and output for explainability purposes. Explaining the system can enable the development of techniques for attacking and confusing the system and more accurate and efficient ways to improve the system's privacy and security (Arrieta et al., 2020).

4.1.4 Completeness

The completeness constraint in AI has been described as “the absence of a possible use of the machine beyond design requirements” (D'Acquisto, 2020, p. 896). Completeness is a by-product of Gödel's first incompleteness theorem (Gödel, 1931), which suggests that logics based on a set of axioms, along with rules of symbolic combinations of statements about those axioms, cannot be proved or explained in formal systems (such as an algorithm). This has major implications for the design and explanation of AI (D'Acquisto, 2020).

Research on AI completeness aspires to ensure machines do not generate or lead to the generation of harmful outputs (Silver et al., 2016). More specifically, the goal of ensuring AI systems act ethically and non-maleficent requires a realistic technical design that stops the system from reaching any known harmful states in which its behavior can be considered dangerous to humans (Arrieta et al., 2020). However, achieving this goal needs to deal with two challenges: (1) AI may progress to a point currently understood as safe, but which may later turn out to be unsafe, for instance, in specific complex models, and (2) there may exist unfamiliar or unobservable conditions that humans can only infer, which are reachable by AI and unsafe to humans—for instance, when an AI system is used in an unprecedented application (Arrieta et al., 2020).

Silver et al. (2016) stress that in examining completeness in AI, researchers should consider that humans' cognitive abilities, and thus their ability to process and understand complex AI models, are limited. These limitations can lead to incompleteness in AI systems, which can make AI explainability a partially achievable goal. Silver et al. argue that factors such as the competitiveness or unpredictability of the contexts in which AI systems operate or interact with humans can make explainability of the system merely wishful thinking, which leaves us wondering if humans should strive for less ambitious but more practical explainability expectations.

While explanations can help system developers, decision-makers and other stakeholders to understand AI systems better or to challenge their incorrect assumptions, one needs to note that explanations alone do not produce understanding (Kroll, 2018). In order to reduce the chances of unwanted harmful outcomes, the explanation needs to be complemented by other evidence to be believable, cover potential causes of an outcome and engage and be targeted to particular stakeholders (Preece, 2018).

4.1.5 Confidence in AI

Trust has been discussed widely in the technology use literature as a key factor in AI explainability. Similarly, gaining users' and other stakeholders' trust is considered a major pillar of the explainability of AI systems (Páez, 2019; Lecue, 2019). Yet, Mittelstadt et al. (2016) show that while explainability of AI systems is important to maintain a trusting relationship with data subjects, a lot of confidence is already placed in some AI systems, especially if the system provider is well known to the user or the system has been in use for a long time. While increased confidence in AI can be associated with increased AI use, overconfidence in AI outputs can lead to de-responsibilization of human stakeholders or a tendency to “hide behind the computer” and assume that AI outputs are correct by default. Thus, it is important to determine the level of trust in AI that would be needed or is even possible and the expected and unexpected consequences of trust in these systems.

Pieters (2011) makes a distinction between trust and confidence and suggests that for trust, unlike confidence, risks are apparent and compared. Pieters argues that explainability makes AI systems more observable, which creates two scenarios: explanation-for-trust, which allows users to compare options by describing them in detail and explanation-for-confidence, which enables them to be confident in using a system without having to consider options. In the former, the AI black box needs to be “opened” to build trust, whereas in the latter, it is often used to give the user confidence in the system's outputs without the need to open it. Felzmann et al. (2019) emphasize the contexts in which AI and humans operate and interact and that it is in some conditions that many things or interactions can be considered trustworthy. This, in turn, implies whether explainability is always needed for establishing trust with the user. Miller (2019) highlights that the quest to build trust in AI systems is not just a computational or machine-related problem. As stressed by other scholars (Bertino et al., 2019), the involvement of several other actors (e.g. those who examine or audit compliance of an AI system) needs to be considered in establishing trust and confidence. Sometimes these parties are semi-trusted, and entrusting them with AI codes could give them the power to risk the interests of the AI owner or developer or violate transparency regulations. Harper (2019) argues that building trust between human and AI actors requires scrutiny of complex interactions between and within them, which requires a multidisciplinary engagement between business, IS and HCI scholars and AI engineers. Harper suggests that a lack of such engagement has often led to a less-than-comprehensive design of human–AI interactions by technical engineers. While some of their designs are innovative, they are not good solutions in terms of interface and interactions with humans, damaging trust in the system.

4.2.1 Pragmatism in explainability

Explainability in AI and its effects, consequences and expectations start with its initial intentions and what we mean by it. Earlier, we presented various perspectives on expectations of explainability and their contradictory effects due to the lack of a comprehensive definition. In addressing this, Garibaldi (2019) stresses that if human experts are prone to error and are not expected to achieve 100% performance in all tasks and explain the underlying reasons, then why should we not expect the same from “expert” computer systems? An answer to this question is Robbins' (2019a) argument that the property of requiring explainability and accountability should be assigned to a particular task or output rather than the entity (i.e. AI system) undertaking the task. This invites scholars and practitioners to focus on the contexts of tasks and their potential damage rather than the process by which tasks are undertaken. This makes more sense when AI is employed for low-risk purposes, for which justification is unnecessary. Requiring explainability and accountability from AI would prevent or slow down the realization of the benefits of AI in many low-risk and some high-risk situations (Robbins, 2019b; Waltl and Vogl, 2018). When Google's AlphaGo defeated the world champion Go player Ke Jie, it did not offer any explanations for its moves, and this lack of explanation did not concern or hurt anyone. As such, Robbins (2019a) argues that many AI systems fall into this category.

Based on the arguments outlined above, scholars are increasingly calling for a more pragmatic approach in understanding and interpreting explainability in AI, one that relies on the context, balanced agency between AI and humans, and measurement of explainability. For instance, Felzmann et al. (2019) propose to understand explainability relationally, meaning the interaction between humans and AI is considered communication between technology and its stakeholders. Its trustworthiness is gauged by contextual and other factors that mediate the value of explainability of the communication. In helping to operationalize such a pragmatic understanding of explainability, Miller (2019) proposes four characteristics:

• (1)Explanations should be contrastive—responsive to particular counterfactual events. For instance, we do not ask why event P occurred, but we ask why event P happened instead of some event Q;

• (2)Explanations should be selected. People rarely expect a full explanation that contains all possible causes of an event. Rather, influenced by certain cognitive biases, they choose one or two causes from all potential causes;

• (3)Probabilities probably do not matter. While people appreciate truth and likelihoods, referring to probabilities or statistics in explanation is not as effective as knowing the causes; and

• (4)Explanations are social. People understand explanations as part of the information exchanged in a conversation or interaction relative to their beliefs and background.

4.2.2 Contextualization of the explanation

In its recent communication “Artificial Intelligence for Europe” (European Commission, 2018), the European Commission portrays the distinctive characteristics of AI systems in the presence of “a certain degree of autonomy” in decision-making. It stresses that the crucial issue in the design and dissemination of these systems is the context in which they operate. Explanations occur and make sense in a context that includes the tasks, capabilities and expectations of the user of the AI system (Bellotti and Edwards, 2001). Thus, interpretations of transparency and explainability in AI are domain-dependent and should not occur in isolation from the particularity of the domain. However, context is rarely considered in existing research on AI explainability (Lecue, 2019).

Lawless et al. (2019) stress the importance of context in managing AI systems and describe context as everything that shapes and influences our perception, cognition and actions in a given environment. They promote the interdependence between humans and AI, which needs to be mastered and properly designed for efficient human–AI teamwork. Miller (2019) builds on the four characteristics above of explanation in AI, emphasizing that in understanding AI outputs, the focus should not just be on associations and causes, but on the context in which the system operates. This study proposes three important points that need to be considered in building truly explainable AI systems: (1) of various potential causes of an event, the explainee usually cares about a small subset (located in the context), (2) the explainer picks a subset (founded on various criteria) for offering the explanation and (3) an interaction or argument may occur between explainer and explainee about this explanation.

Overall, contextualization of the explanation appears to rely heavily on the interaction between the actors, which are the AI system and the users or stakeholders influenced by the system outputs. Bellotti and Edwards (2001) argue that there exist human elements of context that the machine cannot recognize. Therefore, AI systems cannot be designed or expected to act on our behalf. Instead, the system needs to comply with users and relevant regulations in an efficient and non-obtrusive fashion. These authors put stress on the interaction between humans and AI systems and the need to conceptualize the expectations from each party. They highlight that it is unlikely that AI developers and designers can sufficiently clearly determine human aspects of context to allow the system to act on humans' behalf. Rather, what is needed is a set of design principles to enable humans to understand and be guided about interactions with the AI system and to the reason for themselves how best to proceed.

4.2.3 Cohabitation of human agency and AI agency

The third perspective aims to help improve our understanding of the interactions between human and AI systems and shed some light on responsibility for the outcomes. Research on the explainability and transparency of AI has predominantly focused on technical and algorithmic factors, and relatively little has been done on human factors and interactions between human and AI systems (Adadi and Berrada, 2018; Kroll, 2018). Human factors are equally (if not more) important considerations in understanding AI explainability. For instance, in medicine, while explainability of AI systems is very necessary for applications such as education, research and clinical decision-making, human experts must stay engaged in these processes to understand and to review the AI systems' decision processes and outputs (Holzinger et al., 2019).

Research on the explainability of AI is still unsettled with respect to the agency of AI and how responsibility for AI decisions and consequences should be distributed between humans and AI systems. Extant literature widely assumes that only humans should be regarded as responsible agents (Adadi and Berrada, 2018). The idea that all responsibility can be allocated to humans and not machines assumes that AI systems are not stakeholders and have no interests to defend (D'Acquisto, 2020). This idea suggests that regulations and ethics are only applicable to humans and that it is humans' responsibility to allow or avoid the point of no return of AI autonomy, and therefore humans are the only responsible agents for AI system decisions.

In contrast to the concepts discussed above, scholars are increasingly asking about and arguing for the agency of the AI system. Ågerfalk (2020) posits the idea of digital agency for AI and stresses that information systems like AI are not solely demonstrating an external reality; rather, these systems are dynamic participants in reality which also engage in forming it. Coeckelbergh (2009) points out that AI systems do things that influence us in many ways, and what happens as the result of this influence needs to be discussed in terms of right and wrong, which in turn implies that we can regard AI systems as agents or moral agents. Coeckelbergh argues that if non-humans (natural and artificial) can significantly affect our lives, it is undesirable and unhelpful to exclude them from moral equations. This raises the idea of whether AI systems themselves can be responsible agents (Coeckelbergh, 2020), which in turn engenders the idea of a hybrid approach in deciding the responsibility of AI systems. In this hybrid approach, human agency and AI agency would coexist, and responsibility for the outputs would be distributed across a network of humans and machines, acknowledging that:

1) technology shapes human action in a way that goes beyond a merely instrumental role, that (2) the actions of AI-driven technology can be morally relevant…, and that (3) advanced AI technologies may give the appearance of being responsible agents. (Coeckelbergh, 2020, p. 2,054)

This discussion highlights that organizations and decision-makers need to make responsibility expectations clear to all stakeholders. Furthermore, societies need to think about strategies that can promote policies that ensure machine outcomes can be controlled and harmful consequences can be assessed and acted upon by a trusted centralized entity (D'Acquisto, 2020). In assessing and managing machine outcomes, individual experiences can be distinguished from group experiences. An explanation of AI outcomes is prioritized for group effects and for safeguarding the public interest.

4.2.4 Metrics and standardization

An important aspect of evaluating AI systems and managing their performance is metrics that measure a model's performance in evaluating a particular aspect of explainability and make sense to stakeholders (Garibaldi, 2019). However, very few of the studies we reviewed provided clear metrics for evaluating or measuring explainability effects or performance. This result is in line with earlier reviews and surveys; for instance, Adadi and Berrada (2018) found very little work on evaluating explainability methods and quantifying their relevance.

Scholars and practitioners are increasingly calling for metrics that can help compare, evaluate and quantify explainability methods (Preece, 2018). The field clearly needs unified concepts for measuring explainability effects and outcomes necessary for enabling and guiding future research on techniques and methods for AI explainability. Metrics would enable clarity in the evaluation of how well a variable would fit with the explainable description. Without such mechanisms, every argument in the literature lacks a stable foundation (Mohseni et al., 2018).

While existing methods lack quantitative and comprehensive scales for measuring various aspects of explainability of AI and comparing explainable techniques (Arrieta et al., 2020; Gunning and Aha, 2019; Gunning et al., 2019), some tools offer useful insights for the development of the next generation of explainability metrics. For example, Mohseni et al. (2018) present a multiscale tool for measuring AI explainability that includes the goodness of the model fit, explanation satisfaction indicators, assessment of mental models, indicators of the reliability of computations and explanation trustworthiness. D'Acquisto (2020) promotes value transparency in measuring the usefulness of AI systems by disclosing criteria humans and machines can employ to settle disputes. Poursabzi-Sangdeh et al. (2018) propose a general model called Network Dissection, which aims to quantify interpretability in terms of alignment with a set of human-interpretable concepts.

Doshi-Velez and Kim (2017) identify three types of interpretability assessments:

• (1)Application-grounded—a user (normally a domain expert) tests the explanation in a particular application domain,

• (2)Human-grounded—laypeople, rather than experts, assess the explanation in an experimental format and

• (3)Functionally-grounded—explanation is assessed through pre-specified models and with no human subjects engaged.

Mohseni and Ragan (2018) build on the above three assessment types and develop a human-grounded evaluation standard for estimating instant explanations of images and textual data. By associating the explanation results from classification models with the evaluation standard, they test the quality and relevance of local explanations. Bertino et al. (2019) offer the standardization of transparency data exchange and transparency score; the former stresses the use of acceptable data formats to share transparency information about the data, while the latter presents the level of transparency that is available for a user about a dataset. Finally, Gunning and Aha (2019) develop an evaluation framework for DAPRA that includes five dimensions: user satisfaction (clarity and utility of the explanation, utility of the explanation), mental model (assessing individual decisions and their strength/weakness), task performance (impact of the explanation on the user's decision/task performance), trust assessment (users' trust and the likelihood of AI use in future) and correctability (opportunities for identification and correction of errors).

4.2.5 Regulatory and ethical principles

Despite the increasing amount of research on the explainability of AI, there remains substantial ambiguity about what elements of explainability need to be incorporated in AI systems, how, and to what extent, and how users can be made aware of their rights in interactions with machines. There is an academic and professional consensus that ethical and legal frameworks can help; governments in Australia (Australian Government, 2019), China (Zhang, 2019), the EU (European Commission, 2019) and elsewhere have developed and promoted principles for explainability and ethics in AI. Major technology companies like Microsoft and Google and the World Economic Forum have drafted AI ethics guidelines under the umbrella of “explicability” (Robbins, 2019a). Many of these principles share the general aim of ensuring that AI supports and does not limit human autonomy; to achieve this aim, we need to understand how humans may be affected by AI and make sure we know how AI will act on our behalf (Floridi et al., 2018).

However, many of the frameworks mentioned above are still under development, and there is uncertainty about the specifics of their underlying ethical and legal principles (Bertino et al., 2019; Robbins, 2019a). Bertino et al. (2019) highlight the need for regulatory requirements for data transparency to be enforced by authorized entities like governments, standards bodies, or corporate governing authorities to maintain and encourage visibility and explainability in the use and propagation of AI systems. These requirements need to acknowledge that specific criteria must be applied on a case-by-case basis so that different purposes, context and actors can be considered in making judgments about outcomes and effects of AI use and explainability. Tóth (2019) calls for more non-profit and research organizations to develop their own enforcement algorithms to create more balanced competition and transparency for emerging platforms. This will lead to an increased number of actors engaged in AI development, improve understanding of explainability effects and alleviate some of the pressure caused by trade secrecy and competition between for-profit organizations. Stahl and Wright (2018) suggest the notion of responsible research and innovation (RRI) as a framework for ensuring the acceptability and sustainability of technologies in society. The European Commission uses RRI to expand on six key issues: public engagement, ethics, science education, gender equality, open access, governance (European Commission, 2019, 2020). Yet, despite RRI's benefits in providing a guide for key elements of explainability of AI systems, it lacks a clear understanding of the effects and impacts of AI systems on relevant stakeholders and how to respond to such effects.

There is debate about the regulations or principles that may help with AI explainability and accountability in the law and ethics literature. Hacker et al. (2020) point out that most current legal debate focuses on data protection, and particularly on whether the GDPR implies a right to an explanation of automated decisions. Hacker et al. (2020) emphasize that the law and AI are interwoven and often mutually reinforcing. They argue that explainability of the AI system is an imperative legal category, not just in data protection law but in contract and tort law. They propose that contract and tort law are more likely than data protection law to lead to enforceable legal requirements in regard to explainable machine learning models.

Bellotti and Edwards (2001) four general categories for accountability of context-aware systems could be used as a basis for identifying ethical and legal principles for enforcing and guiding the use of explainability in AI systems:

• (1)Inform the user of the system about its abilities and possibilities,

• (2)Offer users feedback about using the system:

  • Feedforward—so that the user understands the consequences if they take a particular action,

  • Confirmation—so that the user understands what they have done,

• (3)Enforce identity and disclosure, particularly with sharing sensitive data and

• (4)Provide users with control over the system, especially when user actions may influence them.

4.2.6 Other emerging solutions

Research into making AI more explainable and transparent is dynamic and evolving. Scholars are continuously looking for new techniques to complement existing methods or innovative approaches to advance the field. In particular, the review of the selected articles revealed a growing interest in using AI enveloping, blockchain and fuzzy systems in improving and managing AI explainability.

Robbins (2019a) presents an alternative narrative to AI transparency and explainability, suggesting that instead of requiring transparency from machines, the focus should limit their power within physical and virtual microenvironments. This ensures machines can perform their functions whilst reducing the risk of harm to humans; in robotics, this is called “envelopment” (Floridi, 2011). Floridi offers the example of dishwashers. These machines are properly enveloped within a certain context to perform their operations, which has made them useful and effective tools; the alternative would be an ineffective humanoid dishwashing robot. Robbins (2019a) builds on the idea of envelopment and argues that AI will be effective if it is successfully constrained within an “envelope” to produce desired outputs within a given limited capacity. The key requirement for AI envelopment is a sound knowledge of its inputs, outputs, functions, training data and boundaries, which is often lacking. Researchers working on AI envelopment continue to look for ways to obtain and incorporate such knowledge into the AI system.

To improve the trustworthiness of explainability, Nassar et al. (2020) propose a framework that leverages blockchain features. Their framework uses smart contracts, trusted oracles and decentralized storage to model explainability decisions subject to a consensus between distributed agents, with the assumption that the majority of agents involved are truthful. They pose that blockchain-enabled explainability can offer transparency and visibility, immutability, traceability and nonrepudiation, and smart contracts. Kshetri (2019) argue that AI and blockchain complement each other substantially. The study poses that blockchain can break the dominance of a few major players in AI because blockchain predictions are immutable, and the reputation of AI providers would depend more on their service quality capabilities rather than providers' size and reputation. Bertino et al. (2019) stress that to improve transparency and explainability, unified policies and contracts are needed to implement ethical principles. Blockchain is, therefore, in a good position to encode such policies into smart contracts for enforcement among networks of peers and help with tracking the origins of data as it evolves, which in turn provides a dynamic consensus in the network for handling special situations.

Fuzzy sets were introduced in 1965 by Zadeh (1965) specifically to deal with difficult “classes of objects encountered in the real physical world” which are “imprecisely defined” and yet “play an important role in human thinking”. Fernandez et al. (2019) present a detailed discussion of evolutionary fuzzy systems and show the synergy between fuzzy systems and evolutionary algorithms in AI. They assert that such synergy can offer a decent trade-off between the accuracy and explainability of AI models. They discuss how evolutionary fuzzy systems can incorporate key features of explainability in machine learning systems for making outputs more understandable to stakeholders. Garibaldi (2019) highlights the inherent uncertainty in real-world knowledge and data and that fuzzy systems are well placed to deal with uncertainty in decision support and knowledge representation and inference. Garibaldi presents indistinguishability as the key factor in the assessment of computerized decision support systems and argues that the technical and non-technical aspects of AI explainability are interconnected and need to be used together and in an understandable fashion when the system offers explanations to humans. Following this, Garibaldi (2019) seeks answers to the question “can AI explain itself?” and proposes using the Turing Test (Turing, 2009) to answer it. This test poses a problem involving an AI system and humans and checks if the user can distinguish between the system and the human object. If the user cannot make the distinction, then AI presents an acceptable level of performance regardless of its accuracy.