Timothée Schmude

Making algorithmic decisions explainable and contestable is essential to ensure the responsible use of high-risk AI systems. Yet the human-centered design and implementation of explainability and contestability remain underexplored in empirical research. In this paper, we follow a participatory design approach in three stages to develop an interface that supports social counselors in understanding and contesting a welfare fraud detection system. We analyze counselors’ information needs, derive functional and usability requirements, and design and evaluate both low-fidelity and high-fidelity interfaces that implement these requirements. Our findings show that contesting administrative decisions is a complex socio-technical process involving social counselors, clients, and social agencies, requiring carefully tuned explanation and contestation elements. Social counselors prioritized procedural information and tools for interactive exploration of model predictions, while emphasizing that explanations must be concise, easy to understand, and sensitive to the emotional impact of information on clients. We find that human intervention remains crucial in contestation processes, that explanations can support counselors by surfacing contestation reasons and facilitating client-side communication, and that conversational explanations enable flexible inquiry. Our insights contribute to trustworthy AI implementations by outlining the potential and challenges of a human-centered explanation and contestation interface for individuals affected by high-risk AI systems.

Explainability and its emerging counterpart contestability are key normative and design principles for trustworthy AI, enabling users and subjects to understand and challenge AI decisions. Yet realizing these principles is difficult, as they take on different meanings across technical, legal, and organizational dimensions of AI regulation. To address this conceptual polysemy, we report findings from an interview study with 14 experts examining the intersection and implementation of explainability and contestability, and their interpretations in different research communities. We outline differentiations between descriptive and normative explainability, judicial and non-judicial channels of contestation, and individual and collective contestation action. We also identify key points of friction in realizing both principles, including alignment between top-down and bottom-up regulation, assignment of responsibility, and the need for interdisciplinary collaboration. Finally, we offer three AI policy recommendations to operationalize explainability and contestability through a Regulation-by-Design perspective. Our contributions inform policy research and regulation of these core principles, and support more effective and equitable design, development, and deployment of trustworthy public AI systems.

Everyone affected by high-risk AI systems should have the chance to understand them and consider the merits and harms of their deployment. Explanations of AI systems can support this goal, yet it is rarely explored how lay audiences use them for deliberation. In this paper, we examine how explanations support groups and individuals of AI novices in learning and deciding about an AI system. We use question-driven explanations spanning four information categories and evaluate them in a task-based interview study with 8 focus groups and 12 individuals. We find that groups allow participants to use team cognition to make sense of explanations, but rely heavily on social dynamics. In contrast, single settings support focused understanding but lack exchange and discussion. We contribute suggestions to make explanation designs suitable for AI novices and discuss their use in individual and collective settings to support understanding and deliberation of AI systems.

Every AI system that makes decisions about people has stakeholders who are affected by its outcomes. These stakeholders, whom we call decision subjects, have a right to understand how their outcome was produced and to challenge it. Explanations should support this process by making the algorithmic system transparent and creating an understanding of its inner workings. However, we argue that while current explanation approaches focus on descriptive explanations, decision subjects also require normative explanations or justifications. In this position paper, we advocate for justifications as a key component in explanation approaches for decision subjects and make three claims to this end, namely that justifications i) fulfill decision subjects’ information needs, ii) shape their intent to accept or contest decisions, and iii) encourage accountability considerations throughout the system’s lifecycle. We propose four guiding principles for the design of justifications, provide two design examples, and close with directions for future work. With this paper, we aim to provoke thoughts on the role, value, and design of normative information in explainable AI for decision subjects.

Every AI system that makes decisions about people has a group of stakeholders that are personally affected by these decisions. However, explanations of AI systems rarely address the information needs of this stakeholder group, who often are AI novices. This creates a gap between conveyed information and information that matters to those who are impacted by the system’s decisions, such as domain experts and decision subjects. To address this, we present the “XAI Novice Question Bank”, an extension of the XAI Question Bank (Liao et al., 2020) containing a catalog of information needs from AI novices in two use cases: employment prediction and health monitoring. The catalog covers the categories of data, system context, system usage, and system specifications. We gathered information needs through task based interviews where participants asked questions about two AI systems to decide on their adoption and received verbal explanations in response. Our analysis showed that participants’ confidence increased after receiving explanations but that their understanding faced challenges. These included difficulties in locating information and in assessing their own understanding, as well as attempts to outsource understanding. Additionally, participants’ prior perceptions of the systems’ risks and benefits influenced their information needs. Participants who perceived high risks sought explanations about the intentions behind a system’s deployment, while those who perceived low risks rather asked about the system’s operation. Our work aims to support the inclusion of AI novices in explainability efforts by highlighting their information needs, aims, and challenges. We summarize our findings as five key implications that can inform the design of future explanations for lay stakeholder audiences.

Ethical principles for algorithms are gaining importance as more and more stakeholders are affected by "high-risk" algorithmic decision-making (ADM) systems. Understanding how these systems work enables stakeholders to make informed decisions and to assess the systems’ adherence to ethical values. Explanations are a promising way to create understanding, but current explainable artificial intelligence (XAI) research does not always consider existent theories on how understanding is formed and evaluated. In this work, we aim to contribute to a better understanding of understanding by conducting a qualitative task-based study with 30 participants, including users and affected stakeholders. We use three explanation modalities (textual, dialogue, and interactive) to explain a "high-risk" ADM system to participants and analyse their responses both inductively and deductively, using the "six facets of understanding" framework by Wiggins & McTighe [63]. Our findings indicate that the "six facets" framework is a promising approach to analyse participants’ thought processes in understanding, providing categories for both rational and emotional understanding. We further introduce the "dialogue" modality as a valid explanation approach to increase participant engagement and interaction with the "explainer", allowing for more insight into their understanding in the process. Our analysis further suggests that individuality in understanding affects participants’ perceptions of algorithmic fairness, demonstrating the interdependence between understanding and ADM assessment that previous studies have outlined. We posit that drawing from theories on learning and understanding like the "six facets" and leveraging explanation modalities can guide XAI research to better suit explanations to learning processes of individuals and consequently enable their assessment of ethical values of ADM systems.