User-centric Explanation of Machine Learning Model for Human-AI Collaboration
Abstract:
Artificial Intelligence (AI) has been deployed in various aspects of human lives. While most AI models remain to be black-boxes, there is an increasing demand to explain the machine to human users to moderate trust in the machine and support user decision-making together with the AI systems, especially for critical decisions. However, humans and machines reason and explain in different ways. Many explainable AI (XAI) techniques are not designed from the human user's point of view. While both the machine and the human can be wrong in making decisions, the machine should fit the human mental model so that the machine can be better understood, trusted, and improved. More importantly, the machine should also handle the case when the human tends to make mistakes and improve human decision-making by providing suitable AI explanations. We focus on the user-centric XAI and advocate that explanations of machine learning models should be designed to 1) fit the user mental model and 2) to improve user decision making.
Our work aim to tighten the connections between human reasoning and explaining AI especially when human and machine models are flawed. Specifically, we worked on three research questions. Firstly, we investigated the theoretical underpinnings of the connections between human decision-making and AI explanations by asking what types of explanations should be provided to users that can fit the user's reasoning model and mitigate the user's reasoning errors. To resolve this, we first looked into the human reasoning theories of philosophy, psychology and cognitive science, and we reviewed the literature on XAI techniques. Then we proposed a conceptual framework that links human reasoning theories to XAI techniques as a guideline to select explanation types and methods. The framework was further evaluated in a case study by applying it to build a XAI medical diagnosis system. Secondly, we investigated more deeply how machine learning explanations should be built for different human reasoning preferences in a challenging case where people have different strategies to handle uncertain information and averse to risk. We questioned how to communicate the explanation uncertainty due to input noise to users. We proposed two approaches that show or suppress explanation uncertainty and evaluated them in a simulation study and user studies. Results demonstrated that suppressing uncertainty can improve user decision-making quality, trust to model, and confidence, while showing uncertainty slows users down but improves their understanding of the model. Lastly, when the user and the model reason and explain differently, we study how to bring the model explanation closer to the user mental model in a common but challenging case of explaining the deep learning model on unstructured data. Human explains their decision by high-level concepts and causality, but existing explanation methods on deep learning models like Convolutional Neural Networks (CNN) are based on spurious correlations and uninterpretable features learned by the model. We propose a neural-network based framework that generates conceptual and causal explanation, and we add priors to the model explanation by regularizing the model training so the model explanation is closer to prior knowledge and correct causal relation. In the experiment, we evaluate our model on synthetic datasets and a real-world chest X-ray diagnosis dataset, and we demonstrate our conceptual causal explainable model is closer to ground truth knowledge. In summary, we have strengthened the theoretical connection between human reasoning and AI explanation, provided technical solutions to explain and improve machine learning models inspired by human mental models, and empirically evaluated our techniques for better human decision-making.