From clinical decision support to clinical reasoning support systems.

Despite the great promises that artificial intelligence (AI) holds for health care, the uptake of such technologies into medical practice is slow. In this paper, we focus on the epistemological issues arising from the development and implementation of a class of AI for clinical practice, namely clinical decision support systems (CDSS). We will first provide an overview of the epistemic tasks of medical professionals, and then analyse which of these tasks can be supported by CDSS, while also explaining why some of them should remain the territory of human experts. Clinical decision making involves a reasoning process in which clinicians combine different types of information into a coherent and adequate 'picture of the patient' that enables them to draw explainable and justifiable conclusions for which they bear epistemological responsibility. Therefore, we suggest that it is more appropriate to think of a CDSS as clinical reasoning support systems (CRSS). Developing CRSS that support clinicians' reasoning process therefore requires that: (a) CRSSs are developed on the basis of relevant and well-processed data; and (b) the system facilitates an interaction with the clinician. Therefore, medical experts must collaborate closely with AI experts developing the CRSS. In addition, responsible use of an CRSS requires that the data generated by the CRSS is empirically justified through an empirical link with the individual patient. In practice, this means that the system indicates what factors contributed to arriving at an advice, allowing the user (clinician) to evaluate whether these factors are medically plausible and applicable to the patient. Finally, we defend that proper implementation of CRSS allows combining human and artificial intelligence into hybrid intelligence, were both perform clearly delineated and complementary empirical tasks. Whereas CRSSs can assist with statistical reasoning and finding patterns in complex data, it is the clinicians' task to interpret, integrate and contextualize.

van Baalen S ，Boon M ，Verhoef P 《-》

Clinicians' roles and necessary levels of understanding in the use of artificial intelligence: A qualitative interview study with German medical students.

Artificial intelligence-driven Clinical Decision Support Systems (AI-CDSS) are being increasingly introduced into various domains of health care for diagnostic, prognostic, therapeutic and other purposes. A significant part of the discourse on ethically appropriate conditions relate to the levels of understanding and explicability needed for ensuring responsible clinical decision-making when using AI-CDSS. Empirical evidence on stakeholders' viewpoints on these issues is scarce so far. The present study complements the empirical-ethical body of research by, on the one hand, investigating the requirements for understanding and explicability in depth with regard to the rationale behind them. On the other hand, it surveys medical students at the end of their studies as stakeholders, of whom little data is available so far, but for whom AI-CDSS will be an important part of their medical practice. Fifteen semi-structured qualitative interviews (each lasting an average of 56 min) were conducted with German medical students to investigate their perspectives and attitudes on the use of AI-CDSS. The problem-centred interviews draw on two hypothetical case vignettes of AI-CDSS employed in nephrology and surgery. Interviewees' perceptions and convictions of their own clinical role and responsibilities in dealing with AI-CDSS were elicited as well as viewpoints on explicability as well as the necessary level of understanding and competencies needed on the clinicians' side. The qualitative data were analysed according to key principles of qualitative content analysis (Kuckartz). In response to the central question about the necessary understanding of AI-CDSS tools and the emergence of their outputs as well as the reasons for the requirements placed on them, two types of argumentation could be differentiated inductively from the interviewees' statements: the first type, the clinician as a systemic trustee (or "the one relying"), highlights that there needs to be empirical evidence and adequate approval processes that guarantee minimised harm and a clinical benefit from the employment of an AI-CDSS. Based on proof of these requirements, the use of an AI-CDSS would be appropriate, as according to "the one relying", clinicians should choose those measures that statistically cause the least harm. The second type, the clinician as an individual expert (or "the one controlling"), sets higher prerequisites that go beyond ensuring empirical evidence and adequate approval processes. These higher prerequisites relate to the clinician's necessary level of competence and understanding of how a specific AI-CDSS works and how to use it properly in order to evaluate its outputs and to mitigate potential risks for the individual patient. Both types are unified in their high esteem of evidence-based clinical practice and the need to communicate with the patient on the use of medical AI. However, the interviewees' different conceptions of the clinician's role and responsibilities cause them to have different requirements regarding the clinician's understanding and explicability of an AI-CDSS beyond the proof of benefit. The study results highlight two different types among (future) clinicians regarding their view of the necessary levels of understanding and competence. These findings should inform the debate on appropriate training programmes and professional standards (e.g. clinical practice guidelines) that enable the safe and effective clinical employment of AI-CDSS in various clinical fields. While current approaches search for appropriate minimum requirements of the necessary understanding and competence, the differences between (future) clinicians in terms of their information and understanding needs described here can lead to more differentiated approaches to solutions.

Funer F ，Tinnemeyer S ，Liedtke W ，Salloch S ... -  《BMC Medical Ethics》

A Clinical Decision Support System for Sleep Staging Tasks With Explanations From Artificial Intelligence: User-Centered Design and Evaluation Study.

Despite the unprecedented performance of deep learning algorithms in clinical domains, full reviews of algorithmic predictions by human experts remain mandatory. Under these circumstances, artificial intelligence (AI) models are primarily designed as clinical decision support systems (CDSSs). However, from the perspective of clinical practitioners, the lack of clinical interpretability and user-centered interfaces hinders the adoption of these AI systems in practice. This study aims to develop an AI-based CDSS for assisting polysomnographic technicians in reviewing AI-predicted sleep staging results. This study proposed and evaluated a CDSS that provides clinically sound explanations for AI predictions in a user-centered manner. Our study is based on a user-centered design framework for developing explanations in a CDSS that identifies why explanations are needed, what information should be contained in explanations, and how explanations can be provided in the CDSS. We conducted user interviews, user observation sessions, and an iterative design process to identify three key aspects for designing explanations in the CDSS. After constructing the CDSS, the tool was evaluated to investigate how the CDSS explanations helped technicians. We measured the accuracy of sleep staging and interrater reliability with macro-F1 and Cohen κ scores to assess quantitative improvements after our tool was adopted. We assessed qualitative improvements through participant interviews that established how participants perceived and used the tool. The user study revealed that technicians desire explanations that are relevant to key electroencephalogram (EEG) patterns for sleep staging when assessing the correctness of AI predictions. Here, technicians wanted explanations that could be used to evaluate whether the AI models properly locate and use these patterns during prediction. On the basis of this, information that is closely related to sleep EEG patterns was formulated for the AI models. In the iterative design phase, we developed a different visualization strategy for each pattern based on how technicians interpreted the EEG recordings with these patterns during their workflows. Our evaluation study on 9 polysomnographic technicians quantitatively and qualitatively investigated the helpfulness of the tool. For technicians with <5 years of work experience, their quantitative sleep staging performance improved significantly from 56.75 to 60.59 with a P value of .05. Qualitatively, participants reported that the information provided effectively supported them, and they could develop notable adoption strategies for the tool. Our findings indicate that formulating clinical explanations for automated predictions using the information in the AI with a user-centered design process is an effective strategy for developing a CDSS for sleep staging.

Hwang J ，Lee T ，Lee H ，Byun S ... -  《JOURNAL OF MEDICAL INTERNET RESEARCH》

An interoperable clinical decision-support system for early detection of SIRS in pediatric intensive care using openEHR.

Clinical decision-support systems (CDSS) are designed to solve knowledge-intensive tasks for supporting decision-making processes. Although many approaches for designing CDSS have been proposed, due to high implementation costs, as well as the lack of interoperability features, current solutions are not well-established across different institutions. Recently, the use of standardized formalisms for knowledge representation as terminologies as well as the integration of semantically enriched clinical information models, as openEHR Archetypes, and their reuse within CDSS are theoretically considered as key factors for reusable CDSS. We aim at developing and evaluating an openEHR based approach to achieve interoperability in CDSS by designing and implementing an exemplary system for automated systemic inflammatory response syndrome (SIRS) detection in pediatric intensive care. We designed an interoperable concept, which enables an easy integration of the CDSS across different institutions, by using openEHR Archetypes, terminology bindings and the Archetype Query Language (AQL). The practicability of the approach was tested by (1) implementing a prototype, which is based on an openEHR based data repository of the Hannover Medical School (HaMSTR), and (2) conducting a first pilot study. We successfully designed and implemented a CDSS with interoperable knowledge bases and interfaces by reusing internationally agreed-upon Archetypes, incorporating LOINC terminology and creating AQL queries, which allowed retrieving dynamic facts in a standardized and unambiguous form. The technical capabilities of the system were evaluated by testing the prototype on 16 randomly selected patients with 129 days of stay, and comparing the results with the assessment of clinical experts (leading to a sensitivity of 1.00, a specificity of 0.94 and a Cohen's kappa of 0.92). We found the use of openEHR Archetypes and AQL a feasible approach to bridge the interoperability gap between local infrastructures and CDSS. The designed concept was successfully transferred into a clinically evaluated openEHR based CDSS. To the authors' knowledge, this is the first openEHR based CDSS, which is technically reliable and capable in a real context, and facilitates clinical decision-support for a complex task. Further activities will comprise enrichments of the knowledge base, the reasoning processes and cross-institutional evaluations.

Wulff A ，Haarbrandt B ，Tute E ，Marschollek M ，Beerbaum P ，Jack T ... -  《-》

Artificial Intelligence-Augmented Clinical Decision Support Systems for Pregnancy Care: Systematic Review.

Lin X ，Liang C ，Liu J ，Lyu T ，Ghumman N ，Campbell B ... -  《JOURNAL OF MEDICAL INTERNET RESEARCH》