Assessment of artificial intelligence (AI) reporting methodology in glioma MRI studies using the Checklist for AI in Medical Imaging (CLAIM).

The Checklist for Artificial Intelligence in Medical Imaging (CLAIM) is a recently released guideline designed for the optimal reporting methodology of artificial intelligence (AI) studies. Gliomas are the most common form of primary malignant brain tumour and numerous outcomes derived from AI algorithms such as grading, survival, treatment-related effects and molecular status have been reported. The aim of the study is to evaluate the AI reporting methodology for outcomes relating to gliomas in magnetic resonance imaging (MRI) using the CLAIM criteria. A literature search was performed on three databases pertaining to AI augmentation of glioma MRI, published between the start of 2018 and the end of 2021 RESULTS: A total of 4308 articles were identified and 138 articles remained after screening. These articles were categorised into four main AI tasks: grading (n= 44), predicting molecular status (n= 50), predicting survival (n= 25) and distinguishing true tumour progression from treatment-related effects (n= 10). The average CLAIM score was 20/42 (range: 10-31). Studies most consistently reported the scientific background and clinical role of their AI approach. Areas of improvement were identified in the reporting of data collection, data management, ground truth and validation of AI performance. AI may be a means of producing high-accuracy results for certain tasks in glioma MRI; however, there remain issues with reporting quality. AI reporting guidelines may aid in a more reproducible and standardised approach to reporting and will aid in clinical integration.

Bhandari A ，Scott L ，Weilbach M ，Marwah R ，Lasocki A ... -  《-》

The Low Rate of Adherence to Checklist for Artificial Intelligence in Medical Imaging Criteria Among Published Prostate MRI Artificial Intelligence Algorithms.

To determine the rigor, generalizability, and reproducibility of published classification and detection artificial intelligence (AI) models for prostate cancer (PCa) on MRI using the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) guidelines, a 42-item checklist that is considered a measure of best practice for presenting and reviewing medical imaging AI research. This review searched English literature for studies proposing PCa AI detection and classification models on MRI. Each study was evaluated with the CLAIM checklist. The additional outcomes for which data were sought included measures of AI model performance (eg, area under the curve [AUC], sensitivity, specificity, free-response operating characteristic curves), training and validation and testing group sample size, AI approach, detection versus classification AI, public data set utilization, MRI sequences used, and definition of gold standard for ground truth. The percentage of CLAIM checklist fulfillment was used to stratify studies into quartiles. Wilcoxon's rank-sum test was used for pair-wise comparisons. In all, 75 studies were identified, and 53 studies qualified for analysis. The original CLAIM items that most studies did not fulfill includes item 12 (77% no): de-identification methods; item 13 (68% no): handling missing data; item 15 (47% no): rationale for choosing ground truth reference standard; item 18 (55% no): measurements of inter- and intrareader variability; item 31 (60% no): inclusion of validated interpretability maps; item 37 (92% no): inclusion of failure analysis to elucidate AI model weaknesses. An AUC score versus percentage CLAIM fulfillment quartile revealed a significant difference of the mean AUC scores between quartile 1 versus quartile 2 (0.78 versus 0.86, P = .034) and quartile 1 versus quartile 4 (0.78 versus 0.89, P = .003) scores. Based on additional information and outcome metrics gathered in this study, additional measures of best practice are defined. These new items include disclosure of public dataset usage, ground truth definition in comparison to other referenced works in the defined task, and sample size power calculation. A large proportion of AI studies do not fulfill key items in CLAIM guidelines within their methods and results sections. The percentage of CLAIM checklist fulfillment is weakly associated with improved AI model performance. Additions or supplementations to CLAIM are recommended to improve publishing standards and aid reviewers in determining study rigor.

Belue MJ ，Harmon SA ，Lay NS ，Daryanani A ，Phelps TE ，Choyke PL ，Turkbey B ... -  《-》

Deep learning in knee imaging: a systematic review utilizing a Checklist for Artificial Intelligence in Medical Imaging (CLAIM).

Our purposes were (1) to explore the methodologic quality of the studies on the deep learning in knee imaging with CLAIM criterion and (2) to offer our vision for the development of CLAIM to assure high-quality reports about the application of AI to medical imaging in knee joint. A Checklist for Artificial Intelligence in Medical Imaging systematic review was conducted from January 1, 2015, to June 1, 2020, using PubMed, EMBASE, and Web of Science databases. A total of 36 articles discussing deep learning applications in knee joint imaging were identified, divided by imaging modality, and characterized by imaging task, data source, algorithm type, and outcome metrics. A total of 36 studies were identified and divided into: X-ray (44.44%) and MRI (55.56%). The mean CLAIM score of the 36 studies was 27.94 (standard deviation, 4.26), which was 66.53% of the ideal score of 42.00. The CLAIM items achieved an average good inter-rater agreement (ICC 0.815, 95% CI 0.660-0.902). In total, 32 studies performed internal cross-validation on the data set, while only 4 studies conducted external validation of the data set. The overall scientific quality of deep learning in knee imaging is insufficient; however, deep learning remains a promising technology for diagnostic or predictive purpose. Improvements in study design, validation, and open science need to be made to demonstrate the generalizability of findings and to achieve clinical applications. Widespread application, pre-trained scoring procedure, and modification of CLAIM in response to clinical needs are necessary in the future. • Limited deep learning studies were established in knee imaging with mean score of 27.94, which was 66.53% of the ideal score of 42.00, commonly due to invalidated results, retrospective study design, and absence of a clear definition of the CLAIM items in detail. • A previous trained data extraction instrument allowed reaching moderate inter-rater agreement in the application of the CLAIM, while CLAIM still needs improvement in scoring items and result reporting to become a wide adaptive tool in reviews of deep learning studies.

Si L ，Zhong J ，Huo J ，Xuan K ，Zhuang Z ，Hu Y ，Wang Q ，Zhang H ，Yao W ... -  《-》

Applications of artificial intelligence in magnetic resonance imaging of primary pediatric cancers: a scoping review and CLAIM score assessment.

To review the uses of AI for magnetic resonance (MR) imaging assessment of primary pediatric cancer and identify common literature topics and knowledge gaps. To assess the adherence of the existing literature to the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) guidelines. A scoping literature search using MEDLINE, EMBASE and Cochrane databases was performed, including studies of > 10 subjects with a mean age of < 21 years. Relevant data were summarized into three categories based on AI application: detection, characterization, treatment and monitoring. Readers independently scored each study using CLAIM guidelines, and inter-rater reproducibility was assessed using intraclass correlation coefficients. Twenty-one studies were included. The most common AI application for pediatric cancer MR imaging was pediatric tumor diagnosis and detection (13/21 [62%] studies). The most commonly studied tumor was posterior fossa tumors (14 [67%] studies). Knowledge gaps included a lack of research in AI-driven tumor staging (0/21 [0%] studies), imaging genomics (1/21 [5%] studies), and tumor segmentation (2/21 [10%] studies). Adherence to CLAIM guidelines was moderate in primary studies, with an average (range) of 55% (34%-73%) CLAIM items reported. Adherence has improved over time based on publication year. The literature surrounding AI applications of MR imaging in pediatric cancers is limited. The existing literature shows moderate adherence to CLAIM guidelines, suggesting that better adherence is required for future studies.

Tsang B ，Gupta A ，Takahashi MS ，Baffi H ，Ola T ，Doria AS ... -  《-》

Review of study reporting guidelines for clinical studies using artificial intelligence in healthcare.

High-quality research is essential in guiding evidence-based care, and should be reported in a way that is reproducible, transparent and where appropriate, provide sufficient detail for inclusion in future meta-analyses. Reporting guidelines for various study designs have been widely used for clinical (and preclinical) studies, consisting of checklists with a minimum set of points for inclusion. With the recent rise in volume of research using artificial intelligence (AI), additional factors need to be evaluated, which do not neatly conform to traditional reporting guidelines (eg, details relating to technical algorithm development). In this review, reporting guidelines are highlighted to promote awareness of essential content required for studies evaluating AI interventions in healthcare. These include published and in progress extensions to well-known reporting guidelines such as Standard Protocol Items: Recommendations for Interventional Trials-AI (study protocols), Consolidated Standards of Reporting Trials-AI (randomised controlled trials), Standards for Reporting of Diagnostic Accuracy Studies-AI (diagnostic accuracy studies) and Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis-AI (prediction model studies). Additionally there are a number of guidelines that consider AI for health interventions more generally (eg, Checklist for Artificial Intelligence in Medical Imaging (CLAIM), minimum information (MI)-CLAIM, MI for Medical AI Reporting) or address a specific element such as the 'learning curve' (Developmental and Exploratory Clinical Investigation of Decision-AI) . Economic evaluation of AI health interventions is not currently addressed, and may benefit from extension to an existing guideline. In the face of a rapid influx of studies of AI health interventions, reporting guidelines help ensure that investigators and those appraising studies consider both the well-recognised elements of good study design and reporting, while also adequately addressing new challenges posed by AI-specific elements.

Shelmerdine SC ，Arthurs OJ ，Denniston A ，Sebire NJ ... -  《-》