Deep neural networks are superior to dermatologists in melanoma image classification.

Melanoma is the most dangerous type of skin cancer but is curable if detected early. Recent publications demonstrated that artificial intelligence is capable in classifying images of benign nevi and melanoma with dermatologist-level precision. However, a statistically significant improvement compared with dermatologist classification has not been reported to date. For this comparative study, 4204 biopsy-proven images of melanoma and nevi (1:1) were used for the training of a convolutional neural network (CNN). New techniques of deep learning were integrated. For the experiment, an additional 804 biopsy-proven dermoscopic images of melanoma and nevi (1:1) were randomly presented to dermatologists of nine German university hospitals, who evaluated the quality of each image and stated their recommended treatment (19,296 recommendations in total). Three McNemar's tests comparing the results of the CNN's test runs in terms of sensitivity, specificity and overall correctness were predefined as the main outcomes. The respective sensitivity and specificity of lesion classification by the dermatologists were 67.2% (95% confidence interval [CI]: 62.6%-71.7%) and 62.2% (95% CI: 57.6%-66.9%). In comparison, the trained CNN achieved a higher sensitivity of 82.3% (95% CI: 78.3%-85.7%) and a higher specificity of 77.9% (95% CI: 73.8%-81.8%). The three McNemar's tests in 2 × 2 tables all reached a significance level of p < 0.001. This significance level was sustained for both subgroups. For the first time, automated dermoscopic melanoma image classification was shown to be significantly superior to both junior and board-certified dermatologists (p < 0.001).

Brinker TJ ，Hekler A ，Enk AH ，Berking C ，Haferkamp S ，Hauschild A ，Weichenthal M ，Klode J ，Schadendorf D ，Holland-Letz T ，von Kalle C ，Fröhling S ，Schilling B ，Utikal JS ... -  《-》

Systematic outperformance of 112 dermatologists in multiclass skin cancer image classification by convolutional neural networks.

Recently, convolutional neural networks (CNNs) systematically outperformed dermatologists in distinguishing dermoscopic melanoma and nevi images. However, such a binary classification does not reflect the clinical reality of skin cancer screenings in which multiple diagnoses need to be taken into account. Using 11,444 dermoscopic images, which covered dermatologic diagnoses comprising the majority of commonly pigmented skin lesions commonly faced in skin cancer screenings, a CNN was trained through novel deep learning techniques. A test set of 300 biopsy-verified images was used to compare the classifier's performance with that of 112 dermatologists from 13 German university hospitals. The primary end-point was the correct classification of the different lesions into benign and malignant. The secondary end-point was the correct classification of the images into one of the five diagnostic categories. Sensitivity and specificity of dermatologists for the primary end-point were 74.4% (95% confidence interval [CI]: 67.0-81.8%) and 59.8% (95% CI: 49.8-69.8%), respectively. At equal sensitivity, the algorithm achieved a specificity of 91.3% (95% CI: 85.5-97.1%). For the secondary end-point, the mean sensitivity and specificity of the dermatologists were at 56.5% (95% CI: 42.8-70.2%) and 89.2% (95% CI: 85.0-93.3%), respectively. At equal sensitivity, the algorithm achieved a specificity of 98.8%. Two-sided McNemar tests revealed significance for the primary end-point (p < 0.001). For the secondary end-point, outperformance (p < 0.001) was achieved except for basal cell carcinoma (on-par performance). Our findings show that automated classification of dermoscopic melanoma and nevi images is extendable to a multiclass classification problem, thus better reflecting clinical differential diagnoses, while still outperforming dermatologists at a significant level (p < 0.001).

Maron RC ，Weichenthal M ，Utikal JS ，Hekler A ，Berking C ，Hauschild A ，Enk AH ，Haferkamp S ，Klode J ，Schadendorf D ，Jansen P ，Holland-Letz T ，Schilling B ，von Kalle C ，Fröhling S ，Gaiser MR ，Hartmann D ，Gesierich A ，Kähler KC ，Wehkamp U ，Karoglan A ，Bär C ，Brinker TJ ，Collabrators ... -  《-》

A convolutional neural network trained with dermoscopic images performed on par with 145 dermatologists in a clinical melanoma image classification task.

Recent studies have demonstrated the use of convolutional neural networks (CNNs) to classify images of melanoma with accuracies comparable to those achieved by board-certified dermatologists. However, the performance of a CNN exclusively trained with dermoscopic images in a clinical image classification task in direct competition with a large number of dermatologists has not been measured to date. This study compares the performance of a convolutional neuronal network trained with dermoscopic images exclusively for identifying melanoma in clinical photographs with the manual grading of the same images by dermatologists. We compared automatic digital melanoma classification with the performance of 145 dermatologists of 12 German university hospitals. We used methods from enhanced deep learning to train a CNN with 12,378 open-source dermoscopic images. We used 100 clinical images to compare the performance of the CNN to that of the dermatologists. Dermatologists were compared with the deep neural network in terms of sensitivity, specificity and receiver operating characteristics. The mean sensitivity and specificity achieved by the dermatologists with clinical images was 89.4% (range: 55.0%-100%) and 64.4% (range: 22.5%-92.5%). At the same sensitivity, the CNN exhibited a mean specificity of 68.2% (range 47.5%-86.25%). Among the dermatologists, the attendings showed the highest mean sensitivity of 92.8% at a mean specificity of 57.7%. With the same high sensitivity of 92.8%, the CNN had a mean specificity of 61.1%. For the first time, dermatologist-level image classification was achieved on a clinical image classification task without training on clinical images. The CNN had a smaller variance of results indicating a higher robustness of computer vision compared with human assessment for dermatologic image classification tasks.

Brinker TJ ，Hekler A ，Enk AH ，Klode J ，Hauschild A ，Berking C ，Schilling B ，Haferkamp S ，Schadendorf D ，Fröhling S ，Utikal JS ，von Kalle C ，Collaborators ... -  《-》

Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists.

Haenssle HA ，Fink C ，Schneiderbauer R ，Toberer F ，Buhl T ，Blum A ，Kalloo A ，Hassen ABH ，Thomas L ，Enk A ，Uhlmann L ，Reader study level-I and level-II Groups ，Alt C ，Arenbergerova M ，Bakos R ，Baltzer A ，Bertlich I ，Blum A ，Bokor-Billmann T ，Bowling J ，Braghiroli N ，Braun R ，Buder-Bakhaya K ，Buhl T ，Cabo H ，Cabrijan L ，Cevic N ，Classen A ，Deltgen D ，Fink C ，Georgieva I ，Hakim-Meibodi LE ，Hanner S ，Hartmann F ，Hartmann J ，Haus G ，Hoxha E ，Karls R ，Koga H ，Kreusch J ，Lallas A ，Majenka P ，Marghoob A ，Massone C ，Mekokishvili L ，Mestel D ，Meyer V ，Neuberger A ，Nielsen K ，Oliviero M ，Pampena R ，Paoli J ，Pawlik E ，Rao B ，Rendon A ，Russo T ，Sadek A ，Samhaber K ，Schneiderbauer R ，Schweizer A ，Toberer F ，Trennheuser L ，Vlahova L ，Wald A ，Winkler J ，Wölbing P ，Zalaudek I ... -  《-》

Artificial Intelligence and Its Effect on Dermatologists' Accuracy in Dermoscopic Melanoma Image Classification: Web-Based Survey Study.

Early detection of melanoma can be lifesaving but this remains a challenge. Recent diagnostic studies have revealed the superiority of artificial intelligence (AI) in classifying dermoscopic images of melanoma and nevi, concluding that these algorithms should assist a dermatologist's diagnoses. The aim of this study was to investigate whether AI support improves the accuracy and overall diagnostic performance of dermatologists in the dichotomous image-based discrimination between melanoma and nevus. Twelve board-certified dermatologists were presented disjoint sets of 100 unique dermoscopic images of melanomas and nevi (total of 1200 unique images), and they had to classify the images based on personal experience alone (part I) and with the support of a trained convolutional neural network (CNN, part II). Additionally, dermatologists were asked to rate their confidence in their final decision for each image. While the mean specificity of the dermatologists based on personal experience alone remained almost unchanged (70.6% vs 72.4%; P=.54) with AI support, the mean sensitivity and mean accuracy increased significantly (59.4% vs 74.6%; P=.003 and 65.0% vs 73.6%; P=.002, respectively) with AI support. Out of the 10% (10/94; 95% CI 8.4%-11.8%) of cases where dermatologists were correct and AI was incorrect, dermatologists on average changed to the incorrect answer for 39% (4/10; 95% CI 23.2%-55.6%) of cases. When dermatologists were incorrect and AI was correct (25/94, 27%; 95% CI 24.0%-30.1%), dermatologists changed their answers to the correct answer for 46% (11/25; 95% CI 33.1%-58.4%) of cases. Additionally, the dermatologists' average confidence in their decisions increased when the CNN confirmed their decision and decreased when the CNN disagreed, even when the dermatologists were correct. Reported values are based on the mean of all participants. Whenever absolute values are shown, the denominator and numerator are approximations as every dermatologist ended up rating a varying number of images due to a quality control step. The findings of our study show that AI support can improve the overall accuracy of the dermatologists in the dichotomous image-based discrimination between melanoma and nevus. This supports the argument for AI-based tools to aid clinicians in skin lesion classification and provides a rationale for studies of such classifiers in real-life settings, wherein clinicians can integrate additional information such as patient age and medical history into their decisions.

Maron RC ，Utikal JS ，Hekler A ，Hauschild A ，Sattler E ，Sondermann W ，Haferkamp S ，Schilling B ，Heppt MV ，Jansen P ，Reinholz M ，Franklin C ，Schmitt L ，Hartmann D ，Krieghoff-Henning E ，Schmitt M ，Weichenthal M ，von Kalle C ，Fröhling S ，Brinker TJ ... -  《JOURNAL OF MEDICAL INTERNET RESEARCH》