Highlights
-
•
We performed a head-to-head benchmark of AutoML platforms for DR screening.
-
•
Amazon SageMaker Canvas and AutoGluon achieved the highest AUCs for RDR and STDR.
-
•
Model performance varied markedly across platforms and probability thresholds.
-
•
Several AutoML tools showed moderate agreement with the FDA-approved IDx-DR system.
-
•
External validation and threshold calibration are essential for real-world deployment.
Purpose
To benchmark multiple automated machine learning (AutoML) platforms for diabetic retinopathy (DR) screening from fundus photographs using a unified training and evaluationframework, with human consensus grading and an FDA-approved autonomous system (IDx-DR) as reference standards.
Design
Retrospective, diagnostic performance and benchmarking study.
Methods
Image classifiers were trained on large public datasets labeled according to the International Clinical Diabetic Retinopathy (ICDR) scale (APTOS, n = 5590; DDR, n = 12,524; EyePACS, n = 31,557) after automated image-quality filtering. Performance was evaluated on an independent, institutionally collected patient-level test cohort (n = 726) using the highest DR grade across all images for patient-level classification. The evaluated platforms included Google Vertex AI, Amazon Rekognition, Amazon SageMaker Canvas, AutoGluon, AutoKeras, and Apple CreateML. Models were assessed for 3 screening endpoints—any DR, referable DR (RDR), and sight-threatening DR (STDR)—across probability thresholds of 20%, 50%, and 70%. The primary endpoints were the Area Under the Receiver Operating Characteristic Curve (AUC) and sensitivity for RDR at a 50% decision threshold. Secondary endpoints included specificity, positive predictive value, negative predictive value, accuracy, and F1-score with 95% confidence intervals. Pairwise comparisons were performed using bootstrap testing (n = 1000) for AUC differences and McNemar’s test (at a 50% threshold) for binary outcomes, both subjects to Bonferroni correction ( P <.0033). Grad-CAM was applied to locally deployable convolutional neural network–based models.
Results
Using human consensus grading as the reference standard, Amazon SageMaker Canvas and AutoGluon demonstrated the strongest overall discrimination, achieving AUC values up to0.96 for STDR and 0.93 to 0.94 for RDR. At the 50% decision threshold, Canvas showed the most balanced performance for RDR (sensitivity 88.3%, specificity 85.5%, accuracy 86.0%), whereasAutoGluon favored sensitivity (any DR sensitivity 95.9%) at the expense of specificity. Vertex AI showed consistently weaker and unstable performance (any DR AUC 0.58; RDR AUC 0.38). Relative to IDx-DR, Amazon Rekognition and Canvas showed the highest agreement, particularly for STDR (AUC up to 0.88-0.90; κ up to ∼0.56). Agreement with human graders was generallylow to moderate (κ ≈ 0.3-0.6) and increased at higher probability thresholds.
Conclusions
AutoML platforms can achieve clinically meaningful performance for DR screening. Differences across tools and thresholds reflect their adaptability to diverse clinical settings, underscoring the importance of external validation and threshold calibration.
Introduction
D iabetic retinopathy (DR) remains a leading cause of visual impairment in working-age adults, and its burden is projected to grow with the rising prevalence of diabetes; estimates suggest the global diabetes population may reach ∼700 million by 2045, with roughly one quarter developing DR. Although advances in diabetes management have reduced the relative contribution of DR to vision loss in many countries, population-based DR screening remains a major clinical and organizational undertaking. Screening programs must accommodate large and growing numbers of patients, ensure consistent grading quality, and operate within constrained ophthalmology workforce capacity, limiting coverage and long-term sustainability.
In parallel, deep-learning (DL) systems for DR have matured beyond proof-of-concept. Guideline-aligned, multicenter platforms have achieved high external performance (AUROC ∼0.98-0.99; sensitivity ∼0.97) and parity with expert graders, supporting their suitability for large-scale screening. , Real-world deployments further indicate scalability; for example, an artificial intelligence (AI) assisted screening program spanning 54,353 physical-examination encounters reported negative predictive value (NPV) ≥96% and (positive predictive value) PPV ≥90% while detecting a broad spectrum of retinal abnormalities. However, the development of conventional DL models typically requires advanced programming expertise and substantial engineering infrastructure, limiting direct clinician involvement in model design, training, and local adaptation. Beyond these technical barriers, screening programs are characterized by substantial heterogeneity in imaging devices, grading workflows, population risk profiles, and regulatory requirements. This variability limits the generalizability of single, centrally developed AI solutions and creates a need for flexible approaches that can be locally trained, calibrated, or adapted while maintaining acceptable performance, governance, and quality assurance standards. From an organizational perspective, reliance on a small number of proprietary, end-to-end AI systems may also introduce vendor lock-in, limited transparency, and restricted adaptability over time—factors that are increasingly relevant for procurement decisions and long-term program sustainability.
In this context, automated machine learning (AutoML), also referred to as code-free deep learning (CFDL), has emerged as an alternative paradigm that enables clinician-led model development without coding expertise. By lowering technical barriers, AutoML platforms allow performance, operating thresholds, and deployment strategies to be evaluated and adjusted across multiple tools, supporting flexible integration within diverse screening environments. The diagnostic performance of AutoML models in ophthalmology has been evaluated across multiple imaging modalities. On optical coherence tomography (OCT), Google Vertex AI AutoML has classified common macular conditions with high accuracy (AUPRC = 0.991; sensitivity = 95.9%; specificity = 96.9%), underscoring its potential for scalable image analysis. Using color fundus photographs, AutoML has also achieved strong multiclass discrimination of vascular diseases, including DR, central retinal vein occlusion (CRVO), and branch retinal vein occlusion (BRVO), with AUROC values up to 0.995 and generally robust external validation. Importantly, head-to-head evaluations suggest that CFDL approaches can match bespoke, expert-engineered pipelines when trained on the same data, while offering substantially lower technical overhead.
Recently, AutoML-based approaches have demonstrated robust diagnostic performance for DR across diverse datasets and deployment settings. Code-free and self-training frameworks trained on large public and regional fundus datasets have achieved high discrimination for referral-required DR, with reported AUROC values ranging from ∼0.93 to 0.99 and accuracies exceeding 93%, while remaining accessible to clinicians without coding expertise. ,,, However, beyond diagnostic accuracy, large-scale deployment raises important public health and programmatic considerations. Real-world screening performance is influenced not only by model discrimination but also by image quality requirements and decision thresholds, which directly affect screening coverage, referral rates, and healthcare system capacity. Strict image quality filtering has been shown to increase exclusion rates and reduce effective screening uptake, particularly in community-based settings, while threshold-dependent operating points can substantially alter sensitivity–specificity trade-offs and downstream referral burden. , Moreover, screening programs must ensure quality assurance, consistency across sites, and the ability to audit and cf system performance over time. Comparative evaluation of multiple AutoML platforms under standardized conditions therefore serves not only diagnostic benchmarking, but also program-level decision-making related to scalability, risk management, and sustainable implementation. Existing studies have predominantly evaluated single platforms or narrow experimental settings, often without systematic cross-platform comparison under such real-world constraints. The aim of the present study is therefore to address this gap by providing a comprehensive, head-to-head evaluation of multiple cloud-based and locally deployable AutoML platforms for DR detection from fundus photographs under a unified training and validation protocol. By combining external validation using independent, clinician-annotated datasets with multithreshold performance assessment and qualitative evaluation of usability and efficiency, this study offers a pragmatic benchmark of AutoML readiness for clinical integration.
MATERIALS AND METHODS
This study was designed as a retrospective, diagnostic performance and benchmarking study evaluating multiple AutoML platforms for diabetic retinopathy screening using fundus photographs. This study was approved by the local ethics committee, and the requirement for informed consent was waived due to its retrospective, observational nature. The study adhered to the tenets of the Declaration of Helsinki.
DATA SOURCES AND IMAGE SELECTION
Training dataset
To train each model, we used 3 datasets: APTOS (n = 5590 images), DDR (n = 12,524 images) and EyePACS (n = 31,557 images), containing color fundus photographs labelled according to the International Clinical Diabetic Retinopathy (ICDR) severity scale, comprising normal fundus appearance and 4 stages of DR. These datasets were selected based on their large sample size, well-defined DR annotations, high image resolution, minimal presence of imaging artifacts, and ease of access with appropriate legal permissions for research use. All images in the included datasets were annotated according to DR severity classification or labeled as unreadable. To avoid overly restrictive exclusion criteria and to preserve the heterogeneity of clinically relevant information within the training and validation data, no additional manual filtering of suboptimal images was initially performed.
The fundus photographs were obtained from patients predominantly residing in East Asia, particularly China and India, thereby representing a population distinct from that used in the final model evaluation. Images were captured with a 45-degree field of view. For objective image quality assessment, the training datasets were screened using the MCF-Net framework, which classified each image into 1 of 3 readability categories: (1) “unreadable,” representing images with severe quality degradation that precluded any clinical assessment; (2) “usable,” referring to images with minor optical artifacts or imperfections that did not obscure critical diagnostic regions; and (3) “good,” representing high-quality images without significant artifacts. Only images categorized as “usable“ or “good” were retained for model training as they provided sufficient clarity for clinical interpretation, while those labeled “unreadable” were excluded. Across the combined training datasets, the distribution of DR severity comprised 33,090 images with no retinopathy, 3302 with mild DR, 10,357 with moderate DR, 1123 with severe nonproliferative DR, and 1799 with proliferative DR.
Test dataset
We used an independent, institutionally collected dataset to evaluate the AutoML-based models. Sample size estimation was performed assuming a 2-sided α error of 0.05, a precision of 10%, an expected sensitivity of 85%, and an estimated prevalence of referable diabetic retinopathy (RDR; defined as moderate or worse nonproliferative DR and/or presence of macular edema) of 7%, yielding a required sample size of 700 participants. To account for potential image-quality exclusions (anticipated at ∼10%), the target sample size was increased to 781 participants.
Eligible images were obtained from consecutive Caucasian patients with established diabetes mellitus undergoing routine DR screening in Poland. Nonmydriatic fundus photographs were captured by trained operators using a Topcon NW-400 (Topcon Corp., Tokyo, Japan) camera. For each patient, a minimum of four 45° field-of-view images were acquired, including 2 disc-centered and 2 macula-centered images, with additional images obtained when necessary using the built-in image-quality feedback system of IDx-DR, the first FDA-approved autonomous DR screening system. Patients with insufficient image quality, defined as the absence of at least 1 disc- and 1 macula-centered image passing the IDx-DR quality check were excluded.
Following quality assessment by IDx-DR, 107 patients were excluded due to insufficient image quality. An additional 21 patients were excluded after quality filtering by the RetCAD AI system. The final dataset comprised 3427 fundus images of sufficient quality from 758 patients. As the source screening program only provided access to encounters that passed the IDx-DR quality criteria, all images included in the analysis were deemed acceptable by IDx-DR; therefore, direct comparison of image-quality rejection performance between systems was not feasible. To ensure consistency across datasets, the test set was further screened using the MCF-Net framework, resulting in the exclusion of 914 unreadable images and a final cohort of 726 patients for model evaluation. The image-quality filtering steps were sequential and nonoverlapping: patients failing the initial IDx-DR quality assessment were excluded prior to dataset access, while MCF-Net–based filtering was subsequently applied only to the remaining images to ensure standardized quality criteria across training and evaluation datasets.
Each patient was independently graded for DR severity by 3 ophthalmologists and by the IDx-DR system. Of the included cases, 303 patients were assessed by 3 retinal specialists in Poland, and 423 by 3 certified graders in India. Inter-grader agreement was evaluated using Fleiss’ kappa coefficient, yielding values of 0.44 and 0.54 for the Polish and Indian graders, respectively. DR severity was graded according to the ICDR classification as no DR, mild, moderate, or severe nonproliferative DR, or proliferative DR. Macular edema was identified based on surrogate markers, with clinically significant diabetic macular edema defined by the presence of hard exudates within 1 disc diameter of the fovea. Grading was performed on a per-eye basis, and the final patient-level diagnosis was determined by the more severely affected eye. Consensus grading between expert graders was used as the reference standard for all model comparisons, and all analyses were conducted at the patient level. While inter-grader agreement among human readers was assessed using Fleiss’ kappa on multiclass ICDR severity grades, all agreement analyses comparing AutoML models and IDx-DR with the reference standard were performed on binary classification outputs corresponding to predefined screening endpoints (any DR, RDR, and sight-threatening DR). For each patient, the final DR grade was determined by selecting the highest (most severe) ICDR score identified across all available images from both eyes. This “worst-eye” approach ensures that the patient-level diagnosis captures the highest degree of pathology detected in any single photograph, consistent with international screening standards.
All fundus photographs in training and test sets were fully anonymized prior to grading and model development, with removal of patient identifiers and embedded metadata. Human graders were provided with access only to the retinal images and were masked to all clinical information, demographic variables, and model predictions. Images were reviewed using standard digital displays under routine viewing conditions. No additional image preprocessing (such as cropping, contrast enhancement, normalization, or data augmentation) was performed prior to human grading or model training, apart from automated image quality assessment and exclusion of unreadable images as described above. Images were otherwise used in their original form.
IMAGE QUALITY CONTROL
To ensure data consistency and clinical reliability, all images underwent a 2-stage independent quality control protocol. It is a fundamental principle in clinical decision support systems that the quality assessment module operates independently of the diagnostic module. In this study, training images were prescreened using the MCF-Net architecture. For the test dataset, quality was verified by the IDx-DR quality assessment module, which is functionally separate from its diagnostic component and was not involved in the training or decision-making processes of the analyzed AutoML platforms. Images failing to meet these strict quality criteria were excluded from further processing. This approach mirrors real-world clinical workflows where diagnostic analysis is only performed on images of sufficient technical quality, thereby preventing “garbage-in, garbage-out” errors and ensuring that the performance comparison is based on clinically viable data.
MODEL TRAINING
For comparative evaluation, we included AutoML platforms accessible either as public cloud–based services or as locally deployable tools running on edge devices. AutoML platforms were selected based on the availability of a clinician-accessible, code-free or low-code interface, support for image classification tasks, and feasibility of deployment either as widely used cloud-based services or locally executable tools. Platforms requiring extensive custom coding or lacking stable image-classification workflows were not considered. Based on these criteria, the following platforms were evaluated: Google Vertex AI, Microsoft Azure Machine Learning, Amazon Rekognition, Amazon SageMaker Canvas, AutoGluon, AutoKeras, and Apple Create ML. Owing to the lack of accessible graphical processing units (GPUs) and unresolved resource allocation limitations within the Microsoft Azure cloud environment during the experimental period, Azure Machine Learning was excluded from further analyses.
All AutoML platforms were trained using the same pooled training datasets, identical ICDR labels, and the same image quality filtering strategy applied at the dataset level. Beyond these standardized inputs, internal data handling, augmentation, and optimization procedures were determined by each platform’s proprietary training pipeline and were not manually modified.
Model training was conducted using both cloud-based virtual resources and local computational hardware, depending on platform requirements. Cloud-based AutoML services operate under a serverless computing paradigm, in which computational resources are dynamically allocated and may vary throughout the training process; consequently, precise reporting of hardware specifications was not feasible. For locally executed AutoML tools, 2 distinct hardware configurations were employed. AutoGluon and AutoKeras were trained on a workstation running Arch Linux, equipped with 64 GB of RAM, an AMD Ryzen 9 3950X processor, and an NVIDIA T4 GPU with 16 GB of video memory. Due to platform-specific constraints, Apple Create ML models were trained on a MacBook Pro running macOS version 14.6, featuring an Apple M1 Pro processor and 16 GB of unified memory.
The characteristics of each AutoML platform, together with overall and model-specific training durations, are summarized in Table 1 . As shown, substantial differences in training duration were observed across the evaluated AutoML platforms. These differences primarily reflect variations in model architecture, search strategy, and computational orchestration. Platforms employing extensive neural architecture search and iterative hyperparameter optimization, such as Amazon Canvas, Amazon Rekognition, AutoKeras or Google Vertex AI. These tools required markedly longer training times, whereas cloud-based services relying on more constrained architectural templates and managed resource allocation achieved shorter effective training durations. Locally deployable tools based on vectorized image representations demonstrated the shortest training times, likely due to limited architectural search space. Overall, training time should be interpreted as an intrinsic platform characteristic that strongly depends on the model architecture and training method, as well as on the available computational resources.
TABLE 1
Characteristics of the AutoML Platforms and Tools Used in the Study
| Type | Cost | Type of Learning | Type of Interface | Resourcefulness of Training | Overall Training Time | Model-Specific Training Time (h) | |
|---|---|---|---|---|---|---|---|
| Google Vertex AI | Cloud service | Paid | CNN | Graphical UI | High | 36 h, 11 min | 36,08 |
| Azure Machine Learning Studio | Cloud service | Paid | CNN | Graphical UI | High | Excluded | Excluded |
| Amazon Rekognition | Cloud service | Paid | CNN | Graphical UI | Low | 11 h, 06 min | 6,54 |
| Amazon Canvas | Cloud service | Paid | CNN | Graphical UI | High | 26 h, 32 min | 3,06 |
| Autogluon | Local tool/ Python library | Free | Vectorized image | Programming library | Low | 1 h, 12 min | 1,20 |
| Autokeras | Local tool/ Python library | Free | CNN | Programming library | High | 146 h, 41 min | 67,68 |
| Apple CreateML | Local tool/ part of Xcode | Free | Vectorized image | Graphical UI | Low | 26 min | 0,43 |
CNN = convolutional neural network; UI = user interface.
Model performance was evaluated using an independent test dataset after exclusion of unreadable images. Model outputs were compared against the reference standard established by consensus grading of 3 human experts, as well as against the IDx-DR.
Decision models were trained to perform 5-class classification in accordance with the ICDR severity scale. Model predictions were generated as 5-class probability distributions for each image and subsequently transformed into binary screening outcomes representing the likelihood of disease presence. This probabilistic framework enabled evaluation across multiple clinically relevant screening endpoints, including any DR, RDR, and sight-threatening diabetic retinopathy (STDR). To account for variability in model calibration and decision stability, 3 probability thresholds (20%, 50%, and 70%) were applied to define positive screening results.
Primary performance measures included sensitivity, specificity, PPV, NPV, overall accuracy, F1-score, and area under the receiver operating characteristic curve (AUC), each reported with 95% CIs (CIs). CIs were estimated using bootstrap resampling with 1000 iterations. Inter-rater and model agreement were further assessed using Cohen’s kappa and quadratic weighted kappa coefficients. Receiver operating characteristic (ROC) curves were generated for all models, and pairwise comparisons between model performances of paired binary classification results in the standard 50% decision threshold were conducted using McNemar’s test.
Statistical significance for differences in discriminative performance was assessed using a bootstrap method (n = 1000 iterations) to cf correlated ROC curves and calculate 95% CIs for AUC differences. This analysis was restricted to the standard and most general 50% decision threshold to evaluate model performance under default conditions. To account for the inflation of Type I error due to multiple comparisons, a Bonferroni correction was applied to all pairwise tests (both AUC and McNemar). With 15 pairwise comparisons, the adjusted significance threshold was set at P <.0033 (0.05/15).
MODEL OUTPUT AND BINARIZATION
The AutoML platforms were evaluated in their native state without post hoc calibration to reflect their “out-of-the-box” performance. Each model generated a probability distribution across 5 severity levels based on the ICDR scale. These multiclass distributions were binarized for screening endpoints by aggregating the probabilities. Specifically, for Any DR, probabilities for grades 1 to 4 were summed against grade 0; for RDR, grades 2 to 4 against 0 to 1; and for STDR, grades 3 to 4 against 0 to 2. This approach follows standard machine learning practices where models approximate probability distributions of the target classes.
DECISION THRESHOLDS
Three decision thresholds were selected to evaluate model performance: 20%, 50%, and 70%. The 20% threshold was chosen to simulate a high-sensitivity screening scenario where minimizing missed cases (false negatives) is a priority. The 70% threshold represented a high-specificity approach aimed at reducing false alarms. The 50% threshold served as the default, balanced operational point and was the primary threshold used for formulating clinical conclusions and statistical comparisons. The 20% and 70% thresholds were intended for illustrative purposes to demonstrate how model performance characteristics shift depending on the selected decision point.
STUDY ENDPOINTS
To evaluate and cf the performance of the AutoML platforms, we defined primary and secondary endpoints. The primary endpoints were chosen based on their clinical relevance to DR screening: (1) AUC, representing the overall discriminative capacity of the models, and (2) sensitivity for RDR at a standard 50% decision threshold, as high sensitivity is critical for preventing missed cases in screening programs. Secondary endpoints included specificity, PPV, NPV, accuracy, and F1-score, providing a comprehensive assessment of the models’ diagnostic accuracy and reliability.
MODEL EXPLANATION
To enhance model interpretability and provide insight into the spatial features underlying classification decisions, we applied Gradient-weighted Class Activation Mapping (Grad-CAM). Grad-CAM analysis was restricted to models that satisfied 2 predefined criteria: (1) availability in a locally runnable format and (2) implementation as convolutional neural network (CNN)—based architectures operating directly on input image data. Among the evaluated AutoML platforms, only AutoKeras and AutoGluon met these requirements and were therefore included in the explainability analysis. Models trained using Apple Create ML were excluded, as this framework relies on vectorized image embeddings rather than end-to-end convolutional feature extraction.
For qualitative assessment, 4 fundus photographs exhibiting at least any-grade DR were randomly selected and subjected to Grad-CAM visualization using both eligible models.
For the AutoGluon-based model, which employed the CaiT-derived CaFormer-B36 architecture, Grad-CAM was applied to the downsampling convolution layer at the fourth stage of the network, corresponding to the Conv2D projection layer responsible for stage transitions within the hierarchical transformer structure. In contrast, for the AutoKeras-based model utilizing the EfficientNetB7 architecture, Grad-CAM was generated from the final convolutional feature extraction block, specifically the top_conv layer, a depthwise separable convolution preceding the global average pooling operation.
RESULTS
The evaluation focused primarily on the predefined primary endpoints: AUC and sensitivity for RDR detection. Secondary measures (specificity, PPV, NPV, accuracy, and F1-score) were analyzed to provide additional context regarding the models’ operational characteristics at the default threshold.
PATIENT DEMOGRAPHICS
The final study cohort comprised 726 patients. The mean age was 56.3 ± 15.5 years, with a median age of 60.5 years (range, 18-90), and included 50.3% male and 49.7% female participants. Age distribution was as follows: <40 years, 17.7%; 40 to 49 years, 8.8%; 50 to 59 years, 19.7%; 60 to 69 years, 35.0%; and ≥70 years, 18.8%.
OVERALL MODEL PERFORMANCE
Model performance was evaluated for 3 clinically relevant screening endpoints—any diabetic retinopathy (Any DR), RDR, and STDR—using probability thresholds of 20%, 50%, and 70%. In this study, no additional calibration was performed on the probability outputs. This decision was intentional, as the primary objective was to cf the raw, inherent capabilities of different AutoML platforms. This provides a realistic assessment of how these tools would perform for users who may not have the expertise to perform complex post-training calibration.
At the predefined 50% decision threshold, performance measures for all AutoML models are summarized in Table 2 . When human expert grading was used as the reference standard, AutoGluon and Amazon SageMaker Canvas consistently achieved the most balanced performance across screening tasks. For Any DR detection, AutoGluon reached a sensitivity of 95.9% with an accuracy of 63.1%, whereas Canvas demonstrated a more balanced profile with 83.3% sensitivity, 82.1% specificity, and an overall accuracy of 82.4%. For RDR, Canvas achieved a sensitivity of 88.3%, specificity of 85.5%, and accuracy of 86.0%, while AutoGluon showed higher sensitivity (95.0%) at the expense of lower specificity (67.1%). AutoKeras demonstrated comparable sensitivity for Any DR (87.8%) and RDR (85.0%) but slightly reduced specificity (62.7% and 66.0%, respectively). In contrast, Create ML exhibited markedly lower sensitivity across all endpoints (Any DR: 24.4%; RDR: 27.5%) despite maintaining high specificity (>96%).
TABLE 2
Statistical Parameters of Diabetic Retinopathy Detection With 50% Decision Threshold
| Human | IDx-DR | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Sensitivity | Specificity | PPV | NPV | Accuracy | F1 | Sensitivity | Specificity | PPV | NPV | Accuracy | F1 | ||
| Any DR | Autoguon | 95.94 (92.12, 98.22) | 51.04 (46.6, 55.28) | 42.09 (37.35, 46.69) | 97.13 (94.41, 98.75) | 63.14 (59.46, 66.61) | 58.39 (53.61, 63.16) | 77.10 (73.06, 80.71) | 69.87 (63.48, 75.51) | 83.96 (80.19, 87.21) | 59.86 (53.69, 65.53) | 74.69 (71.33, 77.78) | 80.34 (77.19, 83.49) |
| Autokeras | 87.82 (82.33, 91.99) | 62.71 (58.37, 66.77) | 46.63 (41.32, 51.71) | 93.28 (90.13, 95.63) | 69.46 (65.93, 72.76) | 60.78 (55.29, 66.26) | 61.96 (57.41, 66.21) | 71.55 (65.24, 77.08) | 81.67 (77.29, 85.44) | 47.90 (42.46, 53.08) | 65.06 (61.42, 68.48) | 70.4 (66.24, 74.55) | |
| Canvas | 83.25 (77.18, 88.12) | 82.11 (78.54, 85.25) | 63.32 (56.97, 69.08) | 92.96 (90.24, 95.1) | 82.39 (79.4, 85.07) | 71.81 (66.29, 77.32) | 49.08 (44.46, 53.51) | 92.05 (87.81, 95.13) | 92.66 (88.74, 95.51) | 46.91 (42.2, 51.43) | 63.14 (59.46, 66.61) | 64.08 (59.69, 68.46) | |
| CreateML | 24.37 (18.18, 30.58) | 96.61 (94.69, 97.97) | 72.73 (59.81, 82.69) | 77.49 (74.08, 80.59) | 77.03 (73.76, 80.01) | 36.02 (25.13, 46.91) | 12.88 (9.88, 15.99) | 98.74 (96.36, 99.74) | 95.45 (87.1, 99.04) | 35.65 (31.9, 39.34) | 40.99 (37.31, 44.59) | 22.42 (17.47, 27.37) | |
| Rekognition | 100 (98.14, 100) | 0.19 (0, 0.69) | 27.1 (23.8, 30.38) | 100 (100, 100) | 27.1 (23.8, 30.38) | 42.52 (38.51, 46.52) | 100 (99.25, 100) | 0.42 (0, 1.54) | 67.26 (63.67, 70.63) | 100 (100, 100) | 67.26 (63.67, 70.63) | 80.4 (77.95, 82.84) | |
| VertexAI | 28.43 (21.89, 34.91) | 82.49 (78.94, 85.6) | 37.58 (29.37, 45.48) | 75.65 (71.89, 79.05) | 67.81 (64.23, 71.16) | 31.98 (16.93, 47.02) | 26.79 (22.77, 30.8) | 92.47 (88.31, 95.46) | 87.92 (81.46, 92.63) | 38.17 (34.09, 42.16) | 48.28 (44.52, 51.91) | 40.88 (35.2, 46.56) | |
| RDR | Autoguon | 95.0 (89.35, 98.13) | 67.11 (63.15, 70.78) | 36.31 (30.78, 41.69) | 98.55 (96.86, 99.47) | 71.66 (68.19, 74.88) | 52.31 (46.2, 58.43) | 76.33 (71.02, 80.96) | 80.14 (75.99, 83.78) | 72.93 (67.56, 77.69) | 82.85 (78.82, 86.32) | 78.54 (75.34, 81.45) | 74.51 (69.88, 79.14) |
| Autokeras | 85.0 (77.15, 90.78) | 65.95 (61.97, 69.67) | 33.01 (27.58, 38.34) | 95.7 (93.28, 97.43) | 69.05 (65.5, 72.36) | 47.31 (40.19, 54.42) | 64.33 (58.49, 69.65) | 72.9 (68.35, 77.0) | 62.46 (56.67, 67.77) | 74.46 (69.93, 78.52) | 69.33 (65.79, 72.62) | 63.26 (57.2, 69.32) | |
| Canvas | 88.33 (81.05, 93.42) | 85.53 (82.45, 88.21) | 54.64 (47.1, 61.57) | 97.38 (95.63, 98.56) | 85.97 (83.21, 88.4) | 67.31 (60.47, 74.14) | 57.67 (51.7, 63.2) | 95.09 (92.58, 96.93) | 89.18 (83.85, 93.14) | 76.22 (72.32, 79.73) | 79.64 (76.5, 82.49) | 69.92 (64.71, 75.12) | |
| CreateML | 27.5 (19.18, 35.79) | 97.04 (95.35, 98.23) | 64.71 (49.19, 77.08) | 87.15 (84.37, 89.56) | 85.56 (82.77, 88.02) | 37.87 (22.79, 52.95) | 16.33 (12.08, 20.71) | 99.53 (98.32, 99.94) | 96.08 (86.29, 99.51) | 62.92 (59.1, 66.52) | 65.2 (61.56, 68.62) | 27.51 (20.99, 34.02) | |
| Rekognition | 100 (96.95, 100) | 10.86 (8.36, 13.44) | 18.13 (15.15, 21.15) | 100 (94.48, 100) | 25.45 (22.22, 28.67) | 30.51 (26.31, 34.72) | 97.67 (95.24, 99.05) | 13.79 (10.48, 17.2) | 44.26 (40.35, 48.06) | 89.39 (79.06, 95.56) | 48.28 (44.52, 51.91) | 60.83 (57.08, 64.59) | |
| VertexAI | 0.83 (0, 3.05) | 100 (99.39, 100) | 100 (100, 100) | 83.63 (80.71, 86.23) | 83.63 (80.71, 86.23) | — | 0.33 (0, 1.23) | 100 (99.14, 100) | 100 (100, 100) | 58.87 (55.14, 62.42) | 58.87 (55.14, 62.42) | — | |
| STDR | Autoguon | 8.0 (0.11, 21.12) | 98.29 (97.03, 99.11) | 14.29 (0.19, 36.03) | 96.78 (95.2, 97.94) | 95.19 (93.36, 96.62) | — | 4.62 (1.27, 8.81) | 98.66 (97.38, 99.42) | 42.86 (13.86, 68.42) | 82.63 (79.62, 85.32) | 81.84 (78.82, 84.56) | — |
| Autokeras | 16.0 (2.66, 32.36) | 99.15 (98.15, 99.69) | 40.0 (7.49, 70.07) | 97.08 (95.56, 98.18) | 96.29 (94.63, 97.54) | — | 6.15 (2.21, 10.86) | 99.67 (98.8, 99.96) | 80.0 (39.99, 97.19) | 83.01 (80.03, 85.66) | 82.94 (79.98, 85.59) | 10.14 (0.38, 19.91) | |
| Canvas | 24.0 (7.13, 42.15) | 99.15 (98.15, 99.69) | 50.0 (16.75, 76.62) | 97.35 (95.88, 98.39) | 96.56 (94.96, 97.76) | — | 6.15 (2.21, 10.86) | 99.33 (98.29, 99.82) | 66.67 (30.79, 89.07) | 82.96 (79.98, 85.62) | 82.67 (79.69, 85.33) | — | |
| CreateML | 8.0 (0.11, 21.12) | 99.15 (98.15, 99.69) | 25.0 (0.36, 57.87) | 96.81 (95.24, 97.96) | 96.01 (94.31, 97.31) | — | 4.62 (1.27, 8.81) | 99.67 (98.8, 99.96) | 75.0 (29.04, 96.33) | 82.78 (79.79, 85.45) | 82.67 (79.69, 85.33) | — | |
| Rekognition | 96.0 (78.88, 99.89) | 84.21 (81.28, 86.81) | 17.78 (11.2, 24.63) | 99.83 (99.06, 100.0) | 84.59 (81.74, 87.12) | 29.11 (19.29, 38.94) | 65.38 (56.23, 73.29) | 91.64 (89.11, 93.72) | 62.96 (53.92, 70.88) | 92.41 (89.96, 94.4) | 86.93 (84.24, 89.28) | 63.88 (54.8, 72.96) | |
| VertexAI | 4.0 (0, 14.25) | 100 (99.48, 100) | 100 (100, 100) | 96.7 (95.12, 97.87) | 96.7 (95.12, 97.87) | — | 0.77 (0, 2.82) | 100 (99.38, 100) | 100 (100, 100) | 82.26 (79.25, 84.95) | 82.26 (79.25, 84.95) | — | |
Stay updated, free articles. Join our Telegram channel
Full access? Get Clinical Tree
