Related Experiment Video
Updated: Jan 25, 2026

Setting Up a Stroke Team Algorithm and Conducting Simulation-based Training in the Emergency Department - A Practical Guide
Published on: January 15, 2017
The role of public challenges and data sets towards algorithm development, trust, and use in clinical practice
Veronica Rotemberg1, Allan Halpern1, Steven Dusza1
1Dermatology Service, Memorial Sloan-Kettering Cancer Center, New York, NY.
This article reviews how public data sets and open challenges help improve artificial intelligence tools for analyzing skin images. It explores why these tools are difficult to implement in clinics and how shared resources can help overcome these barriers.
Area of Science:
- Medical informatics and algorithm development within clinical practice
- Artificial intelligence and machine learning in dermatology
Background:
No prior work had resolved the specific barriers preventing the widespread adoption of artificial intelligence in dermatology. While machine learning has achieved human-level performance in many domains, medical imaging remains a complex challenge. Skin imaging suffers from a lack of standardized hardware, lighting, and capture techniques. Unlike radiology, skin appearance varies significantly based on patient skin tone and a vast array of potential diseases. Automated systems require massive, high-quality, annotated data sets to account for this inherent clinical diversity. This gap motivated researchers to investigate how public challenges might bridge the divide between laboratory performance and clinical utility. Integrating these technologies into existing hospital workflows presents unique operational hurdles that remain poorly understood. That uncertainty drove the need for a comprehensive review of how shared data resources can catalyze progress in this field.
Purpose Of The Study:
The aim of this article is to summarize recent advancements in machine learning with a specific focus on skin imaging. The authors seek to clarify the role of public challenges and data sets in the progression of these diagnostic technologies. This study addresses the difficulty of using automated systems to improve sensitivity and specificity in global healthcare networks. Researchers explore why skin imaging presents unique complexities compared to other medical fields like radiology. The motivation for this work is the persistent gap between laboratory performance and real-world clinical utility. By highlighting remaining hurdles, the authors aim to provide a roadmap for more effective implementation of these tools. This review discusses how collaborative data initiatives can catalyze the development of robust, reliable solutions. The study ultimately provides a perspective on how to foster trust and facilitate the use of artificial intelligence in daily clinical practice.
Main Methods:
The review approach involved synthesizing recent advancements in machine learning as applied to medical diagnostics. Researchers examined the impact of public challenges on the progression of automated pattern analysis technologies. The authors conducted a focused analysis of how large, annotated data sets influence algorithm development. This study evaluated the specific complexities associated with integrating new software into existing clinical workflows. Reviewers assessed the literature to identify remaining hurdles preventing effective implementation in healthcare networks. The methodology prioritized studies that addressed the unique challenges of skin-based diagnostic imaging. Investigators compared the performance of various models against the requirements for clinical-grade sensitivity and specificity. This systematic evaluation provided a clear perspective on how collaborative data initiatives catalyze solutions for complex medical problems.
Main Results:
Key findings from the literature indicate that machine learning has achieved or exceeded human-level performance in domains like speech processing and image recognition. The authors report that skin imaging remains the most difficult subdomain due to significant variability in hardware and lighting. Results show that automated algorithm development relies heavily on the availability of high-quality, annotated data sets. The literature suggests that current systems struggle to maintain high sensitivity and specificity when deployed in diverse global healthcare networks. Findings highlight that skin appearance is uniquely affected by patient skin tone and a wide range of pathological conditions. The review demonstrates that public challenges are effective at gathering the necessary breadth of diagnostic variety for training robust models. Evidence shows that these collaborative efforts help address the lack of standardization in imaging techniques. The authors conclude that these data-driven approaches are essential for overcoming the current limitations in clinical skin diagnostics.
Conclusions:
The authors propose that public challenges serve as a primary mechanism for accelerating the development of robust diagnostic algorithms. These initiatives provide the necessary scale and diversity of annotated data to improve model performance. Synthesis of the literature indicates that standardized data sets help mitigate issues related to imaging variability and skin tone differences. The researchers suggest that these collaborative efforts are essential for building trust among clinicians. Future progress depends on the successful integration of these tools into existing medical workflows. The authors emphasize that overcoming implementation hurdles requires ongoing engagement between data scientists and healthcare providers. These findings imply that shared resources can effectively address the current limitations in skin diagnostic sensitivity and specificity. The review concludes that public data initiatives are a viable pathway toward more reliable clinical artificial intelligence applications.
Frequently Asked Questions
The researchers propose that public challenges catalyze progress by providing large, diverse, annotated data sets. These resources help algorithms overcome variability in skin tone, lighting, and imaging hardware, which are otherwise major obstacles to achieving high diagnostic sensitivity and specificity in clinical settings.
The authors identify the lack of standardized imaging hardware, capture techniques, and lighting conditions as major hurdles. Furthermore, the broad range of skin diseases and variations in patient skin tone complicate the creation of reliable, universally applicable automated diagnostic systems.
The researchers argue that integration is necessary because clinical workflows are complex and differ from controlled laboratory environments. Without addressing these operational realities, even highly accurate algorithms may fail to improve patient outcomes or gain trust within healthcare networks.
The authors highlight that annotated image data sets act as the foundation for training automated systems. These collections must incorporate the breadth of circumstantial and diagnostic variety to ensure that algorithms perform reliably across diverse patient populations and different clinical settings.
The researchers analyze the phenomenon of performance gaps between laboratory settings and real-world clinical practice. They note that while accuracy often reaches human levels in controlled tests, these systems struggle to maintain that performance when applied to the messy, non-standardized data found in actual hospitals.
The authors suggest that public challenges and shared data sets are key to building clinician trust. By fostering transparency and rigorous testing, these initiatives help bridge the gap between experimental technology and practical, reliable tools that can be safely used in patient care.
Related Concept Videos
Design Example: Setting a Curve Using Design Data
Statistical Software for Data Analysis and Clinical Trials
Trial and Error and Algorithm
Sustainable Development
Characteristics of Practical Op Amps
The ratio of differential gain to the common-mode gain is defined as the common-mode rejection ratio (CMRR). This ratio quantifies the ability of operational amplifiers (op-amps) to reject common-mode...
Theoretical Foundations of Nursing Practice
Theories provide a perspective to assess patients' conditions and organize data and methods. They also assist in analyzing and interpreting information. They represent a...

