Related Experiment Video
Updated: Aug 14, 2025
04:54
Iterative Development of an Innovative Smartphone-Based Dietary Assessment Tool: Traqq
Published on: March 19, 2021
4.7K
Agreement Between Experts and an Untrained Crowd for Identifying Dermoscopic Features Using a Gamified App: Reader
Jonathan Kentley1,2, Jochen Weber2, Konstantinos Liopyris3
1Department of Dermatology, Chelsea and Westminster Hospital, London, United Kingdom.
JMIR Medical Informatics
|January 18, 2023
Summary
Crowdsourcing can reliably label dermoscopic structures in pigmented lesions, achieving agreement comparable to expert dermatologists. This scalable method aids in developing machine learning tools for skin lesion analysis.
Area of Science:
- Dermatology
- Medical Imaging
- Machine Learning
Background:
- Dermoscopy is crucial for pigmented lesion evaluation, but expert agreement on structures is often poor.
- Expert annotation of medical images is a bottleneck for machine learning (ML) development.
- Crowdsourcing offers a cost-effective and time-efficient alternative for medical image annotation.
Purpose of the Study:
- To demonstrate crowdsourcing's reliability in labeling basic dermoscopic structures.
- To compare the reliability of nonexpert crowd labeling with expert dermatologists.
- To validate crowdsourcing as a viable method for annotating dermoscopic images.
Main Methods:
- 248 melanocytic lesion images were labeled by 20 experts for 31 subfeatures, then collapsed into 6 superfeatures.
- A nonexpert crowd annotated the 6 superfeatures using the DiagnosUs platform.
- A group of 7 dermatologists repeated the annotation for direct comparison; Cohen κ measured interrater reliability.
Main Results:
- Crowd agreement varied by feature, with lower agreement for dots/globules and higher for network structures/vessels.
- Expert raters showed a similar pattern of agreement across features.
- Median κ values between the crowd and averaged experts were good to excellent for all 6 superfeatures.
Conclusions:
- Interrater reliability for dermoscopic features varies among experts and similarly in nonexpert crowds.
- Crowdsourcing demonstrates good to excellent agreement with expert annotations for dermoscopic images.
- Crowdsourcing is a feasible and dependable method for large-scale dermoscopic image annotation, supporting ML tool development.
