Dermacen analytica: A novel methodology integrating multi-modal large language models with machine learning in...

To design, implement, evaluate, and quantify a novel and adaptable Artificial Intelligence-empowered methodology aimed at supporting a dermatologist's workflow in assessing and diagnosing skin conditions, leveraging AI's deep image analytic power and reasoning. Skin presents diverse conditions that no single AI solution can comprehensively address, suggesting that mimicking a medical professional's diagnostic process and creating strategic AI interventions may enhance decision-making.

We employ large language, transformer-based vision models for image analysis, sophisticated machine learning tools for guideline-based segmentation, and measuring tasks in our system. As no single technology is sufficient on its own for efficient use by dermatologists, we apply a sequential logic with agency to improve outcomes.

Using natural language processing methods and incorporating human expert evaluation, our system achieved a weighted accuracy of 87% on the dataset used, demonstrating its reasoning and diagnostic capabilities.

This study serves as a proof of concept for the application of AI in dermatology, highlighting its potential to enhance the patient journey for which we approximate the value of such interventions in healthcare using graph theory with an associated cost-optimization objective function.