The Social Construction of Categorical Data: Mixed Methods Approach to Assessing Data Features in Publicly Available...

Area of Science:

• Machine Learning
• Data Science
• Medical Informatics

Background:

• Computer scientists use categorical data (e.g., gender, skin color) to enhance machine learning models in data-scarce fields like healthcare.
• The impact of categorical data on model accuracy and equity in diverse populations remains underexplored.

Purpose of the Study:

• To investigate the influence of categorical data on machine learning model performance.
• To analyze data collection and publication processes affecting categorical data utility.
• To propose a mixed-methods approach for evaluating categorical data before machine learning training.

Main Methods:

• A mixed-methods approach was developed, grounded in the social construction of categories.
• Quantitative analysis assessed the impact of including/excluding categorical features on a transformer-based model using a Brazilian dermatological dataset (PAD-UFES 20).
• Qualitative analysis involved interviews with dataset authors to understand data collection and publication rationales.

Main Results:

• Quantitative analysis revealed inconsistent effects of categorical data on model predictions across different classes.
• Qualitative insights explained observed quantitative effects by highlighting the social construction and context-dependency of data categories.
• Findings underscore limitations of using publicly available datasets in contexts different from their origin.

Conclusions:

• Caution is advised when using categorical data from public datasets without considering their social construction and context.
• A social scientific, context-dependent analysis using mixed methods is recommended to assess categorical data utility for intended populations.
• This approach aids in ensuring equitable machine learning model development, especially in data-scarce domains.