A computational validation for the health concept maturity levels questionnaire

The healthcare market is undergoing rapid transformation, requiring the integration of user needs from the earliest stages of product and service design. Living Labs are emerging as a model for the co-creation and evaluation of user-centered innovations. In this work, we developed a Health Concept Maturity Levels grid and questionnaire to assess the maturity of health concepts.

The research process included multiple stages, starting with the creation of the Association Innov'Autonomie - Health Concept Maturity Levels Questionnaire - 178-items (CMLH questionnaire), designed to evaluate health concept maturity levels. Speech acts from Health Concept Maturity Levels expert interventions were then annotated and used as data for our machine learning and deep learning models. We used the CatBoost algorithm in the first experiment to discern individual Health Concept Maturity Levels factors from speech acts to generate factor probabilities used to feed a neural network trained to take the final decision, to evaluate whether the network could accurately identify the membership factors of Health Concept Maturity Levels criteria when presented with items from the CMLH questionnaire, thus establishing computational semantic validity.

The results of the study indicate that only the models trained with the true factors are able to correctly identify the corresponding factor in the sequentially encoded texts, with the exception of the Need domain's sensitivity metric, which showed artefactual performance. The general performance of the different CatBoost algorithms used to predict one factor versus the other two showed similar performance. For the questionnaire, the models trained with the real factors also showed better performance in identifying the matching factors compared to the random factors. A marginal difference was observed between the "Need" and "Technology" factors.

This study introduces computational semantic validity as a novel complementary approach to traditional psychometric validation, providing evidence that supports both convergent and content validity for the CMLH questionnaire. This computational method demonstrates semantic alignment between expert discourse and questionnaire structure through machine learning and deep learning techniques. However, overlaps between "Programmatic" and "Need" factors indicate a need for improvement in the Concept Maturity Levels Health model. Future work will focus on enhancing these models and investigating their potential application as a complementary validation method for other psychometric tools.