Assessing a large language model for consistent and discriminatory performance in wound care bioengineering research and development tasks

Despite the growing adoption of large language models (LLMs) in healthcare-related contexts, their integration into applied academic and industry research workflows remains underexplored, particularly with respect to reproducibility and semantic stability of their outputs.

To evaluate the semantic consistency, discriminatory performance, and response stability of ChatGPT-4 (GUI-accessed) in supporting biomedical and bioengineering research tasks, using wound-care research and development as an early test case, with an exploratory GPT-4o (API-accessed) extension screening output stability under varied hyperparameters.

Two structured 30-query datasets, comprising wound-related queries, semantically equivalent paraphrases, and unrelated controls, were evaluated using GPT-4 via a graphical user interface (GUI). Four natural language processing text representation methods (TRMs): TF-IDF, Word2Vec, Sent2Vec, and Bag of Words, were used to compute similarity scores between outputs. Additional evaluations included device- and time-dependent variation testing and an exploratory case study on wound dressing stiffness using API (OpenAI) across parameter-space screening of varied hyperparameters.

Sent2Vec demonstrated the highest discriminatory power, with similarity-score differentials of 0.49 and 0.50 in the LLM-generated and expert-generated query phases, respectively. ChatGPT-4 (GUI) maintained semantic consistency across reworded queries and under different usage conditions. In the exploratory case study, GPT-4o (API)-generated stiffness values remained relatively stable across parameter settings. The resulting mean stiffness values ranged from 25.3 to 36.5 kPa and fell within the approximately 15-90 kPa range reported for hydrogel-based dressing materials in the cited biomechanical literature.

ChatGPT, evaluated as GPT-4 (GUI) for semantic consistency and GPT-4o (API) for the stiffness screening, offers preliminary evidence of semantic consistency and stable output behavior when responding to wound care research and development queries. These findings are exploratory and should be interpreted strictly as preliminary hypothesis generation and textual exploration, not as evidence of scientific reliability, predictive validity, or decision-support capability. This study presents a novel and pilot framework for characterizing LLM output consistency and stability in specialized biomedical contexts, intended to inform the design of future, more comprehensive validation studies.