Evaluating Microsoft Copilot in Qualitative Health Research: Accurate for Manifest Content Coding but Limited in...

Introduction Artificial intelligence (AI) is becoming more integrated in different research assignments, and this ongoing development opens opportunities to optimize resources, e.g., using AI in resource-intensive and time-consuming tasks like qualitative analysis of interview data. We aimed to test if Microsoft's Copilot could perform a content analysis on interview data using Graneheim and Lundman's method comparable to human analysis. Methodology We used a company-protected version of Microsoft's AI-powered assistant Copilot, which is based on large language models. The company-protected Copilot version ensured data security. A manual analysis of six interviews was conducted before this study using Graneheim and Lundman's method of content analysis. We conducted four analyses using Copilot and compared the results with those obtained through manual analysis. Copilot was prompted to use Graneheim and Lundman's method, and we tried providing it with an objective and a context. Results When prompted to use Graneheim and Lundman's method, Copilot was able to perform content analyses with high resemblance to the manual one, especially in terms of selecting meaningful units, as well as when coding them, which is within the descriptive analysis. It could also create subthemes and overarching themes resembling the manual ones; however, the interpretive analysis lacked nuances compared to the manual one. Copilot produced more accurate manifest content when only given Graneheim and Lundman's method. When given the objective, the analysis was shorter with fewer meaningful units. When given the context of the interviews, Copilot over-interpreted, and the analysis was mainly descriptive. Conclusions Copilot was able to perform a content analysis very similar to the manual one regarding the descriptive analyses on the manifest content using Graneheim and Lundman's method. However, it's interpretation of latent content lacked nuance - a limitation Copilot itself acknowledged. Copilot performed best when guided by the methodological framework alone, rather than the study's objective or context. While content analysis remains, a co-creative process requiring manual input, especially during interpretation, Copilot shows promising potential in supporting the early stages of analysis focused on manifest content.