Patient- and Caregiver-Informed Considerations for the Design and Implementation of Generative AI-Supported...

Generative artificial intelligence (AI) has the potential to impact health care by transforming workflows and improving outcomes. Patient-centered clinical decision support (PC CDS) are digital tools that use patient-specific information and patient-centered outcomes research to improve health care decision-making. Generative AI is increasingly being incorporated into PC CDS tools. As patient-facing digital tools continue to expand within the health ecosystem, it is important to gather patient and caregiver perspectives about engaging with generative AI-supported PC CDS tools.

This study aimed to generate a prioritized list of patient- and caregiver-informed considerations for the design, implementation, and use of generative AI in PC CDS.

We conducted 6 small group discussions across 2 phases, with a total of 16 participants comprising patient and caregiver advocates. The first phase explored perspectives on generative AI-supported PC CDS. Using an iterative qualitative approach, we synthesized themes after each session to monitor saturation, which informed the development of an initial list of 7 key considerations for the implementation and use of generative AI-supported PC CDS. During the second phase, we generated a refined list of considerations using a prioritization ranking activity and peer validation approach.

Participants believed that generative AI-supported PC CDS tools have the potential to enhance efficiency, support clinicians, and improve health care decision-making but recognized that they could introduce challenges. Trust and willingness to use these tools are shaped by individuals' health care experiences and familiarity with technology. Concerns included transparency, data security and accuracy, and potential for bias and mistrust in health care. Participants emphasized the importance of customizability and seamless integration into patient-clinician interactions. The tools' success depends on clinicians' skills and use. Our final list of 7 considerations includes the development of standards and design principles for generative AI-supported PC CDS tools, co-design with end users, considerations for mistrust in the health care system, monitoring and evaluation to ensure accuracy, education and training to understand and use AI, use of generative AI-supported tools that complement clinicians' work and uphold the patient-clinician relationship, and AI that holistically uses patient data and tailors outputs.

This study offers a patient- and caregiver-informed set of considerations for generative AI-supported PC CDS including holistic data use, continuous monitoring, addressing mistrust, and ensuring human oversight to mitigate risks such as errors in AI outputs. Participants emphasized the importance of co-design and the need for education, training, and user choice to support meaningful engagement. These interconnected considerations can inform future research and guide the design and implementation of patient-facing AI-supported PC CDS tools.