The application of generative AI to healthcare regulation

The article investigates the application of generative artificial intelligence (GenAI), specifically Microsoft Copilot 365, in automating the drafting of regulatory reports for a national healthcare regulator. The initiative aimed to assess the feasibility and effectiveness of AI technologies in reducing manual effort, maintaining quality and accuracy of report and enhancing operational efficiency in the regulatory reporting processes.

A proof of concept (PoC) was conducted to evaluate Microsoft Copilot 365's capabilities in generating high-quality draft reports. Sample data were extracted and structured from a data repository to train AI prompts. Test cases were selected to simulate real-world scenarios. AI-generated documents were compared with human-authored reports. Accuracy, completeness and overall quality was compared using standardised BERTScore and ROUGE-L metrics.

AI-generated reports closely matched human-authored documents in terms of accuracy, completeness and quality. BERTScore and ROUGE-L metrics indicated high alignment, with Copilot outputs showing strong consistency across various regulations. Feedback from stakeholders highlighted the AI's performance and its potential impact on time reduction.

The successful implementation of this PoC underscores the transformative potential of AI technologies in regulatory environments. Automating routine tasks may allow regulators to reallocate human resources to onsite inspection activities, enhancing overall operational effectiveness. The findings support the integration of AI in regulatory reporting, paving the way for future advancements in this domain.

This is the first paper to our knowledge that investigates the application of GenAI, specifically Microsoft Copilot 365, in automating the drafting of regulatory reports for a healthcare regulator.