The AI in Health and Care Award, part of the NHS AI Lab plan, was launched to speed up the development and adoption of AI technologies in healthcare
From 2020 to 2024, the award allocated over £100 million to support various AI projects across four phases, each reflecting different stages of real-world implementation.
AI in Health and Care Award
The final stage, Phase 4, focused on the multi-site deployment and evaluation of AI technologies. This NHS released a document reviewing the lessons learned from the evaluations of 13 Phase 4 technologies, showing practical insights on designing and conducting evaluations of AI in healthcare.
AI evaluations
The evaluations followed a structured approach guided by NICE’s MedTech Early Technical Assessments (META). These assessments identified clear gaps, which helped shape the design of independent evaluations, focusing on eight key domains: safety, accuracy, effectiveness, value, fit with sites, implementation, scalability, and sustainability.
Evaluators also examined how patient characteristics influenced the outcomes across these domains to ensure no group was left behind.
One important takeaway from the evaluation process was the importance of co-producing deployment and evaluation plans. This involves collaboration between technology suppliers, independent evaluators, and adopting sites, including clinical and patient users. This helps AI technologies to specific real-world settings, ensuring better fit and smoother implementation.
Another understanding was the need to allow sufficient time, at least two years, for evaluating AI deployments across multiple sites. The complexity of real-world environments, including variations in IT infrastructure and clinical practices, requires longer timelines to capture meaningful results.
Mixed-method evaluation approaches
Future national AI programmes should also prioritise mixed-method evaluation designs, which combine quantitative and qualitative approaches, to provide a more comprehensive assessment of AI impacts.
The evaluations show the importance of focusing explicitly on health inequalities. Including this as a reliable evaluation domain would help ensure AI technologies do not unintentionally widen existing differences in healthcare access or outcomes.
The rapid pace of technological change means that teams need up-to-date national guidance and resources. Dissemination of findings, including interim reports and various forms of communication such as webinars and academic submissions, was vital to ensure that lessons learned from these evaluations could inform future decisions on AI in healthcare.
