Canada’s Drug Agency issues guidance on AI-generated real-world evidence for HTA

Canada’s Drug Agency has issued guidance on reporting and appraising real-world evidence generated using AI methods to extract variables from unstructured health data for health technology assessment.

The Baseline

• Canada’s Drug Agency has issued guidance on reporting AI methods used to extract variables from unstructured health data to generate real-world evidence for health technology assessment.
• The guidance sets expectations for human oversight, transparent reporting, validation, and accountability when AI is used to extract information from unstructured health data.
• It also addresses risks related to areas such as bias and reproducibility, the potential impact of extraction errors, and higher-risk applications such as causal inference.

Canada’s Drug Agency (CDA-AMC) has published new guidance setting out its expectations for how AI methods should be reported when used to generate real-world evidence (RWE) submitted for health technology assessment (HTA). The document, Use of Artificial Intelligence Methods to Generate Real World Evidence Submitted to Canada’s Drug Agency for HTA, is intended for those who generate and submit evidence to CDA-AMC, as well as those involved in reviewing and appraising submissions.

The guidance focuses specifically on AI methods used to convert information contained in unstructured health data into structured variables for RWE studies. Unstructured data can include free-text clinical narratives, laboratory information, descriptions of imaging and other diagnostic tests, and image-based documents within electronic health records. Methods covered include large language models (LLMs), machine learning (ML), and rule-based natural language processing (NLP).

CDA-AMC states that established and explainable non-AI approaches, including human abstraction, should remain the primary approach where they can produce robust results. AI may be used in a supplementary role where its use is justified, and associated risks have been identified and mitigated.

The guidance also sets expectations for human involvement. Drawing on guidance from the UK’s National Institute for Health and Care Excellence (NICE), CDA-AMC affirms NICE’s position that:

“Any use of AI methods should be based on the principle of augmentation, not replacement, of human involvement.”

CDA-AMC states that this should involve active oversight throughout evidence generation rather than “cursory sign off” at the end of the process, with those generating and submitting evidence remaining accountable for their submissions.

The document adapts NICE guidance for the Canadian HTA context and follows CDA-AMC’s 2025 position statement on AI. Governance considerations include justification for using AI, human oversight and validation, ethical principles, cybersecurity, intellectual property, and compliance with applicable legal, regulatory, technical, and scientific requirements. Risks related to areas such as bias, transparency, and reproducibility should be reported alongside measures taken to address them.

CDA-AMC also sets out information expected when reporting AI-enabled data extraction and validation, including data preprocessing, the model and version used, extraction and adjudication procedures, training and test datasets, annotator expertise, and performance metrics. For LLMs, reporting should also cover areas such as model configuration, fine-tuning, parameter selection, and prompting strategies.

As part of validation, evidence developers should describe differences between training or test data and the analytical dataset that could affect generalizability. Performance should be assessed using appropriate metrics and across relevant subpopulations, alongside consideration of how extraction errors could affect the intended analysis.

The guidance identifies AI methods used to estimate comparative treatment effects as a higher-risk application. CDA-AMC states that AI-based causal inference should be accompanied by sensitivity analyses, validation against suitable alternative methods, and triangulation with available clinical evidence.

CDA-AMC states that “transparent reporting of AI methods is essential to ensure credibility.” The guidance will be reviewed as methodological developments emerge or at regular intervals, with AI-related updates potentially incorporated into its Methods Guide for Health Technology Assessment.