Who guides us on how to perform good RWE studies?

With the increasing use of RWE for decision making among various stakeholders, comes an increasing need for guidelines and good practice principles for the planning and execution of RWE studies and for dissemination of the data to ensure confidence in RWE. Learn more about existing guidelines in this article.

For RCTs, regulatory authorities and journal publishers have long established good procedural practices for the acceptance in regulatory decision making and the dissemination of results in medical journals, demanding  pre-specification of endpoints and analyses in the trial protocol, including a commitment to make all results publicly available.

For RWE, early guidelines on methodology and reporting of RWE studies were prepared; however, each these guidelines focused on single elements in  generating and publishing evidence (e.g. International Society for Health Economics and Outcomes Research (ISPOR) for methods on evidence generation and reporting, STROBE and ENCePP for reporting of results) Learn more here.

RWE is increasingly used to inform regulatory decision making on drug effectiveness and benefit-risk balance of a treatment. So far, the European Medicines Agency (EMA) and the US Food and Drug Administration (FDA) have primarily issued guidelines for conducting post-approval safety studies using real-world data or non-interventional studies. However, both agencies are expected to issue guidelines for RWE studies used to inform effectiveness of treatments in the near future. Learn more here.

In recent years, as the use of RWE has evolved significantly, there has been a call for action for more comprehensive guidelines and statements of good practice in RWE, to ensure quality and transparency in this field. As a result, a network initiated by ISPOR has published recommendations covering topics such as study registration, replicability, and stakeholder involvement in RWE studies. As part of these good practice recommendations, a Real-World Evidence Transparency Initiative has been created to establish a culture of transparency for study analysis and reporting of RWE studies on treatment effects. Recommendations and a roadmap on how to encourage public registration of observational studies/RWE were recently published. Learn more here.

While most guidelines inform the generation and reporting of evidence, only few are specifically designed to help decision makers evaluate published effectiveness research for its relevance and credibility. A network initiated by ISPOR has developed a questionnaire to help evaluate the relevance and credibility of research. Learn more here.

Subsequently, this questionnaire has been adopted in the field of Allergy and Asthma. The Respiratory Effectiveness Group (REG) and the European Academy of Allergy & Clinical Immunology (EAACI) established a joint task force to define a set of quality standards for RWE studies to inform clinical practice. The outcome is the Real Life Evidence AssessmeNt Tool (RELEVANT). The EAACI Methodology Committee plans to assess all observational research for quality and risk of bias. Learn more here.

The ISPOR Good Research Practices Taskforce have published their recommendations on good practices for comparative effectiveness research (i.e. RWE looking into the effectiveness rather than safety of a treatment) in 3 parts:

Part 1 focuses on defining, reporting and interpreting non-randomised studies. The report suggests that to optimize the validity of findings from observational studies designed to inform health-care policy decisions, researchers include a priori defined hypotheses in protocols and data analysis plans before study implementation, that they follow reporting standards that make transparent to readers if, why, and how their analysis plans evolved, as well as providing a justification of the suitability of the database to test their hypotheses. Find the publication here.

Part 2 provides recommendations and tools for researchers to mitigate threats to validity from bias and confounding in measurement of exposure and outcome. Recommendations for the design of studies included: the need for data analysis plan with causal diagrams; detailed attention to classification bias in definition of exposure and clinical outcome; careful and appropriate use of restriction; extreme care to identify and control for confounding factors, including time-dependent confounding. Find the publication here.

Part 3 provides recommendations on analytical techniques and best practices to produce valid findings of causal therapeutic benefits. Recommendations include using stratification analysis before multivariable modeling, multivariable regression including model performance and diagnostic testing, propensity scoring, instrumental variable, and structural modeling techniques including marginal structural models, where appropriate for secondary data. Find the publication here.

The European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP) has developed a checklist to stimulate consideration of important principles when designing and writing a pharmacoepidemiological or pharmacovigilance study protocol.

ENCePP Checklist for Study Protocols can be found here.

The CONSORT statement is a well-established checklist for the reporting of RCTs. The statement was expanded to include pragmatic trials. The CONSORT extension for pragmatic trials builds upon the existing CONSORT checklist and gives specific guidance for 8 of the 22 checklist items in relation to pragmatic trials. Find more information about the CONSORT statement and its extension for pragmatic trials here.

The STROBE (Strengthening the reporting of observational studies in epidemiology) statement established a checklist of items that should be included in articles reporting observational studies. It is an international, collaborative initiative of epidemiologists, methodologists, statisticians, researchers and journal editors involved in the conduct and dissemination of observational studies, to help readers understand what was planned (and what was not), what was done, what was found, and what the results of an observational study mean. Find more information and the STROBE checklist here.

As part of the 21^st Century Cures Act, FDAs RWE Program must evaluate the potential use of RWD to generate RWE of product effectiveness to help support approval of new indications for already approved drugs. FDA has established a framework for this evaluation and has published guidelines on the definition of RWD and RWE, as well as how to submit such data to FDA. Furthermore, as part of the program, FDA plans to issue guidance about observational study design, including reporting requirements for such studies. Learn more here.

EMA is also moving ahead with issuing guidelines for observational studies. Currently, guidelines for registry-based studies are in development. The public consultation for these guidelines ended in 2020 and the publication of these guidelines is expected during 2021. Learn more here.

The ISPOR/ISPE Task Force recommendations cover seven topics such as study registration, replicability, and stakeholder involvement in RWE studies. These recommendations, together with earlier recommendations about study methodology, provide a trustworthy foundation for the expanded use of RWE in health care decision making. Find the publication here.

The Real-World Evidence Transparency Initiative Partnership is a joint collaboration and ongoing effort between ISPOR, the International Society for Pharmacoepidemiology, the Duke-Margolis Center for Health Policy, and the National Pharmaceutical Council and including various other stakeholders. The objective of this initiative is to establish a culture of transparency for study analysis and reporting of hypothesis evaluating real-world evidence studies on treatment effects. A key aspect to improve credibility of RWE studies is that they are conducted transparently with tests that follow a prespecified analytic protocol. Preregistration of such study protocols on a public website would help build trust that their results can be used for decision-making purposes.

Together with experts from amongst others the European Medicines Agency (EMA), the National Institute for Health and Care Excellence (NICE), The National Pharmaceutical Council (NPC), The Dutch National Health Care Institute (ZIN) , and representatives from academia and industry, a plan to encourage routine registration of noninterventional real-world evidence studies used to evaluate treatment effects was published. Find the publication here.

While various grading systems and checklists exists for the evaluation of RCTs, there has so far only been few tools for the assessment of RWE. Recently, ISPOR has together with the Academy of Managed Care Pharmacy (AMCP) and the National Pharmaceutical Council (NPC) developed a questionnaire to help evaluate the relevance and credibility of prospective and retrospective observational research. Find the publication here.

This has subsequently been adopted by the Respiratory Effectiveness Group (REG). REG is a global think-tank within respiratory medicine research, which aim is to identify and fulfil the real-life research needs in respiratory medicine to inform clinical practice. Together with the European Academy of Allergy & Clinical Immunology (EAACI) they have developed the Real Life Evidence AssessmeNt Tool (RELEVANT). RELEVANT is an easy-to-use checklist that allows the assessment of the quality of published real life evidence. The tool should help inform the conduct and reporting of studies, as well as in the identification of high-quality RWE for decision making. The EAACI Methodology Committee plans to assess all observational research/RWE for quality and risk of bias.

The RELEVANT tool can be found here.

In these two publications you can read more on the development of the tool: Roche et al. (2019)   Campbell et al. (2019)