Part one of this two part blog series will help answer and demonstrate the need for credible real world data and how utilizing established frameworks can help to evaluate fit-for-purpose.
Real World Data: A powerful tool for informed decision making
Real world data (RWD) are data collected outside of clinical trials as part of routine clinical practice, such as data found in electronic medical records (EMRs), registries, insurance claims, and mobile devices. Real world evidence (RWE) is derived from RWD and can support informed decisions by patients, providers, regulators, and payers throughout the drug's lifecycle around the use, benefits, and risk of medical products and devices.
The adoption of RWD for regulatory and market access purposes has accelerated through the confluence of the following key healthcare trends.
First, RWD is becoming increasingly available for use, driven in part by the advent of new analytic techniques and the widespread, positive effects of the 21st Century Cures Act. New data access channels and RWD types are emerging, fueled by the adoption of healthcare interoperability standards and the growth of digital health and telemedicine, trends that have accelerated in response to the COVID-19 pandemic.
Second, clinical trial designs that incorporate RWE (e.g. external comparators and pragmatic trials) have gained increasing acceptance with regulators for initial approvals and label extensions.
Third, RWE is facilitating the shift from volume to value-based healthcare. Shifting reimbursem*nt schemes and outcomes-based contracting strategies rely on RWD to inform the actual use, outcomes, and value of products in the real world.
Data credibility frameworks – an introduction
Although data are becoming easier to access than in the past, all data sources are not equally suited or credible to answer specific research questions. The term “fit-for-purpose” is often used to describe a data source that can answer certain questions with accuracy and credibility. Assessing the fit of RWD sources is context dependent. A given dataset may be well suited to address a specific set of research questions, and poorly suited to answer another set. To determine if a data source is “fit-for-purpose,” existing data credibility frameworks that have emerged in the last few years can be leveraged.
The Grace Checklist pioneered the idea that RWD could be evaluated on the merits of how well data addresses a particular research question. In 2018, the U.S. Food and Drug Administration (FDA) and European Medicines Agency (EMA) published guidance for assessing RWD, emphasizing consistency, completeness, accuracy, and timeliness. The Duke Margolis Center for Health policy also published guidance that focused on the completeness, transparency, generalizability, timeliness, and scalability of RWD. The theme running through each is to establish data quality standards for creating RWE that is credible and useful for internal and external stakeholders, as well as for regulators.
Leveraging credibility frameworks to assess RWD
Evaluation of data sources is context dependent, but once a list of potential RWD sources has been assembled, a qualitative and quantitative analysis should be conducted on each of the relevant RWD sources identified through secondary research.
Engaging with the data owners to understand capabilities, collection methods, data quality controls, data provenance, lineage, and missing data can identify strengths and weaknesses of the data. In general, each of the data credibility frameworks advises that when assessing data and its fitness for purpose, important consideration must be given to:
- Missing data/errors and correction methods
- Defining the lifecycle, validity, and integrity of data
- Granularity of clinical data collected to ensure sufficient context
- Publication and use history
After refining the list based on a qualitative examination, a quantitative analysis will help determine if the data meet the pre-specified inclusion/exclusion criteria, target sample size needs, and sufficient data quality and completeness.
Once assessed using a standardized approach, each data source can be evaluated to weigh the data suitability and credibility for its intended audience. FIGURE 1 shows a sample data credibility assessment matrix. It is important to note that data set weakness in one or more dimensions does not necessarily mean that the source must be excluded. Some data sets may warrant additional data extraction methods, such as natural language processing (NLP), tokenization, and linking to other data sources, or enrichment of secondary data through prospective primary data collection.
Figure 1: Qualitative and quantitative assessments of RWD sources highlight areas of strength as well as potential gaps.
Identifying credible data source is only one step. Ensuring the RWD source or set of sources must be fit-for-purpose to address stakeholder needs is also essential. Part two in this blog series will discuss how to identify the right set of data sources to address stakeholders’ broad evidence needs.
