Pitfalls in RWE

Some of the biggest challenges when moving out of randomised clinical trials (RCT) and into RWE studies are confounding variables and other types of bias arising from the non-randomised nature of data. Not being aware of and addressing these biases can lead to comparisons that aren’t valid.  Each RWE study should therefore be examined individually for sources of bias and for confounding factors.

Researchers deal with the challenges of bias and confounding through statistical adjustments. These include different forms for logistic regression techniques and matching methods such as propensity score matching.

RWE in allergy

In our experience two of the often-encountered pitfalls when we look at retrospective RWE studies in allergy are:

Confounding arises when the variable analysed, e.g. a specific health intervention, itself is associated with certain characteristic, like severity of the disease in question. In other words, confounding offers an alternative (non-true) explanation for an association between an exposure and outcome. An example could be age distribution in the sample. Onset of asthma is more likely to occur in childhood, so looking at asthma onset as an outcome, age will be an important confounder to account for when analysing data.

Selection bias meaning the population selected for analysis does not represent the true characteristics of the group of interest and favours or disfavours one cohort over another.  An example of this could be disease severity. In the field of allergy, it is typically the more severe patients who initiate allergy immunotherapy. Addressing this bias by either ensuring a good basis for comparison, e.g. matching treated patients and controls, or using statistical models to adjust for severity is therefore of utmost importance if the results are to be trusted.

Pitfalls when using RWE studies to assess effectiveness of allergy immunotherapy

Treatment effectiveness in allergic rhinitis (AR) is difficult to measure as typical medical records or claims data do not contain suitable data to measure disease outcomes of AR, such as symptoms or disease progression. Instead, retrospective RWE studies within AR often use prescriptions of e.g. symptom-relieving or asthma medication as proxy for effectiveness of AIT. However, this approach is not trivial because:

• In many countries more symptom-relieving medication, which used to be by prescription only is also becoming available as over the counter (OTC) medication. However, OTC use is not captured in claims databases leading to a potentially artificial low use of symptom-relieving medication.
• AIT is often provided by specialists, which in our experience may lead to differences in care compared to patients not seen by a specialist with the potential to ultimately cause confounding and bias of the results. Being seen more often and by a specialist could lead to more prescriptions of for example symptom-relieving medication as well as diagnosis for comorbidities such as asthma.
• Being prescribed symptom-relieving medication is not the same as utilization of symptom-relieving medication. In the databases it is only captured whether a patient picked-up the prescribed medication or not, but not if the patient was taken the medication as prescribed.

Assessing the impact of AIT on asthma may be better suited for RWE. Firstly, it is possible to find clinically relevant outcomes, such as diagnosis codes for asthma exacerbations, hospitalisation due to asthma in electronic medical records or claims data. Secondly, asthma medication has not to the same extend become available as OTC, making the frequency of prescriptions more reliable to measure effectiveness in asthma.

Pitfalls when choosing controls in allergy immunotherapy RWE studies

Making sure you are comparing like with like is essential to draw the right conclusions. From our experience some of the specific challenges in the field of allergy are:

• In many databases the results of diagnostic tests are not captured, meaning that most databases lack information on what patients are sensitised/allergic to. While most databases capture which allergen the allergy immunotherapy is targeting, the lack of diagnosis means that choosing controls based on specific allergies is often not possible.
• Another challenge in identifying relevant controls are the difficulty assessing the severity of AR based on the available information in the databases. AR severity is not easily defined from neither diagnosis, comorbidities, nor prescription drug utilisation.
• The confounding and selection bias are not only challenging between AIT and non-AIT patients but could also be found between the different modes of administration of AIT. An example of this is that SLIT seems to be the preferred mode of administration in children.