That is the question posed by an article by Wang, Sreedhara and Schneeweiss (2022). The authors replicate findings from n=250 real-world data studies. The studies were replicated using 3 data sets: (i) Optum Clinformatics claims data, (ii) IBM MarketScan Research Database claims data, Medicare fee-for-service claims data, and (iii) CPRD electronic medical records. . Key findings from the article are below:
The effect sizes of the original and reproduction were positively correlated (Pearson correlation = 0.85), a strong relationship with some room for improvement. The median and interquartile range for the relative effect size (e.g., original risk ratio/replication risk ratio) is 1.0. [0.9, 1.1]range [0.3, 2.1]. While most results are faithfully reproduced, a subset is not. The latter can be explained by incomplete reports and updated data. Greater methodological transparency aligned with new guidance can further improve reproducibility and validity assessment, thereby facilitating evidence-based decision making.
This work won the 2023 Award for Excellence in Health Economics and Outcomes Research Methodology. You can read the full article. here.
