Display options
Cite
Madigan D, Ryan PB, Schuemie M. Does design matter? Systematic evaluation of the impact of analytical choices on effect estimates in observational studies. Ther Adv Drug Saf. 2013;4(2):53-62doi: 10.1177/2042098613477445.
Madigan, D., Ryan, P. B., & Schuemie, M. (2013). Does design matter? Systematic evaluation of the impact of analytical choices on effect estimates in observational studies. Therapeutic advances in drug safety, 4(2), 53-62. https://doi.org/10.1177/2042098613477445
Madigan, David, et al. "Does design matter? Systematic evaluation of the impact of analytical choices on effect estimates in observational studies." Therapeutic advances in drug safety vol. 4,2 (2013): 53-62. doi: https://doi.org/10.1177/2042098613477445
Madigan D, Ryan PB, Schuemie M. Does design matter? Systematic evaluation of the impact of analytical choices on effect estimates in observational studies. Ther Adv Drug Saf. 2013 Apr;4(2):53-62. doi: 10.1177/2042098613477445. PMID: 25083251; PMCID: PMC4110833.
Copy Download .nbib Format: NLM
Share it on

Ther Adv Drug Saf. 2013 Apr;4(2):53-62. doi: 10.1177/2042098613477445.

Does design matter? Systematic evaluation of the impact of analytical choices on effect estimates in observational studies.

Therapeutic advances in drug safety

David Madigan, Patrick B Ryan, Martijn Schuemie

Affiliations

  1. Professor and Chair, Department of Statistics, Columbia University, 1255 Amsterdam Ave., New York, NY 10027, USA.
  2. Observational Medical Outcomes Partnership, Foundation for the National Institutes of Health, Bethesda, MD and Janssen Research and Development LLC, Titusville, NJ, USA.
  3. Observational Medical Outcomes Partnership, Foundation for the National Institutes of Health, Bethesda, MD and Department of Medical Informatics, Erasmus University Medical Center Rotterdam, Rotterdam, The Netherlands.

PMID: 25083251 PMCID: PMC4110833 DOI: 10.1177/2042098613477445

Abstract

BACKGROUND: Clinical studies that use observational databases, such as administrative claims and electronic health records, to evaluate the effects of medical products have become commonplace. These studies begin by selecting a particular study design, such as a case control, cohort, or self-controlled design, and different authors can and do choose different designs for the same clinical question. Furthermore, published papers invariably report the study design but do not discuss the rationale for the specific choice. Studies of the same clinical question with different designs, however, can generate different results, sometimes with strikingly different implications. Even within a specific study design, authors make many different analytic choices and these too can profoundly impact results. In this paper, we systematically study heterogeneity due to the type of study design and due to analytic choices within study design.

METHODS AND FINDINGS: We conducted our analysis in 10 observational healthcare databases but mostly present our results in the context of the GE Centricity EMR database, an electronic health record database containing data for 11.2 million lives. We considered the impact of three different study design choices on estimates of associations between bisphosphonates and four particular health outcomes for which there is no evidence of an association. We show that applying alternative study designs can yield discrepant results, in terms of direction and significance of association. We also highlight that while traditional univariate sensitivity analysis may not show substantial variation, systematic assessment of all analytical choices within a study design can yield inconsistent results ranging from statistically significant decreased risk to statistically significant increased risk. Our findings show that clinical studies using observational databases can be sensitive both to study design choices and to specific analytic choices within study design.

CONCLUSION: More attention is needed to consider how design choices may be impacting results and, when possible, investigators should examine a wide array of possible choices to confirm that significant findings are consistently identified.

Keywords: analysis; health outcomes; healthcare database; study design

References

  1. JAMA. 2001 Apr 11;285(14):1850-5 - PubMed
  2. Stat Med. 2012 Dec 30;31(30):4401-15 - PubMed
  3. PLoS Med. 2005 Aug;2(8):e124 - PubMed
  4. BMJ. 2008 Aug 28;337:a1069 - PubMed
  5. Pharmacoepidemiol Drug Saf. 2012 Jan;21 Suppl 1:32-40 - PubMed
  6. Popul Health Manag. 2010 Jun;13(3):139-50 - PubMed
  7. Int J Epidemiol. 2006 Oct;35(5):1301-8 - PubMed
  8. JAMA. 2010 Aug 11;304(6):657-63 - PubMed
  9. Pharmacoepidemiol Drug Saf. 2007 Mar;16(3):329-36 - PubMed
  10. Ann Intern Med. 2010 Nov 2;153(9):600-6 - PubMed
  11. J Biomed Inform. 2012 Aug;45(4):689-96 - PubMed
  12. Am J Med Qual. 2008 Sep-Oct;23(5):375-81 - PubMed
  13. PLoS One. 2009;4(3):e4720 - PubMed
  14. Int J Epidemiol. 1988 Sep;17(3):680-5 - PubMed
  15. J Am Med Inform Assoc. 2012 Jan-Feb;19(1):54-60 - PubMed
  16. JAMA. 2000 Jun 28;283(24):3205-10 - PubMed
  17. J Clin Epidemiol. 2008 Apr;61(4):344-9 - PubMed
  18. BMJ. 2011 Mar 17;342:d1309 - PubMed
  19. Epidemiology. 2009 Jul;20(4):512-22 - PubMed
  20. BMJ. 2010 Sep 01;341:c4444 - PubMed
  21. Nat Rev Rheumatol. 2011 Jun;7(6):369-72 - PubMed
  22. Basic Clin Pharmacol Toxicol. 2006 Mar;98(3):266-74 - PubMed
  23. Am J Epidemiol. 2004 Aug 15;160(4):384-92 - PubMed
  24. Pharmacoepidemiol Drug Saf. 2009 Dec;18(12):1176-84 - PubMed
  25. Int J Epidemiol. 2001 Feb;30(1):1-11 - PubMed

Publication Types