Prev Sci. 2021 May 13; doi: 10.1007/s11121-021-01252-5. Epub 2021 May 13.

The Role of Clearinghouses in Promoting Transparent Research: A Methodological Study of Transparency Practices for Preventive Interventions.

Prevention science : the official journal of the Society for Prevention Research

Pamela R Buckley, Charles R Ebersole, Christine M Steeger, Laura E Michaelson, Karl G Hill, Frances Gardner

PMID: 33983558 DOI: 10.1007/s11121-021-01252-5

Abstract

Transparency of research methods is vital to science, though incentives are variable, with only some journals and funders adopting transparency policies. Clearinghouses are also important stakeholders; however, to date none have implemented formal procedures that facilitate transparent research. Using data from the longest standing clearinghouse, we examine transparency practices for preventive interventions to explore the role of online clearinghouses in incentivizing researchers to make their research more transparent. We conducted a descriptive analysis of 88 evaluation reports reviewed in 2018-2019 by Blueprints for Healthy Youth Development, when the clearinghouse began checking for trial registrations, and expanded on these efforts by applying broader transparency standards to interventions eligible for an endorsement on the Blueprints website during the study period. Reports were recent, with 84% published between 2010 and 2019. We found that few reports had data, code, or research materials that were publicly available. Meanwhile, 40% had protocols that were registered, but only 8% were registered prospectively, while one-quarter were registered before conducting analyses. About one-third included details in a registered protocol describing the treatment contrast and planned inclusions, and less than 5% had a registered statistical analysis plan (e.g., planned analytical methods, pre-specified covariates). Confirmatory research was distinguished from exploratory work in roughly 40% of reports. Reports published more recently (after 2015) had higher rates of transparency. Preventive intervention research needs to be more transparent. Since clearinghouses rely on robust findings to make well-informed decisions and researchers are incentivized to meet clearinghouse standards, clearinghouses should consider policies that encourage transparency to improve the credibility of evidence-based interventions.

Keywords: Clearinghouses; Open badges system; Preregistration; Registries; Scoping review; TOP guidelines; Transparency; Trial registration

References

1. Altman, D. G., Furberg, C. D., Grimshaw, J. M., & Shanahan, D. R. (2014). Linked publications from a single trial: A thread of evidence. Trials, 15, 1–3. https://doi.org/10.1186/1745-6215-15-369 - PubMed
2. Anderson, D., Spybrook, J., & Maynard, R. (2019). REES: A registry of efficacy and effectiveness studies in education. Educational Researcher, 48, 45–50. https://doi.org/10.3102/0013189X18810513 - PubMed
3. Anderson, M. S., Ronning, E. A., De Vries, R., & Martinson, B. C. (2007). The perverse effects of competition on scientists’ work and relationships. Science and engineering ethics, 13, 437–461. https://doi.org/10.1007/s11948-007-9042-5 - PubMed
4. Azar, M., Riehm, K. E., McKay, D., & Thombs, B. D. (2015). Transparency of outcome reporting and trial registration of randomized controlled trials published in the Journal of Consulting and Clinical Psychology. PLoS One, 10, e0142894. https://doi.org/10.1371/journal.pone.0142894 - PubMed
5. Buckley, P. R., Fagan, A. A., Pampel, F. C. & Hill, K.G. (2020). Making evidence-based interventions relevant for users: A comparison of requirements for dissemination readiness across program registries. Evaluation Review, 44, 51–83. - PubMed
6. Buckley, P. R., Moore, B., Boardman, A. G., Arya, D. J., & Maul, A. (2017). Validating a fidelity scale to understand intervention effects in classroombased studies. American Educational Research Journal, 54, 1378–1413. - PubMed
7. Burkhardt, J. T., Schroter, D. C., Magura, S., Means, S. N., & Coryn, C. L. S. (2015). An overview of evidence-based program registers (EBPRs) for behavioral health. Evaluation and Program Planning, 48, 92–99. https://doi.org/10.1016/j.evalprogplan.2014.09.006 - PubMed
8. Chambers, C. (2017). The 7 deadly sins of psychology. Princeton: Princeton University Press Cybulski, L., Mayo-Wilson, E., & Grant, S. (2016). Improving transparency and reproducibility through registration: the status of intervention trials published in clinical psychology journals. Journal of Consulting and Clinical Psychology, 84, 753. https://doi.org/10.1037/ccp0000115 - PubMed
9. Cybulski, L., Mayo-Wilson, E., & Grant, S. (2016). Improving transparency and reproducibility through registration: The status of intervention trials published in clinical psychology journals. Journal of Consulting and Clinical Psychology, 84, 753. https://doi.org/10.1037/ccp0000115 - PubMed
10. Des Jarlais, D. C., Lyles, C., Crepaz, N., & the Trend Group. (2004). Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: The TREND statement. American Journal of Public Health, 94, 361–366. - PubMed
11. Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A. W., Cronin, E., & Ghersi, D. (2008). Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One, 3, e3081. https://doi.org/10.1371/journal.pone.0003081 - PubMed
12. Eich, E. (2014). Business not as usual. Psychological Science, 25, 3–6. https://doi.org/10.1177/0956797613512465 - PubMed
13. Gates, A., Hartling, L., Vandermeer, B., Caldwell, P., Contopoulos-Ioannidis, D. G., Curtis, S., & Dyson, M. P. (2018). The conduct and reporting of child health research: an analysis of randomized controlled trials published in 2012 and evaluation of change over 5 years. Journal of Pediatrics, 193, 237-244.e37. https://doi.org/10.1016/j.jpeds.2017.09.014 - PubMed
14. Gennetian, L. A., Tamis-LeMonda, C. S., & Frank, M. C. (2020). Advancing transparency and openness in child development research: Opportunities. Child Development Perspectives, 14, 3–8. https://doi.org/10.1111/cdep.12356 - PubMed
15. Gottfredson, D. C., Cook, T. D., Gardner, F., Gorman-Smith, D., Howe, G. W., Sandler, I. N., & Zafft, K. M. (2015). Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: Next generation. Prevention science, 16, 893–926. - PubMed
16. Grant, S., Mayo-Wilson, E., Montgomery, P., Macdonald, G., Michie, S., Hopewell, S., & Moher, D. (2018). CONSORT-SPI 2018 Explanation and Elaboration: Guidance for reporting social and psychological intervention trials. Trials, 19, 406. https://doi.org/10.1186/s13063-018-2735-z - PubMed
17. Hardwicke, T. E., & Ioannidis, J. P. (2018). Mapping the universe of registered reports. Nature Human Behaviour, 2, 793–796. https://doi.org/10.1038/s41562-018-0444-y - PubMed
18. Hildenbrand, A. K., Conour, C., Straus, J. A., Moufarrej, S., & Palermo, T. M. (2019). Trial registration and outcome reporting in child and pediatric psychology: A systematic review. Journal of Pediatric Psychology, 44, 1024–1033. https://doi.org/10.1093/jpepsy/jsz054 - PubMed
19. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532. https://doi.org/10.1177/0956797611430953 - PubMed
20. Jones, P. M., Chow, J. T. Y., Arango, M. F., Fridfinnson, J. A., Gai, N., Lam, K., & Turkstra, T. P. (2017). Comparison of registered and reported outcomes in randomized clinical trials published in anesthesiology journals. Anesthesia & Analgesia, 125, 1292–1300. https://doi.org/10.1213/ANE.0000000000002272 - PubMed
21. Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L. S., & Errington, T. M. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLoS Biology, 14, e1002456. - PubMed
22. Lawson, D. O., Puljak, L., Pieper, D., Schandelmaier, S., Collins, G. S., Brignardello-Petersen, R., & Mbuagbaw, L. (2020). Reporting of methodological studies in health research: A protocol for the development of the MethodologIcal STudy reportIng Checklist (MISTIC). British Medical Journal Open, 10, e040478. - PubMed
23. Little, R. J., & Rubin, D. B. (2019). Statistical analysis with missing data (Vol. 793). John Wiley & Sons. https://doi.org/10.1371/journal.pbio.1002456 - PubMed
24. Mayo-Wilson, E., Grant, S., Supplee, L., Kianersi, S., Amin, A., DeHaven, A. C., & Mellor, D. T. (2021). Evaluating implementation of the transparency and openness promotion guidelines: The TRUST process for rating journal policies, procedures, and practices. MetaArXiv. https://doi.org/10.31222/osf.io/b3wju - PubMed
25. Means, S., Magura, S., Burkhart, B. R., Schroter, D. C., & Coryn, C. L. S. (2015). Comparing rating paradigms for evidence-based program registers in behavioral health: Evidentiary criteria and implications for assessing programs. Evaluation and Program Planning, 48, 100–116. https://doi.org/10.1016/j.evalprogplan.2014.09.007 - PubMed
26. Mihalic, S., & Elliott, D. S. (2015). Evidence-based programs registry: Blueprints for healthy youth development. Evaluation and Program Planning, 48, 124–131. https://doi.org/10.1016/j.evalprogplan.2014.08.004 - PubMed
27. Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., The PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6, e1000097. https://doi.org/10.1371/journal.pmed1000097 - PubMed
28. Morris, M., & Clark, B. (2013). You want me to do WHAT? Evaluators and the pressure to misrepresent findings. American Journal of Evaluation, 34, 57–70. https://doi.org/10.1177/1098214012457237 - PubMed
29. Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., Du Sert, N. P., & Ioannidis, J. P. (2017). A manifesto for reproducible science. Nature human behaviour, 1, 1–9. https://doi.org/10.1038/s41562-016-0021 - PubMed
30. Nosek, B. A. (2017). Center for open science: Strategic plan. Retrieved from:  https://doi.org/10.31219/osf.io/x2w9h - PubMed
31. Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., & Contestabile, M. (2015). Promoting an open research culture. Science, 348, 1422–1425. https://doi.org/10.1126/science.aab2374 - PubMed
32. Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115, 2600–2606. - PubMed
33. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615– 631.  https://doi.org/10.1177/1745691612459058 - PubMed
34. Paulsell, D., Thomas, J., Monahan, S., & Seftor, N. S. (2017). A trusted source of information: How systematic reviews can support user decisions about adopting evidence-based programs. Evaluation Review, 41, 50–77. https://doi.org/10.1177/0193841X16665963 - PubMed
35. Pocock, S. J., Assmann, S. E., Enos, L. E., & Kasten, L. E. (2002). Subgroup analysis, covariate adjustment and baseline comparisons in clinical trial reporting: current practice and problems. Statistics in Medicine, 21, 2917–2930. https://doi.org/10.1002/sim.1296 - PubMed
36. Rogers, T., & Feller, A. (2018). Reducing student absences at scale by targeting parents’ misbeliefs. Nature Human Behavior, 2, 335–342. https://doi.org/10.1038/s41562-018-0328-1 - PubMed
37. Rothstein, H. R., & Hopewell, S. (2009). Grey literature. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The Handbook of Research Synthesis and Meta-Analysis (p. 103–125). Russell Sage Foundation. - PubMed
38. Rotz, D., Luca, D. A., Goesling, B., Cook, E., Murphy, K., & Stevens, J. (2016). Final impacts of the teen options to prevent pregnancy program. U.S. Department of Health and Human Services, Office of Adolescent Health. - PubMed
39. Scheel, A. M., Schijen, M., & Lakens, D. (2020). An excess of positive results: Comparing the standard Psychology literature with Registered Reports. PsyArXiv, 1-14.  https://doi.org/10.31234/osf.io/p6e9c - PubMed
40. Schimanski, L. A., & Alperin, J. P. (2018). The evaluation of scholarship in academic promotion and tenure processes: Past, present, and future. F1000Research, 7, 1605.  https://doi.org/10.12688/f1000research.16493.1 - PubMed
41. Schulz, K. F., Altman, D. G., Moher, D., & Group, C. (2010). CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. Trials, 11, 32. https://doi.org/10.1186/1745-6215-11-32 - PubMed
42. Song, F., Parekh, S., Hooper, L., Loke, Y. K., Ryder, J., Sutton, A. J., & Harvey, I. (2010). Dissemination and publication of research findings: An updated review of related biases. Health Technology Assessment, 14, 1–193. https://doi.org/10.3310/hta14080 - PubMed
43. Steeger, C. M., Buckley, P. R., Pampel, F. C., Gust, C., & Hill, K. G. (2021). Common methodological problems in randomized controlled trials of preventive interventions. Prevention Science. Forthcoming. - PubMed
44. Van den Akker, O., Weston, S. J., Campbell, L., Chopik, W. J., Damian, R. I., Davis-Kean, P., Bakker, M. (2019, November 20). Preregistration of secondary data analysis: A template and tutorial. https://doi.org/10.31234/osf.io/hvfmr - PubMed
45. Vazire, S. (2019). Do we want to be credible or incredible? APS Observer, 33. Retrieved February 5, 2021 from: https://www.psychologicalscience.org/observer/do-we-want-to-be-credible-or-incredible - PubMed
46. Wadhwa, M., & Cook, T. D. (2019). The set of assumptions randomized control trials make and their implications for the role of such experiments in evidence-based child and adolescent development research. New Directions for Child and Adolescent Development, 167, 17–37. https://doi.org/10.1002/sim.1296 - PubMed
47. Walker, S. C., Lyon, A. R., Aos, S., & Trupin, E. W. (2017). The consistencies and vagaries of the Washington state inventory of evidence-based practice: The definition of “evidence-based” in a policy context. Administration and Policy in Mental Health and Mental Health Services Research, 44, 42–54. https://doi.org/10.1007/s10488-015-0652-y - PubMed

Publication Types

• Journal Article