Display options
Cite
van Boxtel JJ, Koch C. Reevaluating excess success in psychological science. Psychon Bull Rev. 2016;23(5):1602-1606doi: 10.3758/s13423-016-1010-0.
van Boxtel, J. J., & Koch, C. (2016). Reevaluating excess success in psychological science. Psychonomic bulletin & review, 23(5), 1602-1606. https://doi.org/10.3758/s13423-016-1010-0
van Boxtel, Jeroen J A, and Koch, Christof. "Reevaluating excess success in psychological science." Psychonomic bulletin & review vol. 23,5 (2016): 1602-1606. doi: https://doi.org/10.3758/s13423-016-1010-0
van Boxtel JJ, Koch C. Reevaluating excess success in psychological science. Psychon Bull Rev. 2016 Oct;23(5):1602-1606. doi: 10.3758/s13423-016-1010-0. PMID: 26873831.
Copy Download .nbib Format: NLM
Share it on

Psychon Bull Rev. 2016 Oct;23(5):1602-1606. doi: 10.3758/s13423-016-1010-0.

Reevaluating excess success in psychological science.

Psychonomic bulletin & review

Jeroen J A van Boxtel, Christof Koch

Affiliations

  1. School of Psychological Sciences and Monash Institute of Cognitive and Clinical Neurosciences, Monash University, Clayton, 3800, VIC, Australia. [email protected].
  2. Allen Institute for Brain Science, Seattle, WA, USA.

PMID: 26873831 DOI: 10.3758/s13423-016-1010-0

Abstract

Francis (Psychonomic Bulletin Review, 21, 1180-1187, 2014) recently claimed that 82 % of articles with four or more experiments published in Psychological Science between 2009 and 2012 cannot be trusted. We critique Francis' analysis and point out the dependence of his approach on including the appropriate experiments and significance tests. We focus on one of the articles (van Boxtel & Koch, in Psychological Science, 23(4), 410-418, 2012) flagged by Francis and show that the inappropriate inclusion of experiments and tests have led Francis to mistakenly flag this article. We found that decisions about whether to include certain tests potentially affect 34 of the 44 articles analyzed by Francis. We further performed p-curve analyses on the articles discussed in Francis' analysis. We found that 9 of 44 studies showed significant evidential value, 11 studies showed insufficient evidential value, and 1 study showed evidence of p-hacking. Our reevaluation is important, because some researchers may have gained the false impression that none of the quoted articles in Psychological Science can be trusted (as stated by Francis). The analysis by Francis is most likely insufficient to warrant this conclusion for some articles and certainly is insufficient with respect to the study by van Boxtel and Koch (Psychological Science, 23, 410-418, 2012).

Keywords: Statistical inference; Statistics

References

  1. J Vis. 2009 Dec 05;9(13):5.1-11 - PubMed
  2. Psychol Sci. 2012 Apr;23(4):410-8 - PubMed
  3. Pers Soc Psychol Rev. 2014 May;18(2):107-18 - PubMed
  4. Clin Trials. 2007;4(3):245-53 - PubMed
  5. J Exp Psychol Gen. 2014 Apr;143(2):534-47 - PubMed
  6. Psychon Bull Rev. 2014 Oct;21(5):1180-7 - PubMed

Publication Types