Display options
Cite
Kinnear B, Kelleher M, May B, et al. Constructing a Validity Map for a Workplace-Based Assessment System: Cross-Walking Messick and Kane. Acad Med. 2021;96(7):S64-S69doi: 10.1097/ACM.0000000000004112.
Kinnear, B., Kelleher, M., May, B., Sall, D., Schauer, D. P., Schumacher, D. J., & Warm, E. J. (2021). Constructing a Validity Map for a Workplace-Based Assessment System: Cross-Walking Messick and Kane. Academic medicine : journal of the Association of American Medical Colleges, 96(7), S64-S69. https://doi.org/10.1097/ACM.0000000000004112
Kinnear, Benjamin, et al. "Constructing a Validity Map for a Workplace-Based Assessment System: Cross-Walking Messick and Kane." Academic medicine : journal of the Association of American Medical Colleges vol. 96,7 (2021): S64-S69. doi: https://doi.org/10.1097/ACM.0000000000004112
Kinnear B, Kelleher M, May B, Sall D, Schauer DP, Schumacher DJ, Warm EJ. Constructing a Validity Map for a Workplace-Based Assessment System: Cross-Walking Messick and Kane. Acad Med. 2021 Jul 01;96(7):S64-S69. doi: 10.1097/ACM.0000000000004112. PMID: 34183604.
Copy Download .nbib Format: NLM
Share it on

Acad Med. 2021 Jul 01;96(7):S64-S69. doi: 10.1097/ACM.0000000000004112.

Constructing a Validity Map for a Workplace-Based Assessment System: Cross-Walking Messick and Kane.

Academic medicine : journal of the Association of American Medical Colleges

Benjamin Kinnear, Matthew Kelleher, Brian May, Dana Sall, Daniel P Schauer, Daniel J Schumacher, Eric J Warm

Affiliations

  1. B. Kinnear is associate professor of internal medicine and pediatrics, Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0003-0052-4130.
  2. M. Kelleher is assistant professor of internal medicine and pediatrics, Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio.
  3. B. May is assistant professor of internal medicine and pediatrics, Department of Internal Medicine, University of Alabama Birmingham School of Medicine, Birmingham, Alabama.
  4. D. Sall is program director, HonorHealth Internal Medicine Residency Program, Scottsdale, Arizona, and assistant professor of internal medicine, University of Arizona College of Medicine, Phoenix, Arizona.
  5. D.P. Schauer is associate professor of internal medicine and associate program director, Department of Internal Medicine, University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0003-3264-8154.
  6. D.J. Schumacher is associate professor of pediatrics at Cincinnati Children's Hospital Medical Center/University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0001-5507-8452.
  7. E.J. Warm is professor of internal medicine and program director, Department of Internal Medicine, University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0002-6088-2434.

PMID: 34183604 DOI: 10.1097/ACM.0000000000004112

Abstract

PROBLEM: Health professions education has shifted to a competency-based paradigm in which many programs rely heavily on workplace-based assessment (WBA) to produce data for summative decisions about learners. However, WBAs are complex and require validity evidence beyond psychometric analysis. Here, the authors describe their use of a rhetorical argumentation process to develop a map of validity evidence for summative decisions in an entrustment-based WBA system.

APPROACH: To organize evidence, the authors cross-walked 2 contemporary validity frameworks, one that emphasizes sources of evidence (Messick) and another that stresses inferences in an argument (Kane). They constructed a validity map using 4 steps: (1) Asking critical questions about the stated interpretation and use, (2) Seeking validity evidence as a response, (3) Categorizing evidence using both Messick's and Kane's frameworks, and (4) Building a visual representation of the collected and organized evidence. The authors used an iterative approach, adding new critical questions and evidence over time.

OUTCOMES: The first map draft produced 25 boxes of evidence that included all 5 sources of evidence detailed by Messick and spread across all 4 inferences described by Kane. The rhetorical question-response process allowed for structured critical appraisal of the WBA system, leading to the identification of evidentiary gaps.

NEXT STEPS: Future map iterations will integrate evidence quality indicators and allow for deeper dives into the evidence. The authors intend to share their map with graduate medical education stakeholders (e.g., accreditors, institutional leaders, learners, patients) to understand if it adds value for evaluating their WBA programs' validity arguments.

Copyright © 2021 by the Association of American Medical Colleges.

References

  1. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system—Rationale and Benefits. N Engl J Med. 2012; 366:1051–1056. - PubMed
  2. Frank JR, Danoff D. The CanMEDS initiative: Implementing an outcomes-based framework of physician competencies. Med Teach. 2007; 29:642–647. - PubMed
  3. Simpson JG, Furnace J, Crosby J, et al. The Scottish doctor—Learning outcomes for the medical undergraduate in Scotland: A foundation for competent and reflective practitioners. Med Teach. 2002; 24:136–143. - PubMed
  4. Confederation of Postgraduate Medical Education Councils. Australian Curriculum Framework for Junior Doctors. Melbourne, Australia: CPMEC; 2012. http://www.cpmec.org.au/files/27112017_watermark.pdf. Accessed March 22, 2021. - PubMed
  5. Holmboe ES, Sherbino J, Englander R, Snell L, Frank JR; ICBME Collaborators. A call to action: The controversy of and rationale for competency-based medical education. Med Teach. 2017; 39:574–581. - PubMed
  6. Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: Theory to practice. Med Teach. 2010; 32:638–645. - PubMed
  7. Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. Shifting paradigms: From Flexner to competencies. Acad Med. 2002; 77:361–367. - PubMed
  8. Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach. 2010; 32:676–682. - PubMed
  9. Lockyer J, Carraccio C, Chan MK, et al.; ICBME Collaborators. Core principles of assessment in competency-based medical education. Med Teach. 2017; 39:609–616. - PubMed
  10. Lurie SJ. History and practice of competency-based assessment. Med Educ. 2012; 46:49–57. - PubMed
  11. Lurie SJ, Mooney CJ, Lyness JM. Commentary: Pitfalls in assessment of competency-based educational objectives. Acad Med. 2011; 86:412–414. - PubMed
  12. Govaerts M, van der Vleuten CP. Validity in work-based assessment: Expanding our horizons. Med Educ. 2013; 47:1164–1174. - PubMed
  13. Downing SM. Validity: On meaningful interpretation of assessment data. Med Educ. 2003; 37:830–837. - PubMed
  14. St-Onge C, Young M, Eva KW, Hodges B. Validity: One word with a plurality of meanings. Adv Health Sci Educ Theory Pract. 2017; 22:853–867. - PubMed
  15. Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: A practical guide to Kane’s framework. Med Educ. 2015; 49:560–575. - PubMed
  16. Messick S. Validity. Linn RL, ed. In: Educational Measurement. New York, NY: Macmillan Publishing; 1989:13–103. - PubMed
  17. Kane MT. Validating the interpretations and uses of test scores. J Educ Meas. 2013; 50:1–73. - PubMed
  18. Warm EJ, Mathis BR, Held JD, et al. Entrustment and mapping of observable practice activities for resident assessment. J Gen Intern Med. 2014; 29:1177–1182. - PubMed
  19. Warm EJ, Held JD, Hellmann M, et al. Entrusting observable practice activities and milestones over the 36 months of an internal medicine residency. Acad Med. 2016; 91:1398–1405. - PubMed
  20. Rekman J, Gofton W, Dudek N, Gofton T, Hamstra SJ. Entrustability scales: Outlining their usefulness for competency-based clinical assessment. Acad Med. 2016; 91:186–190. - PubMed
  21. Crossley J, Johnson G, Booth J, Wade W. Good questions, good answers: Construct alignment improves the performance of workplace-based assessment scales. Med Educ. 2011; 45:560–569. - PubMed
  22. Crossley J, Jolly B. Making sense of work-based assessment: Ask the right questions, in the right way, about the right things, of the right people. Med Educ. 2012; 46:28–37. - PubMed
  23. Kelleher M, Kinnear B, Sall D, et al. A reliability analysis of entrustment-derived workplace-based assessments. Acad Med. 2020; 95:616–622. - PubMed
  24. Toulmin S. The Uses of Argument. Cambridge, UK: Cambridge University Press; 1958. - PubMed
  25. Wigmore JH. The Science of Judicial Proof, as Given by Logic, Psychology, and General Experience, and Illustrated in Judicial Trials. Boston, MA: Little, Brown; 1937. - PubMed
  26. Kinnear B, Kelleher M, May B, et al. Constructing a Validity Map for Workplace Based Assessments. Paper presented at: Research Group for Pursuit of Validity Colloquium; Montreal, Canada; 2019. - PubMed
  27. Kane MT. Current concerns in validity theory. J Educ Meas. 2001; 38:319–342. - PubMed
  28. Van Eemeren FH, Grootendorst R, Johnson RH, Plantin C, Willard CA. Fundamentals of Argumentation Theory: A Handbook of Historical Backgrounds and Contemporary Developments. New York, NY: Routledge; 2013. - PubMed

MeSH terms

Publication Types