Display options
Cite
Middel B, van Sonderen E. Statistical significant change versus relevant or important change in (quasi) experimental design: some conceptual and methodological problems in estimating magnitude of intervention-related change in health services research. Int J Integr Care. 2002;2:e15doi: 10.5334/ijic.65.
Middel, B., & van Sonderen, E. (2002). Statistical significant change versus relevant or important change in (quasi) experimental design: some conceptual and methodological problems in estimating magnitude of intervention-related change in health services research. International journal of integrated care, 2e15. https://doi.org/10.5334/ijic.65
Middel, Berrie, and van Sonderen, Eric. "Statistical significant change versus relevant or important change in (quasi) experimental design: some conceptual and methodological problems in estimating magnitude of intervention-related change in health services research." International journal of integrated care vol. 2 (2002): e15. doi: https://doi.org/10.5334/ijic.65
Middel B, van Sonderen E. Statistical significant change versus relevant or important change in (quasi) experimental design: some conceptual and methodological problems in estimating magnitude of intervention-related change in health services research. Int J Integr Care. 2002;2:e15. doi: 10.5334/ijic.65. Epub 2002 Dec 17. PMID: 16896390; PMCID: PMC1480399.
Copy Download .nbib Format: NLM
Share it on

Int J Integr Care. 2002;2:e15. doi: 10.5334/ijic.65. Epub 2002 Dec 17.

Statistical significant change versus relevant or important change in (quasi) experimental design: some conceptual and methodological problems in estimating magnitude of intervention-related change in health services research.

International journal of integrated care

Berrie Middel, Eric van Sonderen

Affiliations

  1. Department of Health Sciences, Sub-Division Care Science, A. Deusinglaan 1, 9713 AV Groningen, The Netherlands. [email protected]

PMID: 16896390 PMCID: PMC1480399 DOI: 10.5334/ijic.65

Abstract

This paper aims to identify problems in estimating and the interpretation of the magnitude of intervention-related change over time or responsiveness assessed with health outcome measures. Responsiveness is a problematic construct and there is no consensus on how to quantify the appropriate index to estimate change over time between baseline and post-test designs. This paper gives an overview of several responsiveness indices. Thresholds for effect size (or responsiveness index) interpretation were introduced some thirty years ago by Cohen who standardised the difference-scores (d) with the pooled standard deviation (d/SD(pooled)). However, many effect sizes (ES) have been introduced since Cohen's original work and in the formula of one of these ES, the mean change scores are standardised with the SD of those change scores (d/SD(change)). When health outcome questionnaires are used, this effect size is applied on a wide scale and is represented as the Standardized Response Mean (SRM). However, its interpretation is problematic when it is used as an estimate of magnitude of change over time and interpreted with the thresholds, set by Cohen for effect size (ES) which is based on SD(pooled). Thus, in the case of using the SRM, application of these well-known cut-off points for pooled standard deviation units namely: 'trivial' (ES < 0.20), 'small' (ES > or = 0.20 < 0.50), 'moderate' (ES > or = 0.50 < 0.80), or large (ES > or = 0.80), may lead to over- or underestimation of the magnitude of intervention-related change over time due to the correlation between baseline and outcome assessments. Consequently, taking Cohen's thresholds for granted for every version of effect size indices as estimates of intervention-related magnitude of change, may lead to over- or underestimation of this magnitude of intervention-related change over time.

References

  1. Am J Psychiatry. 1995 May;152(5):683-91 - PubMed
  2. Psychiatry Res. 1988 Mar;23(3):301-9 - PubMed
  3. Qual Life Res. 1996 Apr;5(2):223-34 - PubMed
  4. J Clin Epidemiol. 1992 Dec;45(12):1341-5 - PubMed
  5. Int J Integr Care. 2002;2:e17 - PubMed
  6. J Adv Nurs. 1999 Feb;29(2):298-306 - PubMed
  7. Psychopharmacol Bull. 1994;30(2):235-8 - PubMed
  8. Gerontologist. 1990 Aug;30(4):516-21 - PubMed
  9. J Gerontol B Psychol Sci Soc Sci. 1996 Sep;51(5):P254-60 - PubMed
  10. Psychoneuroendocrinology. 1992 Nov;17(6):527-36 - PubMed
  11. J Clin Oncol. 1998 Jan;16(1):139-44 - PubMed
  12. Qual Life Res. 1995 Feb;4(1):47-52 - PubMed
  13. BMJ. 1993 May 29;306(6890):1437-40 - PubMed
  14. J Clin Epidemiol. 2000 Jan;53(1):19-23 - PubMed
  15. J Rheumatol. 1995 Jun;22(6):1193-6 - PubMed
  16. Arch Intern Med. 1986 Jul;146(7):1325-9 - PubMed
  17. J Clin Epidemiol. 1995 Nov;48(11):1369-78 - PubMed
  18. JAMA. 1994 Aug 24-31;272(8):619-26 - PubMed
  19. J Clin Epidemiol. 1993 Oct;46(10):1091-2 - PubMed
  20. J Consult Clin Psychol. 1991 Feb;59(1):12-9 - PubMed
  21. J Clin Epidemiol. 1997 Mar;50(3):239-46 - PubMed
  22. JAMA. 1989 Aug 18;262(7):907-13 - PubMed
  23. Qual Life Res. 1997 Apr;6(3):205-12 - PubMed
  24. J Clin Epidemiol. 1998 Oct;51(10):827-35 - PubMed
  25. J Allergy Clin Immunol. 1996 Aug;98(2):309-16 - PubMed
  26. J Clin Epidemiol. 1997 Jul;50(7):757-63 - PubMed
  27. J Clin Epidemiol. 1996 Nov;49(11):1221-2 - PubMed
  28. Qual Life Res. 1995 Feb;4(1):27-32 - PubMed
  29. Psychopharmacol Bull. 1993;29(2):163-7 - PubMed
  30. J Bone Joint Surg Br. 1994 Mar;76(2):229-34 - PubMed
  31. Annu Rev Public Health. 1994;15:535-59 - PubMed
  32. Arch Ophthalmol. 1995 Dec;113(12):1508-13 - PubMed
  33. J Clin Epidemiol. 1993 Oct;46(10):1113-8 - PubMed
  34. J Chronic Dis. 1987;40(2):171-8 - PubMed
  35. Pain. 1996 Apr;65(1):71-6 - PubMed
  36. J Bone Joint Surg Am. 1997 Jul;79(7):974-83 - PubMed
  37. Int J Cancer Suppl. 1999;12:132-7 - PubMed
  38. Clin Rehabil. 2001 Oct;15(5):489-500 - PubMed
  39. J Epidemiol Community Health. 1996 Jun;50(3):377-80 - PubMed
  40. Med Care. 1993 Mar;31(3):247-63 - PubMed
  41. J Clin Epidemiol. 1998 Feb;51(2):137-45 - PubMed
  42. J Rheumatol. 1999 May;26(5):1163-6 - PubMed
  43. Med Care. 1998 Apr;36(4):491-502 - PubMed
  44. Ann Rheum Dis. 1998 Feb;57(2):82-7 - PubMed
  45. Med Care. 1996 Dec;34(12):1165-70 - PubMed
  46. Med Care. 1989 Mar;27(3 Suppl):S178-89 - PubMed
  47. J Clin Epidemiol. 1989;42(11):1097-105 - PubMed
  48. Epilepsia. 1997 Apr;38(4):430-8 - PubMed
  49. J Clin Epidemiol. 1997 May;50(5):529-40 - PubMed
  50. Behav Res Ther. 1999 Dec;37(12):1219-33 - PubMed
  51. Med Care. 1989 Mar;27(3 Suppl):S217-32 - PubMed
  52. J Rheumatol. 1999 Jan;26(1):217-21 - PubMed
  53. Digestion. 1997;58(3):282-8 - PubMed
  54. J Clin Epidemiol. 1998 Jul;51(7):617-31 - PubMed
  55. Int J Behav Med. 1996;3(2):104-22 - PubMed
  56. J Clin Epidemiol. 1991;44(4-5):417-21 - PubMed
  57. J Allergy Clin Immunol. 1997 Jul;100(1):16-22 - PubMed
  58. J Clin Epidemiol. 1996 Jan;49(1):39-44 - PubMed
  59. J Rheumatol. 1987 Jun;14(3):446-51 - PubMed
  60. Orbit. 2001 Dec;20(4):281-290 - PubMed
  61. J Clin Epidemiol. 1998 Jan;51(1):11-8 - PubMed
  62. J Allergy Clin Immunol. 1997 Feb;99(2):S742-9 - PubMed
  63. Behav Res Ther. 1993 Sep;31(7):693-700 - PubMed
  64. Med Care. 1994 Nov;32(11):1127-33 - PubMed
  65. Psychol Bull. 1960 Sep;57:416-28 - PubMed
  66. Med Care. 1992 Jun;30(6):473-83 - PubMed
  67. Arthritis Care Res. 1992 Sep;5(3):130-6 - PubMed
  68. Med Care. 1992 Oct;30(10):917-25 - PubMed
  69. Ann Rheum Dis. 1992 Nov;51(11):1202-5 - PubMed
  70. Med Care. 1997 Apr;35(4):410-6 - PubMed
  71. Med Care. 1999 Jan;37(1):56-67 - PubMed
  72. J Epidemiol Community Health. 1995 Feb;49(1):102-5 - PubMed
  73. J Clin Epidemiol. 2000 May;53(5):459-68 - PubMed
  74. Control Clin Trials. 1991 Aug;12(4 Suppl):142S-158S - PubMed
  75. Med Care. 1995 Apr;33(4 Suppl):AS280-5 - PubMed
  76. Health Care Anal. 1996 Nov;4(4):265-72 - PubMed
  77. Arch Intern Med. 1990 Jan;150(1):59-62 - PubMed
  78. J Clin Epidemiol. 1996 Jul;49(7):711-7 - PubMed
  79. J Cardiopulm Rehabil. 1998 Mar-Apr;18(2):139-44 - PubMed
  80. JAMA. 2000 Apr 12;283(14):1824-5 - PubMed
  81. Qual Life Res. 1998 Jan;7(1):11-22 - PubMed
  82. Int J Integr Care. 2000 Nov 01;1:e10 - PubMed
  83. J Consult Clin Psychol. 1992 Jun;60(3):402-8 - PubMed
  84. Psychiatry Res. 1988 Mar;23(3):295-9 - PubMed
  85. Ann Intern Med. 1993 Apr 15;118(8):622-9 - PubMed
  86. Med Care. 1995 Apr;33(4 Suppl):AS264-79 - PubMed
  87. Arthritis Rheum. 1985 May;28(5):542-7 - PubMed
  88. J Public Health Med. 1997 Jun;19(2):179-86 - PubMed
  89. J Neurol Neurosurg Psychiatry. 1997 Aug;63(2):204-9 - PubMed
  90. Med Care. 1990 Jul;28(7):632-42 - PubMed
  91. Urology. 1996 Jan;47(1):43-7 - PubMed
  92. J Cardiopulm Rehabil. 2002 Jan-Feb;22(1):47-52 - PubMed
  93. J Clin Epidemiol. 1994 Jul;47(7):787-95 - PubMed
  94. J Clin Epidemiol. 1997 May;50(5):581-8 - PubMed
  95. Age Ageing. 1999 May;28(3):275-81 - PubMed
  96. Behav Res Ther. 1999 Dec;37(12):1169-93 - PubMed

Publication Types