Display options
Cite
Luckett DJ, Laber EB, El-Kamary SS, et al. Receiver operating characteristic curves and confidence bands for support vector machines. Biometrics. 2020;77(4):1422-1430doi: 10.1111/biom.13365.
Luckett, D. J., Laber, E. B., El-Kamary, S. S., Fan, C., Jhaveri, R., Perou, C. M., Shebl, F. M., & Kosorok, M. R. (2021). Receiver operating characteristic curves and confidence bands for support vector machines. Biometrics, 77(4), 1422-1430. https://doi.org/10.1111/biom.13365
Luckett, Daniel J, et al. "Receiver operating characteristic curves and confidence bands for support vector machines." Biometrics vol. 77,4 (2021): 1422-1430. doi: https://doi.org/10.1111/biom.13365
Luckett DJ, Laber EB, El-Kamary SS, Fan C, Jhaveri R, Perou CM, Shebl FM, Kosorok MR. Receiver operating characteristic curves and confidence bands for support vector machines. Biometrics. 2021 Dec;77(4):1422-1430. doi: 10.1111/biom.13365. Epub 2020 Sep 12. PMID: 32865820; PMCID: PMC7914290.
Copy Download .nbib Format: NLM
Share it on

Biometrics. 2021 Dec;77(4):1422-1430. doi: 10.1111/biom.13365. Epub 2020 Sep 12.

Receiver operating characteristic curves and confidence bands for support vector machines.

Biometrics

Daniel J Luckett, Eric B Laber, Samer S El-Kamary, Cheng Fan, Ravi Jhaveri, Charles M Perou, Fatma M Shebl, Michael R Kosorok

Affiliations

  1. Department of Biostatistics, University of North Carolina, Chapel Hill, North Carolina.
  2. Department of Statistics, North Carolina State University, Raleigh, North Carolina.
  3. Department of Epidemiology and Public Health, University of Maryland, Baltimore, Maryland.
  4. Department of Genetics, University of North Carolina, Chapel Hill, North Carolina.
  5. Department of Pediatrics, University of North Carolina, Chapel Hill, North Carolina.
  6. Department of Epidemiology, Yale University, New Haven, Connecticut.

PMID: 32865820 PMCID: PMC7914290 DOI: 10.1111/biom.13365

Abstract

Many problems that appear in biomedical decision-making, such as diagnosing disease and predicting response to treatment, can be expressed as binary classification problems. The support vector machine (SVM) is a popular classification technique that is robust to model misspecification and effectively handles high-dimensional data. The relative costs of false positives and false negatives can vary across application domains. The receiving operating characteristic (ROC) curve provides a visual representation of the trade-off between these two types of errors. Because the SVM does not produce a predicted probability, an ROC curve cannot be constructed in the traditional way of thresholding a predicted probability. However, a sequence of weighted SVMs can be used to construct an ROC curve. Although ROC curves constructed using weighted SVMs have great potential for allowing ROC curves analyses that cannot be done by thresholding predicted probabilities, their theoretical properties have heretofore been underdeveloped. We propose a method for constructing confidence bands for the SVM ROC curve and provide the theoretical justification for the SVM ROC curve by showing that the risk function of the estimated decision rule is uniformly consistent across the weight parameter. We demonstrate the proposed confidence band method using simulation studies. We present a predictive model for treatment response in breast cancer as an illustrative example.

© 2020 The International Biometric Society.

Keywords: classification; diagnostic medicine; machine learning; outcome weighted learning

References

  1. Bartlett, P.L., Jordan, M.I. and McAuliffe, J.D. (2006) Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101, 138-156. - PubMed
  2. Bradley, A.P. (1997) The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30, 1145-1159. - PubMed
  3. Breiman, L. (2001) Random forests. Machine Learning, 45, 5-32. - PubMed
  4. Cai, T. and Dodd, L.E. (2008) Regression analysis for the partial area under the ROC curve. Statistica Sinica, 18, 817-836. - PubMed
  5. Cai, T. and Pepe, M.S. (2002) Semiparametric receiver operating characteristic analysis to evaluate biomarkers for disease. Journal of the American Statistical Association, 97, 1099-1107. - PubMed
  6. Campbell, G. (1994) Advances in statistical methodology for the evaluation of diagnostic and laboratory tests. Statistics in Medicine, 13, 499-508. - PubMed
  7. Claeskens, G., Jing, B.-Y., Peng, L. and Zhou, W. (2003) Empirical likelihood confidence regions for comparison distributions and ROC curves. Canadian Journal of Statistics, 31, 173-190. - PubMed
  8. Cortes, C. and Vapnik, V. (1995) Support-vector networks. Machine Learning, 20, 273-297. - PubMed
  9. Dasgupta, S., Goldberg, Y. and Kosorok, M.R. (2019) Feature elimination in kernel machines in moderately high dimensions. The Annals of Statistics, 47, 497-526. - PubMed
  10. Etzioni, R., Pepe, M., Longton, G., Hu, C. and Goodman, G. (1999) Incorporating the time dimension in receiver operating characteristic curves: a case study of prostate cancer. Medical Decision Making, 19, 242-251. - PubMed
  11. Fan, C., Prat, A., Parker, J.S., Liu, Y., Carey, L.A., Troester, M.A. and Perou, C.M. (2011) Building prognostic models for breast cancer patients using clinical variables and hundreds of gene expression signatures. BMC Medical Genomics, 4, 3. - PubMed
  12. Horváth, L., Horváth, Z. and Zhou, W. (2008) Confidence bands for ROC curves. Journal of Statistical Planning and Inference, 138, 1894-1904. - PubMed
  13. Jensen, K., Müller, H.-H. and Schäfer, H. (2000) Regional confidence bands for ROC curves. Statistics in Medicine, 19, 493-509. - PubMed
  14. Jiang, B., Zhang, X. and Cai, T. (2008) Estimating the confidence interval for prediction errors of support vector machine classifiers. Journal of Machine Learning Research, 9, 521-540. - PubMed
  15. Karatzoglou, A., Smola, A., Hornik, K. and Zeileis, A. (2004) kernlab - an S4 package for kernel methods in R. Journal of Statistical Software, 11, 1-20. - PubMed
  16. Kosorok, M.R. (2008) Introduction to Empirical Processes and Semiparametric Inference. New York, NY: Springer Science & Business Media. - PubMed
  17. Krzyżak, A., Linder, T. and Lugosi, G. (1996) Nonparametric estimation and classification using radial basis function nets and empirical risk minimization. IEEE Transactions on Neural Networks, 7, 475-487. - PubMed
  18. Lin, H.-T., Lin, C.-J. and Weng, R.C. (2007) A note on Platt's probabilistic outputs for support vector machines. Machine Learning, 68, 267-276. - PubMed
  19. Lin, Y. (2002) Support vector machines and the Bayes rule in classification. Data Mining and Knowledge Discovery, 6, 259-275. - PubMed
  20. López-Ratón, M., Rodríguez-Álvarez, M.X., Cadarso-Suárez, C. and Gude-Sampedro, F. (2014) Optimalcutpoints: an R package for selecting optimal cutpoints in diagnostic tests. Journal of Statistical Software, 61, 1-36. - PubMed
  21. Ma, G. and Hall, W. (1993) Confidence bands for receiver operating characteristic curves. Medical Decision Making, 13, 191-197. - PubMed
  22. McIntosh, M.W. and Pepe, M.S. (2002) Combining several screening tests: optimality of the risk score. Biometrics, 58, 657-664. - PubMed
  23. Pepe, M.S. (1997) A regression modelling framework for receiver operating characteristic curves in medical diagnostic testing. Biometrika, 84, 595-608. - PubMed
  24. Pepe, M.S. (2000) An interpretation for the ROC curve and inference using GLM procedures. Biometrics, 56, 352-359. - PubMed
  25. Pepe, M.S. (2003) The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford: Oxford University Press. - PubMed
  26. Platt, J. (1999) Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, 10, 61-74. - PubMed
  27. Provost, F. and Fawcett, T. (1998) Robust classification systems for imprecise environments. Proceedings of the Fifteenth National Conference on Artificial Intelligence, 706-713. - PubMed
  28. Provost, F.J. and Fawcett, T. (1997) Analysis and visualization of classifier performance: comparison under imprecise class and cost distributions. Proceedings of the Third International Conference on Knowledge Discovery and Data Mining, 43-48. - PubMed
  29. R Core Team (2016) R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. - PubMed
  30. Shebl, F.M., El-Kamary, S.S., Saleh, D.A., Abdel-Hamid, M., Mikhail, N., Allam, A., et al., (2009) Prospective cohort study of mother-to-infant infection and clearance of hepatitis C in rural Egyptian villages. Journal of Medical Virology, 81, 1024-1031. - PubMed
  31. Shin, S.J., Wu, Y., Zhang, H.H. and Liu, Y. (2014) Probability-enhanced sufficient dimension reduction for binary classification. Biometrics, 70, 546-555. - PubMed
  32. Spackman, K.A. (1989) Signal detection theory: valuable tools for evaluating inductive learning. Proceedings of the Sixth International Workshop on Machine Learning, 160-163. - PubMed
  33. Steinwart, I. and Christmann, A. (2008) Support Vector Machines. New York, NY: Springer Science & Business Media. - PubMed
  34. Tibshirani, R. (1996) Regression shrinkage and selection via the LASSO. Journal of the Royal Statistical Society, Series B, 58, 267-288. - PubMed
  35. Vapnik, V. (1998) Statistical Learning Theory. New York, NY: Wiley. - PubMed
  36. Veropoulos, K., Campbell, C., and Cristianini, N. (1999) Controlling the sensitivity of support vector machines. Proceedings of the International Joint Conference on AI, 55-60. - PubMed
  37. Zhang, T. (2004) Statistical behavior and consistency of classification methods based on convex risk minimization. Annals of Statistics, 32, 56-85. - PubMed
  38. Zhao, Y., Zeng, D., Rush, A.J. and Kosorok, M.R. (2012) Estimating individualized treatment rules using outcome weighted learning. Journal of the American Statistical Association, 107, 1106-1118. - PubMed
  39. Zhou, X.-H., McClish, D.K. and Obuchowski, N.A. (2002) Statistical Methods in Diagnostic Medicine. New York, NY: John Wiley & Sons. - PubMed

Publication Types

Grant support