Display options
Cite
Clark RD, Liang W, Lee AC, et al. Using beta binomials to estimate classification uncertainty for ensemble models. J Cheminform. 2014;6:34doi: 10.1186/1758-2946-6-34.
Clark, R. D., Liang, W., Lee, A. C., Lawless, M. S., Fraczkiewicz, R., & Waldman, M. (2014). Using beta binomials to estimate classification uncertainty for ensemble models. Journal of cheminformatics, 634. https://doi.org/10.1186/1758-2946-6-34
Clark, Robert D, et al. "Using beta binomials to estimate classification uncertainty for ensemble models." Journal of cheminformatics vol. 6 (2014): 34. doi: https://doi.org/10.1186/1758-2946-6-34
Clark RD, Liang W, Lee AC, Lawless MS, Fraczkiewicz R, Waldman M. Using beta binomials to estimate classification uncertainty for ensemble models. J Cheminform. 2014 Jun 22;6:34. doi: 10.1186/1758-2946-6-34. eCollection 2014. PMID: 24987464; PMCID: PMC4076254.
Copy Download .nbib Format: NLM
Share it on

J Cheminform. 2014 Jun 22;6:34. doi: 10.1186/1758-2946-6-34. eCollection 2014.

Using beta binomials to estimate classification uncertainty for ensemble models.

Journal of cheminformatics

Robert D Clark, Wenkel Liang, Adam C Lee, Michael S Lawless, Robert Fraczkiewicz, Marvin Waldman

Affiliations

  1. Department of Life Sciences, Simulations Plus, Inc., 45205 10th Street West, Lancaster, CA 93534, USA.

PMID: 24987464 PMCID: PMC4076254 DOI: 10.1186/1758-2946-6-34

Abstract

BACKGROUND: Quantitative structure-activity (QSAR) models have enormous potential for reducing drug discovery and development costs as well as the need for animal testing. Great strides have been made in estimating their overall reliability, but to fully realize that potential, researchers and regulators need to know how confident they can be in individual predictions.

RESULTS: Submodels in an ensemble model which have been trained on different subsets of a shared training pool represent multiple samples of the model space, and the degree of agreement among them contains information on the reliability of ensemble predictions. For artificial neural network ensembles (ANNEs) using two different methods for determining ensemble classification - one using vote tallies and the other averaging individual network outputs - we have found that the distribution of predictions across positive vote tallies can be reasonably well-modeled as a beta binomial distribution, as can the distribution of errors. Together, these two distributions can be used to estimate the probability that a given predictive classification will be in error. Large data sets comprised of logP, Ames mutagenicity, and CYP2D6 inhibition data are used to illustrate and validate the method. The distributions of predictions and errors for the training pool accurately predicted the distribution of predictions and errors for large external validation sets, even when the number of positive and negative examples in the training pool were not balanced. Moreover, the likelihood of a given compound being prospectively misclassified as a function of the degree of consensus between networks in the ensemble could in most cases be estimated accurately from the fitted beta binomial distributions for the training pool.

CONCLUSIONS: Confidence in an individual predictive classification by an ensemble model can be accurately assessed by examining the distributions of predictions and errors as a function of the degree of agreement among the constituent submodels. Further, ensemble uncertainty estimation can often be improved by adjusting the voting or classification threshold based on the parameters of the error distribution. Finally, the profiles for models whose predictive uncertainty estimates are not reliable provide clues to that effect without the need for comparison to an external test set.

Keywords: ANNE; Artificial neural network ensemble; Classification; Confidence; Error estimation; Predictive value; QSAR; Uncertainty

References

  1. Altern Lab Anim. 2013 Mar;41(1):111-25 - PubMed
  2. J Cheminform. 2009 Jul 14;1(1):11 - PubMed
  3. Cancer. 1950 Jan;3(1):32-5 - PubMed
  4. Nat Biotechnol. 2009 Nov;27(11):1050-5 - PubMed
  5. J Chem Inf Model. 2008 Sep;48(9):1733-46 - PubMed
  6. J Comput Aided Mol Des. 2013 Mar;27(3):203-19 - PubMed
  7. Environ Health Perspect. 2004 Aug;112(12):1249-54 - PubMed
  8. Biometrics. 1999 Mar;55(1):149-55 - PubMed
  9. J Chem Inf Model. 2013 Nov 25;53(11):2837-50 - PubMed
  10. J Chem Inf Comput Sci. 2000 Jul;40(4):1046-51 - PubMed
  11. J Mol Graph Model. 2008 Jun;26(8):1315-26 - PubMed
  12. Artif Intell Med. 2007 Nov;41(3):197-207 - PubMed
  13. J Chem Inf Comput Sci. 2003 Mar-Apr;43(2):429-34 - PubMed
  14. Environ Health Perspect. 2003 Aug;111(10):1361-75 - PubMed
  15. Mol Inform. 2012 Nov 1;31(11-12):783-792 - PubMed
  16. J Pharm Sci. 2009 Mar;98(3):861-93 - PubMed
  17. Mol Inform. 2014 Jan;33(1):26-35 - PubMed
  18. J Chem Inf Model. 2013 Feb 25;53(2):368-83 - PubMed
  19. J Chem Inf Model. 2010 Dec 27;50(12):2094-111 - PubMed
  20. J Chem Inf Model. 2009 Sep;49(9):2077-81 - PubMed
  21. Stat Med. 2004 May 15;23(9):1351-75 - PubMed
  22. J Toxicol Environ Health. 1988;25(1):135-48 - PubMed

Publication Types