Skip to main content
EURASIP Journal on Bioinformatics and Systems Biology
Download PDF
  • Research Article
  • Open access
  • Published:

Decorrelation of the True and Estimated Classifier Errors in High-Dimensional Settings

EURASIP Journal on Bioinformatics and Systems Biology volume 2007, Article number: 38473 (2007) Cite this article

Abstract

The aim of many microarray experiments is to build discriminatory diagnosis and prognosis models. Given the huge number of features and the small number of examples, model validity which refers to the precision of error estimation is a critical issue. Previous studies have addressed this issue via the deviation distribution (estimated error minus true error), in particular, the deterioration of cross-validation precision in high-dimensional settings where feature selection is used to mitigate the peaking phenomenon (overfitting). Because classifier design is based upon random samples, both the true and estimated errors are sample-dependent random variables, and one would expect a loss of precision if the estimated and true errors are not well correlated, so that natural questions arise as to the degree of correlation and the manner in which lack of correlation impacts error estimation. We demonstrate the effect of correlation on error precision via a decomposition of the variance of the deviation distribution, observe that the correlation is often severely decreased in high-dimensional settings, and show that the effect of high dimensionality on error estimation tends to result more from its decorrelating effects than from its impact on the variance of the estimated error. We consider the correlation between the true and estimated errors under different experimental conditions using both synthetic and real data, several feature-selection methods, different classification rules, and three error estimators commonly used (leave-one-out cross-validation, -fold cross-validation, and .632 bootstrap). Moreover, three scenarios are considered: (1) feature selection, (2) known-feature set, and (3) all features. Only the first is of practical interest; however, the other two are needed for comparison purposes. We will observe that the true and estimated errors tend to be much more correlated in the case of a known feature set than with either feature selection or using all features, with the better correlation between the latter two showing no general trend, but differing for different models.

[12345678910111213141516171819202122]

References

  1. Jain A, Zongker D: Feature selection: evaluation, application, and small sample performance. IEEE Transactions on Pattern Analysis and Machine Intelligence 1997, 19(2):153-158. 10.1109/34.574797

    Article  Google Scholar 

  2. Hughes G: On the mean accuracy of statistical pattern recognizers. IEEE Transactions on Information Theory 1968, 14(1):55-63. 10.1109/TIT.1968.1054102

    Article  Google Scholar 

  3. Hua J, Xiong Z, Lowey J, Suh E, Dougherty ER: Optimal number of features as a function of sample size for various classification rules. Bioinformatics 2005, 21(8):1509-1515. 10.1093/bioinformatics/bti171

    Article  Google Scholar 

  4. Dougherty ER: Small sample issues for microarray-based classification. Comparative and Functional Genomics 2001, 2(1):28-34. 10.1002/cfg.62

    Article  Google Scholar 

  5. Braga-Neto UM, Dougherty ER: Is cross-validation valid for small-sample microarray classification? Bioinformatics 2004, 20(3):374-380. 10.1093/bioinformatics/btg419

    Article  Google Scholar 

  6. Molinaro AM, Simon R, Pfeiffer RM: Prediction error estimation: a comparison of resampling methods. Bioinformatics 2005, 21(15):3301-3307. 10.1093/bioinformatics/bti499

    Article  Google Scholar 

  7. Xiao Y, Hua J, Dougherty ER: Quantification of the impact of feature selection on the variance of cross-validation error estimation. EURASIP Journal on Bioinformatics and Systems Biology 2007., 2007: 11 pages

    Google Scholar 

  8. Braga-Neto U, Hashimoto R, Dougherty ER, Nguyen DV, Carroll RJ: Is cross-validation better than resubstitution for ranking genes? Bioinformatics 2004, 20(2):253-258. 10.1093/bioinformatics/btg399

    Article  Google Scholar 

  9. Sima C, Braga-Neto U, Dougherty ER: Superior feature-set ranking for small samples using bolstered error estimation. Bioinformatics 2005, 21(7):1046-1054. 10.1093/bioinformatics/bti081

    Article  Google Scholar 

  10. Sima C, Attoor S, Braga-Neto U, Lowey J, Suh E, Dougherty ER: Impact of error estimation on feature-selection algorithms. Pattern Recognition 2005, 38(12):2472-2482. 10.1016/j.patcog.2005.03.026

    Article  Google Scholar 

  11. Zhou X, Mao KZ: The ties problem resulting from counting-based error estimators and its impact on gene selection algorithms. Bioinformatics 2006, 22(20):2507-2515. 10.1093/bioinformatics/btl438

    Article  Google Scholar 

  12. Sima C, Dougherty ER: What should be expected from feature selection in small-sample settings. Bioinformatics 2006, 22(19):2430-2436. 10.1093/bioinformatics/btl407

    Article  Google Scholar 

  13. Mehta T, Tanik M, Allison DB: Towards sound epistemological foundations of statistical methods for high-dimensional biology. Nature Genetics 2004, 36(9):943-947. 10.1038/ng1422

    Article  Google Scholar 

  14. Dougherty ER, Datta A, Sima C: Research issues in genomic signal processing. IEEE Signal Processing Magazine 2005, 22(6):46-68.

    Article  Google Scholar 

  15. Michiels S, Koscielny S, Hill C: Prediction of cancer outcome with microarrays: a multiple random validation strategy. The Lancet 2005, 365(9458):488-492. 10.1016/S0140-6736(05)17866-0

    Article  Google Scholar 

  16. Dougherty ER, Braga-Neto U: Epistemology of computational biology: mathematical models and experimental prediction as the basis of their validity. Journal of Biological Systems 2006, 14(1):65-90. 10.1142/S0218339006001726

    Article  MATH  Google Scholar 

  17. Braga-Neto U: Fads and fallacies in the name of small-sample microarray classification—a highlight of misunderstanding and erroneous usage in the applications of genomic signal processing. IEEE Signal Processing Magazine 2007, 24(1):91-99.

    Article  Google Scholar 

  18. Dupuy A, Simon RM: Critical review of published microarray studies for cancer outcome and guidelines on statistical analysis and reporting. Journal of the National Cancer Institute 2007, 99(2):147-157. 10.1093/jnci/djk018

    Article  Google Scholar 

  19. Dougherty ER, Hua J, Bittner ML: Validation of computational methods in genomics. Current Genomics 2007, 8(1):1-19. 10.2174/138920207780076956

    Article  Google Scholar 

  20. van de Vijver MJ, He YD, van 't Veer LJ, et al.: A gene-expression signature as a predictor of survival in breast cancer. New England Journal of Medicine 2002, 347(25):1999-2009. 10.1056/NEJMoa021967

    Article  Google Scholar 

  21. Bhattacharjee A, Richards WG, Staunton J, et al.: Classification of human lung carcinomas by mRNA expression profiling reveals distinct adenocarcinoma subclasses. Proceedings of the National Academy of Sciences of the United States of America 2001, 98(24):13790-13795. 10.1073/pnas.191502998

    Article  Google Scholar 

  22. Devroye L, Gyorfi L, Lugosi G: A Probabilistic Theory of Pattern Recognition. Springer, New York, NY, USA; 1996.

    Book  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX, 77843, USA

    Blaise Hanczar & Edward R Dougherty

  2. Laboratoire d'Informatique Medicale et Bio-informatique (Lim&Bio), Universite Paris 13, Bobigny cedex, 93017, France

    Blaise Hanczar

  3. Computational Biology Division, Translational Genomics Research Institute, Phoenix, AZ, 85004, USA

    Jianping Hua & Edward R Dougherty

Authors
  1. Blaise Hanczar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jianping Hua
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Edward R Dougherty
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Blaise Hanczar.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Hanczar, B., Hua, J. & Dougherty, E.R. Decorrelation of the True and Estimated Classifier Errors in High-Dimensional Settings. J Bioinform Sys Biology 2007, 38473 (2007). https://doi.org/10.1155/2007/38473

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1155/2007/38473

Keywords