Unified ROC Curve Estimator for Diagnosis and Prognosis Studies: The sMSROC Package

Department of Orthopaedics, Dartmouth Health, Lebanon, NH, USA

Faculty of Health Sciences, Universidad Autonoma de Chile, Chile

Department of Statistics, Operational Research and Mathematics Didactics, University of Oviedo, Oviedo (Asturias), Spain

Abstract

The binary classification problem is a hot topic in Statistics. Its close relationship with the diagnosis and the prognosis of diseases makes it crucial in biomedical research. In this context, it is important to identify biomarkers that may help to classify individuals into different classes, for example, diseased vs. not diseased. The Receiver Operating-Characteristic (ROC) curve is a graphical tool commonly used to assess the accuracy of such classification. Given the diverse nature of diagnosis and prognosis problems, the ROC curve estimation has been tackled from separate perspectives in each setting. The Two-stages Mixed-Subjects (sMS) ROC curve estimator fits both scenarios. Besides, it can handle data with missing or incomplete outcome values. This paper introduces the R package sMSROC which implements the sMS ROC estimator, and includes tools that may support researchers in their decision making. Its practical application is illustrated on three real-world datasets.

Anderson-Bergman, Clifford. 2017. “icenReg: Regression Models for Interval Censored Data in R.” Journal of Statistical Software 81 (12): 1–23. https://doi.org/10.18637/jss.v081.i12.

Basu, N. N., S. Ingham, J. Hodson, F. Lalloo, M. Bulman, A. Howell, and D. G. Evans. 2015. “Risk of Contralateral Breast Cancer in BRCA1 and BRCA2 Mutation Carriers: A 30-Year Semi-Prospective Analysis.” Familial Cancer 14 (4): 531–38. https://doi.org/10.1007/s10689-015-9825-9.

Beyene, K. M., and Anouar El Grouch. 2020. “Smoothed Time-Dependent Receiver Operating Characteristic Curve for Right Censored Survival Data.” Statistics in Medicine 39 (24): 3373–96. https://doi.org/10.1002/sim.8671.

Beyene, Kassu Mehari, and Anouar El Ghouch. 2023. cenROC: Estimating Time-Dependent ROC Curve and AUC for Censored Data. https://CRAN.R-project.org/package=cenROC.

Blanche, Paul, Jean-Francois Dartigues, and Helene Jacqmin-Gadda. 2013. “Estimating and Comparing Time-Dependent Areas Under Receiver Operating Characteristic Curves for Censored Event Times with Competing Risks.” Statistics in Medicine 32 (30): 5381–97. https://doi.org/10.1002/sim.5958.

Blanche, P., J. F. Dartigues, and H. Jacqmin-Gadda. 2013. “Review and Comparison of ROC Curve Estimators for a Time-Dependent Outcome with Marker-Dependent Censoring.” Biometrical Journal 55 (5): 687–704. https://doi.org/10.1002/bimj.201200045.

Chambles, L., and G. Diao. 2006. “Estimation of Time-Dependent Area Under ROC Curve for Long-Term Risk Prediction.” Statistics in Medicine 20 (25): 3474–86. https://doi.org/10.1002/sim.2299.

Cox, D. R. 1972. “Regression Models and Life-Tables.” Journal of the Royal Statistical Society. Series B (Methodological) 34 (2): 187–220. https://doi.org/10.1111/j.2517-6161.1972.tb00899.x.

Díaz-Coto, S., N. O. Corral-Blaco, and P. Martínez-Camblor. 2021. “Two-Stage Receiver Operating-Characteristic Curve Estimator for Cohort Studies.” The International Journal of Biostatistics 17: 117–37. https://doi.org/10.1515/ijb-2019-0097.

Díaz-Coto, S., P. Martínez-Camblor, and S. Pérez-Fernández. 2020. “SmoothROCtime: An R Package for Time-Dependent ROC Curve Estimation.” Computational Statistics. https://doi.org/10.1007/s00180-020-00955-7.

Díaz-Coto, Susana, Pablo Martínez-Camblor, and Norberto Octavio Corral-Blanco. 2020. “Cumulative/Dynamic ROC Curve Estimation Under Interval Censorship.” Journal of Statistical Computation and Simulation 90 (9): 1570–90. https://doi.org/10.1080/00949655.2020.1736071.

Epstein, J. I., L. Egevad, M. B. Amin, B. Delahunt, J. R. Srigley, P. A. Humphrey, and the-Grading-Comittee. 2016. “The 2014 International Society of Urological Pathology (ISUP) Consensus Conference on Gleason Grading of Prostatic Carcinoma.” The American Journal of Surgical Pathology 40 (2): 244–52. https://doi.org/10.1097/PAS.0000000000000530.

Etzioni, Ruth, Margaret Pepe, Gary Longton, Chengcheng Hu, and Gary Goodman. 1999. “Incorporating the Time Dimension in Receiver Operating Characteristic Curves: A Case Study of Prostate Cancer.” Medical Decision Making 19 (3): 242–51. https://doi.org/10.1177/0272989X9901900303.

Farrell, Philip M., Beryl J. Rosenstein, Terry B. White, Frank J. Accurso, Carlo Castellani, Garry R. Cutting, Peter R. Durie, et al. 2008. “Guidelines for Diagnosis of Cystic Fibrosis in Newborns Through Older Adults: Cystic Fibrosis Foundation Consensus Report.” The Journal of Pediatrics 153 (2): S4–14. https://doi.org/10.1016/j.jpeds.2008.05.005.

Ferreirós, E. R., C. P. Boissonnet, R. Pizarro, P. F. Merletti, G. Corrado, A. Cagide, and O. O Bazzino. 1999. “Independent Prognostic Value of Elevated C-Reactive Protein in Unstable Angina.” Circulation 100 (19): 1958–63. https://doi.org/10.1161/01.CIR.100.19.1958.

Finkelstein, D. M. 1986. “A Proportional Hazards Model for Interval-Censored Failure Time Data.” Biometrics. 42 (4): 845–54. https://doi.org/10.2307/2530698.

Foucher, Y., P. Daguin, A. Akl, M. Kessler, M. Ladrière, C. Legendre, H. Kreis, et al. 2010. “A Clinical Scoring System Highly Predictive of Long-Term Kidney Graft Survival.” Kidney International 78 (12): 1288–94. https://doi.org/10.1038/ki.2010.232.

Gohel, David, and Panagiotis Skintzos. 2023. Flextable: Functions for Tabular Reporting. https://CRAN.R-project.org/package=flextable.

Gonçalves, L., A. Subtil, M. Rosário Oliveira, and P. De Zea Bermudez. 2014. “ROC Curve Estimation: An Overview.” Statistical Journal 12 (1): 1–20. https://doi.org/10.57805/revstat.v12i1.141.

Hanley, J. A., and B. McNeil. 1982. “The Meaning and Use of the Area Under the Receiver Operating Characteristic (ROC) Curve.” Radiology 20 (143): 29–36. https://doi.org/10.1148/radiology.143.1.7063747.

Harrel, F. E. 2015. Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordonal Regression and Survival Analysis. Springer Series in Statistics. Springer International Publishing.

Harrell Jr, Frank E. 2023. Rms: Regression Modeling Strategies. https://CRAN.R-project.org/package=rms.

Heagerty, P. J., T. Lumley, and M. S. Pepe. 2000. “Time-Dependent ROC Curves for Censored Survival Data and a Diagnostic Marker.” Biometrics. 56 (2): 337–44. https://doi.org/10.1111/j.0006-341x.2000.00337.x.

Heagerty, Patrick J., and Paramita Saha-Chaudhuri. 2022. survivalROC: Time-Dependent ROC Curve Estimation from Censored Survival Data. https://CRAN.R-project.org/package=survivalROC.

Hsieh, Fushing, and Bruce W. Turnbull. 1996. “Nonparametric and Semiparametric Estimation of the Receiver Operating Characteristic Curve.” The Annals of Statistics 24 (1): 25–40. https://doi.org/10.1214/aos/1033066197.

Hung, H., and C. Chiang. 2010. “Optimal Composite Markers for Time-Dependent Receiver Operating Characteristic Curves with Censored Survival Data.” Scandinavian Journal of Statistics 20 (37): 664–79. https://doi.org/10.1111/j.1467-9469.2009.00683.x.

Hurvich, C. M., J. S. Simonoff, and C.-L. Tsai. 1998. “Smoothing Parameter Selection in Nonparametric Regression Using an Improved Akaike Information Criterion.” Journal of the Royal Statistical Society, Series B (Statistical Methodology) 60 (2): 271–93. https://doi.org/10.1111/1467-9868.00125.

Kamarudin, A. N., T. Cox, and R. Kolamunnage-Dona. 2017. “Time-Dependent ROC Curve Analysis in Medical Research: Current Methods and Applications.” BMC Medical Research Methodology 17 (53). https://doi.org/10.1186/s12874-017-0332-6.

Li, L., T. Greene, and B. Hu. 2018. “A Simple Method to Estimate the Time-Dependent Receiver Operating Characteristic Curve and the Area Under the Curve with Right Censored Data.” Statistical Methods in Medical Research 27 (8): 2264–78. https://doi.org/10.1177/0962280216680239.

Li, L., and C. Wu. 2016. tdROC: Non-Parametric Estimation of Time-Dependent ROC Curve for Right Censored Survival Data. https://CRAN.R-project.org/package=tdROC.

Lin, Jiaxing, Yuan Wu, Xiaofei Wang, and Kouros Owzar. 2021. intcensROC: AUC Estimation of Interval Censored Survival Data. https://CRAN.R-project.org/package=intcensROC.

Long, Jianglan, Zhirui Yang, Long Wang, Yumei Han, Cheng Peng, Can Yan, and Dan Yan. 2020. “Metabolite Biomarkers of Type 2 Diabetes Mellitus and Pre-Diabetes: A Systematic Review and Meta-Analysis.” BMC Endocrine Disorders 20 (1): SP174. https://doi.org/10.1186/s12902-020-00653-x.

López-Ratón, Mónica, María Xosé Rodríguez-Álvarez, Carmen Cadarso-Suárez, and Francisco Gude-Sampedro. 2014. “OptimalCutpoints: An R Package for Selecting Optimal Cutpoints in Diagnostic Tests.” Journal of Statistical Software 61 (8): 1–36. https://doi.org/10.18637/jss.v061.i08.

Martínez-Camblor, P. 2011. “Nonparametric cutoff point estimation for diagnostic decisions with weighted errors.” Revista Colombiana de Estadística 34 (1): 133–46. https://doi.org/10.15446/rce.

Martínez-Camblor, P., G. F. Bayón, and S. Pérez-Fernández. 2016. “Cumulative/Dynamic ROC Curve Estimation.” Journal of Statistical Computation and Simulation 86 (17): 3582–94. https://doi.org/10.1080/00949655.2016.1175442.

Martínez-Camblor, P., and J. C. Pardo-Fernández. 2018. “Smooth Time-Dependent Receiver Operating Characteristic Curve Estimators.” Statistical Methods in Medical Research 27 (3): 651–74. https://doi.org/10.1177/0962280217740786.

Microsoft, and Steve Weston. 2022. Foreach: Provides Foreach Looping Construct. https://CRAN.R-project.org/package=foreach.

Ni, Li, and Xander H. Wehrens. 2018. “Cardiac Troponin I - More Than a Biomarker for Myocardial Ischemia?” Annals of Translational Medicine Suppl 1 (6): S17. https://doi.org/10.21037/atm.2018.09.07.

Pepe, M. S. 2003. The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford Statistical Sciences Series.

Pérez-Fernández, S. 2017. nsROC: Non-Standard ROC Curve Analysis. https://CRAN.R-project.org/package=nsROC.

Pérez-Fernández, S., P. Martínez-Camblor, P. Filzmoser, and N. Corral. 2018. “NsROC: An R Package for Non-Standard ROC Curve Analysis.” The R Journal 10 (2): 55–77. https://doi.org/10.32614/RJ-2018-043.

Potapov, Sergej, Werner Adler, and Matthias Schmid. 2023. survAUC: Estimators of Prediction Accuracy for Time-to-Event Data. https://CRAN.R-project.org/package=survAUC.

Robin, X., N. Turck, A. Hainard, N. Tiberti, F. Lisacek, J. C. Sánchez, and M. Müller. 2011. “PROC: An Open-Source Package for R and S+ to Analyze and Compare ROC Curves.” BMC Bioinformatics 12 (3): 77. https://doi.org/10.1186/1471-2105-12-77.

Rodríguez-Álvarez, María Xosé, and Vanda Inácio. 2021. “ROCnReg: An R Package for Receiver Operating Characteristic Curve Inference with and Without Covariates.” The R Journal 13: 525. https://doi.org/10.32614/RJ-2021-066.

Sachs, M. C. 2017. “PlotROC: A Tool for Plotting ROC Curves.” Journal of Statistical Software 79 (2): 1–19. https://doi.org/10.18637/jss.v079.c02.

Sing, T., O. Sander, N. Beerenwinkel, and T. Lengauer. 2005. “ROCR: Visualizing Classifier Performance in R.” Bioinformatics 21 (20): 7881. https://doi.org/10.1093/bioinformatics/bti623.

Song, X., and X. H. Zhou. 2008. “A Semiparametric Approach for the Covariate-Specific ROC Curve with Survival Outcome.” Statistica Sinica 18: 947–65. http://www.jstor.org/stable/24308524.

Terry M. Therneau, and Patricia M. Grambsch. 2000. Modeling Survival Data: Extending the Cox Model. New York: Springer.

Uno, H., T. Cai, L. Tian, and L. J. Wei. 2007. “Evaluating Prediction Rules for t-Year Survivors with Censored Regression Models.” Journal of the American Statistics Association 478 (102): 527–37. https://doi.org/10.1198/016214507000000149.

Vidal-Castiñeira, Jose Ramón, Antonio López-Vázquez, Paula Díaz-Bulnes, Susana Díaz-Coto, Leonardo Márquez-Kisinousky, Jesús Martínez-Borra, Carmen A. Navascues, et al. 2020. “Genetic Contribution of Endoplasmic Reticulum Aminopeptidase 1 Polymorphisms to Liver Fibrosis Progression in Patients with HCV Infection.” Journal of Molecular Medicine 98: 1245–54. https://doi.org/10.1007/s00109-020-01948-1.

Wickham, Hadley. 2016. Ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York. https://ggplot2.tidyverse.org.

Willems, J. P., J. T. Saunders, D. E. Hunt, and J. B. Schorling. 1997. “Prevalence of Coronary Heart Disease Risk Factors Among Rural Blacks: A Community-Based Study.” Southern Medical Journal 90 (8): 814–20. https://doi.org/10.1097/00007611-199708000-00008.

Wu, Y., X. Wang, J. Lin, J. Beilin, and K. Owzar. 2020. “Predictive Accuracy of Markers or Risk Scores for Interval Censored Survival Data.” Statistics in Medicine 39 (18): 2437–46. https://doi.org/10.1002/sim.8547.

Zhou, X.-H., N. A. Obuchowski, and D. K. McClish. 2002. Statistical Methods in Diagnostic Medicine. Wiley Blackwell, New York.