Comparing the performance of statistical, machine learning, and deep learning algorithms to predict time-to-event: A simulation study for conversion to mild cognitive impairment
Authors
Billichová, MartinaCoan, Lauren Joyce
Czanner, Silvester
Kováčová, Monika
Sharifian, Fariba
Czanner, Gabriela
Affiliation
Slovak University of Technology in Bratislava; Liverpool John Moores University; University of Chester;Publication Date
2024-01-22
Metadata
Show full item recordAbstract
Mild Cognitive Impairment (MCI) is a condition characterized by a decline in cognitive abilities, specifically in memory, language, and attention, that is beyond what is expected due to normal aging. Detection of MCI is crucial for providing appropriate interventions and slowing down the progression of dementia. There are several automated predictive algorithms for prediction using time-to-event data, but it is not clear which is best to predict the time to conversion to MCI. There is also confusion if algorithms with fewer training weights are less accurate. We compared three algorithms, from smaller to large numbers of training weights: a statistical predictive model (Cox proportional hazards model, CoxPH), a machine learning model (Random Survival Forest, RSF), and a deep learning model (DeepSurv). To compare the algorithms under different scenarios, we created a simulated dataset based on the Alzheimer NACC dataset. We found that the CoxPH model was among the best-performing models, in all simulated scenarios. In a larger sample size (n = 6,000), the deep learning algorithm (DeepSurv) exhibited comparable accuracy (73.1%) to the CoxPH model (73%). In the past, ignoring heterogeneity in the CoxPH model led to the conclusion that deep learning methods are superior. We found that when using the CoxPH model with heterogeneity, its accuracy is comparable to that of DeepSurv and RSF. Furthermore, when unobserved heterogeneity is present, such as missing features in the training, all three models showed a similar drop in accuracy. This simulation study suggests that in some applications an algorithm with a smaller number of training weights is not disadvantaged in terms of accuracy. Since algorithms with fewer weights are inherently easier to explain, this study can help artificial intelligence research develop a principled approach to comparing statistical, machine learning, and deep learning algorithms for time-to-event predictions.Citation
Billichová, M., Coan, L. J., Czanner, S., Kováčová, M., Sharifian, F., & Czanner, G. (2024). Comparing the performance of statistical, machine learning, and deep learning algorithms to predict time-to-event: A simulation study for conversion to mild cognitive impairment. Plos one, 19(1), e0297190. http://dx.doi.org/10.1371/journal.pone.0297190Publisher
Public Library of Science (PLoS)Journal
PLoS ONEAdditional Links
http://dx.doi.org/10.1371/journal.pone.0297190Type
ArticleLanguage
enISSN
1932-6203EISSN
1932-6203Sponsors
Agentúra na podporu výskumu a vývojaae974a485f413a2113503eed53cd6c53
10.1371/journal.pone.0297190
Scopus Count
Collections
Except where otherwise noted, this item's license is described as https://creativecommons.org/licenses/by/4.0/