• Consistency of ranking was evaluated as new measure for prediction model stability: longitudinal cohort study.

      Li, Yan; Sperrin, Matthew; Ashcroft, Darren M; van Staa, Tjeerd Pieter; email: tjeerd.vanstaa@manchester.ac.uk (2021-07-02)
      Clinical risk prediction models are generally assessed on population level with a lack of measures that evaluate their stability at predicting risks of individual patients. This study evaluated the use of ranking as a measure to assess individual level stability between risk prediction models. A large patient cohort (3.66 million patients with 0.11 million cardiovascular events) extracted from the Clinical Practice Research Datalink was used in the exemplar of cardiovascular disease risk prediction. It was found that 15 models (including machine learning and statistical models) had similar population-level model performance (C statistics about 0.88). For patients with high absolute risks, the models were more consistent in ranking of risk predictions (interquartile range (IQR) of differences in rank percentiles -0.6 to 1.0), but inconsistent in absolute risk (IQR of differences in absolute risk -18.8 to 9.0). At low risk, the reverse was true with inconsistent ranking but more consistent absolute risk. Consistency of ranking of individual risk predictions is a useful measure to assess risk prediction models providing complementary information to absolute risk stability. Model developing guidelines including "TRIPOD" and "PROBAST" should incorporate ranking to assess individual level stability between risk prediction models. [Abstract copyright: Copyright © 2021. Published by Elsevier Inc.]
    • Immune infiltrate diversity confers a good prognosis in follicular lymphoma.

      Tsakiroglou, Anna-Maria; Astley, Susan; Dave, Manàs; Fergie, Martin; Harkness, Elaine; Rosenberg, Adeline; Sperrin, Matthew; West, Catharine; Byers, Richard; orcid: 0000-0003-0796-0365; email: richard.byers@mft.nhs.uk; Linton, Kim; email: kim.linton@manchester.ac.uk (2021-04-30)
      Follicular lymphoma (FL) prognosis is influenced by the composition of the tumour microenvironment. We tested an automated approach to quantitatively assess the phenotypic and spatial immune infiltrate diversity as a prognostic biomarker for FL patients. Diagnostic biopsies were collected from 127 FL patients initially treated with rituximab-based therapy (52%), radiotherapy (28%), or active surveillance (20%). Tissue microarrays were constructed and stained using multiplex immunofluorescence (CD4, CD8, FOXP3, CD21, PD-1, CD68, and DAPI). Subsequently, sections underwent automated cell scoring and analysis of spatial interactions, defined as cells co-occurring within 30 μm. Shannon's entropy, a metric describing species biodiversity in ecological habitats, was applied to quantify immune infiltrate diversity of cell types and spatial interactions. Immune infiltrate diversity indices were tested in multivariable Cox regression and Kaplan-Meier analysis for overall (OS) and progression-free survival (PFS). Increased diversity of cell types (HR = 0.19 95% CI 0.06-0.65, p = 0.008) and cell spatial interactions (HR = 0.39, 95% CI 0.20-0.75, p = 0.005) was associated with favourable OS, independent of the Follicular Lymphoma International Prognostic Index. In the rituximab-treated subset, the favourable trend between diversity and PFS did not reach statistical significance. Multiplex immunofluorescence and Shannon's entropy can objectively quantify immune infiltrate diversity and generate prognostic information in FL. This automated approach warrants validation in additional FL cohorts, and its applicability as a pre-treatment biomarker to identify high-risk patients should be further explored. The multiplex image dataset generated by this study is shared publicly to encourage further research on the FL microenvironment.
    • Impact of sample size on the stability of risk scores from clinical prediction models: a case study in cardiovascular disease

      Pate, Alexander; orcid: 0000-0002-0849-3458; email: alexander.pate@manchester.ac.uk; Emsley, Richard; Sperrin, Matthew; Martin, Glen P.; van Staa, Tjeerd (BioMed Central, 2020-09-09)
      Abstract: Background: Stability of risk estimates from prediction models may be highly dependent on the sample size of the dataset available for model derivation. In this paper, we evaluate the stability of cardiovascular disease risk scores for individual patients when using different sample sizes for model derivation; such sample sizes include those similar to models recommended in the national guidelines, and those based on recently published sample size formula for prediction models. Methods: We mimicked the process of sampling N patients from a population to develop a risk prediction model by sampling patients from the Clinical Practice Research Datalink. A cardiovascular disease risk prediction model was developed on this sample and used to generate risk scores for an independent cohort of patients. This process was repeated 1000 times, giving a distribution of risks for each patient. N = 100,000, 50,000, 10,000, Nmin (derived from sample size formula) and Nepv10 (meets 10 events per predictor rule) were considered. The 5–95th percentile range of risks across these models was used to evaluate instability. Patients were grouped by a risk derived from a model developed on the entire population (population-derived risk) to summarise results. Results: For a sample size of 100,000, the median 5–95th percentile range of risks for patients across the 1000 models was 0.77%, 1.60%, 2.42% and 3.22% for patients with population-derived risks of 4–5%, 9–10%, 14–15% and 19–20% respectively; for N = 10,000, it was 2.49%, 5.23%, 7.92% and 10.59%, and for N using the formula-derived sample size, it was 6.79%, 14.41%, 21.89% and 29.21%. Restricting this analysis to models with high discrimination, good calibration or small mean absolute prediction error reduced the percentile range, but high levels of instability remained. Conclusions: Widely used cardiovascular disease risk prediction models suffer from high levels of instability induced by sampling variation. Many models will also suffer from overfitting (a closely linked concept), but at acceptable levels of overfitting, there may still be high levels of instability in individual risk. Stability of risk estimates should be a criterion when determining the minimum sample size to develop models.
    • Missing data was handled inconsistently in UK prediction models: a review of method used.

      Tsvetanova, Antonia; email: antonia.tsvetanova@manchester.ac.uk; Sperrin, Matthew; Peek, Niels; Buchan, Iain; Hyland, Stephanie; Martin, Glen P (2021-09-11)
      No clear guidance exists on handling missing data at each stage of developing, validating and implementing a clinical prediction model (CPM). We aimed to review the approaches to handling missing data that underly the CPMs currently recommended for use in UK healthcare. A descriptive cross-sectional meta-epidemiological study aiming to identify CPMs recommended by the National Institute for Health and Care Excellence (NICE), which summarized how missing data is handled across their pipelines. 23 CPMs were included through 'sampling strategy'. Six missing data strategies were identified: complete case analysis (CCA), multiple imputation, imputation of mean values, k-nearest neighbours imputation, using an additional category for missingness, considering missing values as risk-factor-absent. 52% of the development articles and 48% of the validation articles did not report how missing data were handled. CCA was the most common approach used for development (40%) and validation (44%). At implementation, 57% of the CPMs required complete data entry, whilst 43% allowed missing values. 3 CPMs had consistent paths in their pipelines. A broad variety of methods for handling missing data underly the CPMs currently recommended for use in UK healthcare. Missing data handling strategies were generally inconsistent. Better quality assurance of CPMs needs greater clarity and consistency in handling of missing data. [Abstract copyright: Copyright © 2021. Published by Elsevier Inc.]