The following reporting guidelines were used; Reporting of studies Conducted using Observational Routinely collected Data (RECORD) [17], and the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) [18].
Source of data
The Health Improvement Network (THIN) database of anonymised GP records was used for analysis and has data for over 17 million patients in the UK (with 3.1 million active patients and > 5% coverage) [19]. THIN includes primary care practices which use Vision software and provides demographic information such as sex, age, Townsend deprivation score, diagnoses, symptoms and prescriptions.
The Bowel Cancer Screening System (BCSS) used in the NHS Bowel Cancer Screening Programme (BCSP) is used to identify participants and record test results. There are interconnections between the BCSS and primary care records. The BCSS receives its data originally from GP records for its participants in the relevant age range (through upload to the NHS Information Authority and the NHS Spine). Since 2009–2010 GP practice systems have been able to opt into receiving electronic screening results from the BCSS using the same system as the Pathology Messaging Implementation Programme (PMIP).
An English BCSP cohort was derived using the electronic notifications received from the Bowel Cancer Screening System to GP records. THIN was used to derive this cohort by identifying men and women with automatically received electronic notifications from the BCSP, aged 60–74 years of age and with at least a years’ worth of health records before taking their latest FOBT (to ensure adequate symptomatic information to be identified). This covered a period between 13th May 2009 (the first FOBT screen date) with follow up to 17th January 2017 (the last follow up date). Patients were excluded if they had a previous CRC diagnosis or if they had a high-risk condition (hereditary nonpolyposis colorectal cancer – HNPCC) or familial adenomatous polyposis (FAP)).
Practice eligibility used the latest of the following: one year after the Vision practice software installation, the acceptable mortality recording (AMR) date [20] and the date in which the electronic BCSP notifications started to be received by the practice (the full details of defining this date for each practice will be published elsewhere). Before electronic notifications were received, data may be incomplete, subject to transcription errors or biased towards positive results.
Predictors
Predictors investigated were taken from the interface between the BCSS (previous positive or negative screening results) and GP records (demographics, lifestyle factors, anthropometrics, laboratory test results, symptoms present within the screening population) and were derived from previous research and NICE guidelines [13, 21,22,23,24].
All previous BCSP FOBT results were extracted in order to have an individual’s screening history and originated from the BCSS. Predictors were derived from the GP database using Read code lists (Read Version 2) for 28 clinical features. Clinical lists developed were subject to a double reviewing process for code set validation.
Last recorded entry was used for the following variables: smoking status, alcohol consumption and family history. The TRIPOD guidelines recommend using a continuous variable rather than dichotomising into different groups as this loses additional predictive information [25]. Cut-offs for certain blood tests are employed in clinical practice since it can indicate underlying disease, therefore categorised blood measurements were also considered for: platelet count, ferritin, haemoglobin concentration and mean cell volume. Variables assessed for univariable and multivariable analysis and how they were operationalised are provided in Supplementary Table S1.
Studies have suggested that large proportions of colorectal cancer screening participants have underlying symptoms [26,27,28] despite recommendations and campaigns for symptomatic individuals to visit their GP. Some of these symptoms can be considered ‘low risk, but not no risk’ [29] and are often self-limiting but in combination can indicate underlying disease [13, 14]. Symptoms present within the screening cohort were measured at the time of entry to the study up to 365.25 days before the index date. Drug code lists were generated for 3 types of prescriptions; anti-motility drugs, antispasmodics and laxatives using the British National Formulary and key word searches. Prescriptions were investigated as a proxy to a particular clinical feature as performed in previous research by the authors [13].
Outcome
The index date used for survival analysis was the date of the latest BCSP FOBT result. The outcome was a diagnosis of CRC/polyps up to 2 years after the index date (latest FOBT) recorded in a patient’s record. Two years represents one screening round in the NHS and allows for the clinical identification of interval cancers. The earliest date of diagnosis was used if both polyps and CRCs had been diagnosed within the 2-year follow up.
Sample size
For stable predictions it has been recommended that multivariable models include at least 10 outcome events per degree of freedom [18]. The dataset for multivariable modelling analysis had 1676 CRC and polyp diagnoses and considered 17 degrees of freedom giving 98.59 outcomes per degree of freedom. The dataset for the model with negative FOBTs only included 735 outcome events and considered 16 degrees of freedom giving 45.94 outcomes per degree of freedom.
Statistical analysis
Overview
To identify predictors for CRC/polyps in a BCSP population, the proportion of individuals with particular clinical features was assessed along with the completeness of data. The level of complete/missing data was recorded in order to determine the availability of predictors from primary care records which could contribute to referral algorithms. The risk of CRC/polyps for these 28 clinical features in a screening population was assessed using univariable Cox regression to estimate hazard ratios.
Two risk prediction models were developed (and internally validated) using Cox Regression with a diagnosis of CRC/polyp recorded in a patient’s record as the outcome. For model development, those with red flag symptoms which includes those defined by NICE guidelines for suspected cancer referral were excluded (rectal bleeding, abdominal mass, abnormal rectal exam, change in bowel habit, abdominal pain, weight loss, iron deficiency anaemia (haemoglobin < 12 g/dL for females < 13 g/dL for men, ferritin < 15 μg/L and MCV < 80 fL). In addition, those with a diagnosis of previous polyps or an FOBT result ordered through primary care were excluded.
The first model used a population with both positive and negative FOBT results to determine the absolute probability of CRC for someone who has taken a screening test. This approach could be used to prioritise screening referrals to colonoscopy for those at highest risk. The second model included only patients with a negative FOBT to determine whether other factors could be used to decide whether a person is at sufficient risk to be referred despite a negative result.
Absolute risk predictions were determined from the models for each patient and their personal predictors (covariate pattern). The negative model was applied to a subset of the population who had complete data and 2 year follow up (n = 25,592). A predetermined risk probability cut-off which represents the NICE guidelines risk level of 3% [21], was used for those with a negative result. Test accuracy of the FOBT alone was compared to a strategy of combining the model positives with FOBT positives (sensitivity, specificity, PPV, NPV reported). The number of extra participants who would need lower gastrointestinal (GI) investigations and number of extra polyps/cancers were determined.
Cox regression (time-to-event) was employed over logistic regression due to the longitudinal nature of the data. Individuals have different lengths of follow up on the database (i.e. reach the study end before the outcome occurs, move GP practices, death etc). Patients who are right-censored in this way provide valuable information up to their final point of follow up [30]. Employing survival models is a more efficient use of the data by maximising events at the tail end. Furthermore, the predictions for these models are over a period of two years and it is argued that predictions for time periods over 6 months should consider time-to-event regression modelling [30]. Similar studies using electronic health records for model development and validation in a primary care setting have also used survival analysis aiding comparability of the model in a screening context [15, 16].
Model development
Analyses used Stata SE Version 15.1. Cox regression and multivariable fractional polynomials with backwards elimination was used to develop each model using the ‘mfp’ function in Stata [31, 32]. Age at FOBT and sex were forced into the models due to clinical relevance. Multivariable fractional polynomials (MFPs) allow non-linear relationships with continuous predictors to be modelled [32]. For backwards elimination, a p-value of 0.05 was used to determine whether to keep a predictor in the model (a variable is removed if dropping it from the model causes a non-significant increase in the deviance) [32]. P-values for testing between fractional polynomial models and for assessing interactions was set at 0.05. Interactions included: age and sex, FOBT result and sex, FOBT result and smoking, smoking and sex. When reporting the final model, the Cox Regression coefficients are provided along with bootstrapped standard errors (100 bootstrap replications due to model complexity and size).
Multiple imputation was considered for missing data however the missing data mechanism for the majority of these predictors would be ‘Missing not at random’ (MNAR), consequently complete cases were used for these analyses. For the multivariable models, alcohol consumption was the predictor which limited the sample size (78% recorded for the derived screening cohort). Other variables such as BMI (95.85%) and smoking status (99.44%) were highly complete.
Model performance
The model performance was assessed using Harrell’s C statistic (to measure discrimination or how well predictions separate those with and without the outcome). Calibration of the models was assessed by plotting a calibration curve for the models once adjusted for optimism. Other performance measures assessed included Somers’ D rank correlation (D = 2(C-0.5)) which ranges from − 1 to 1 [33, 34], the D statistic, R2 and adjusted R2.
The optimism of the models was assessed by calculating the heuristic shrinkage factor of Van Houwelingen [35]. To adjust performance statistics for optimism, internal validation was performed using 100 bootstrap replications for the C statistic, c-slope, D statistic and R2. A split sample approach to model development is generally not recommended; bootstrap validation for assessing statistical optimism is preferred, although less of an issue for large sample sizes with sufficient events and lower model complexity [18].
Absolute risk predictions
Predicted probabilities of CRC/polyps were derived for each patient and their covariate pattern. The baseline CRC free survival was combined with the linear predictor to generate individualised predictions. The full risk equations are provided for both the models.
Non-parametric estimation of the CRC free survival was obtained using a zero covariate value and the methods implemented in Stata. CRC free survival for two years was obtained from the Kaplan-Meier curve and accompanying results. The shrunken linear predictor was used to estimate a new baseline CRC free survival (adjusted for optimism) which was estimated non-parametrically at 2 years. The shrunken linear predictor was combined with the baseline CRC free survival to generate risk predictions. In order to obtain an event probability, the result of this was subtracted from 1 to generate the probability of CRC/polyps being diagnosed over a 2 year period.
Clinical implications
The prediction model developed for those with negative FOBTs could be used to increase the low sensitivity of screening [36] by identifying additional patients for referral based on a combination of symptoms and demographic characteristics. The negative FOBT model was applied to a subset of the population who had complete data and 2 year follow up (n = 25,592). Individualised probabilities for CRC/polyps were determined from the model and an appropriate threshold applied for referral. A predetermined probability cut-off (0.0168) which corresponds to the NICE guidelines PPV risk level of 3% [21], was used for those with a negative result (n = 24,297). This was determined by plotting PPV and NPV against different risk probability cut-offs. The ROC curve for this model was generated and the test characteristics (sensitivity, specificity and NPV) reported. The number of extra participants who would need lower gastrointestinal (GI) investigations and number of extra polyps/cancers were determined.
link