Heat Map Prediction Matrices

Heat Map Nomograms for Cancer Survival

Non-interventional trials or registries allow some leeway in the collection and reporting of observations and procedures by site investigators, leading to a larger degree of missing data than controlled clinical trials. However, controlled clinical trials (CCT), which tend to have more complete data, have an extensive set of inclusion and exclusion criteria which make their patient populations highly selective and unrepresentative of the typical patient presenting at a clinic. Furthermore, sites at CCTs tend to be research or academic sites rather than regular community sites. Registries typically do not work under these constraints and provide ideal data for the development of prognostic models for the typical patient, whereas the generalizability of models derived from CCTs, for reasons listed above, maybe limited. We will describe in this page the performance of prognostic analyses within the context of real-world non-interventional trials while circumventing the drawback of data incompleteness. Further details and additional SAS and R-Code are described in a publication by me and my colleagues in – Shankar S. Srinivasan, Albert Elion-Mboussa, and Li Hua Yue. Validation, Multivariate Modeling, and the Construction of Heat-Map Prediction Matrices for Survival in the Context of Missing Data © Springer Nature Singapore Pte Ltd. 2018 K. E. Peace et al. (eds.), Biopharmaceutical Applied Statistics Symposium [BASS], ICSA Book Series in Statistics, Vol 3, Chapter 17, https://doi.org/10.1007/978-981-10-7820-0_19. The BASS organization is a non-profit organization headed by Dr Karl E. Peace and supports students pursuing diplomas in statistics.

We describe the construction of a nomogram matrix presentation of the predictions of the prognostic model. Similar model representations have been published in peer-reviewed journals by Vastesaeger and colleagues in rheumatoid arthritis and ankylosing spondylitis but a complete description of the methodology surrounding the modeling and construction of this matrix representation is currently unavailable, especially in the context of missing data. We did separate analyses for early mortality (within 180 days using a logistic regression) and late mortality (2, 3- and 4-year survival using a Cox regression) as early mortality often has a different etiology than later mortality, with co-morbidities dominating the former and disease factors more relevant for the latter. The results for early mortality after the diagnosis of Multiple Myeloma, using our methods, and with complete clinical background and interpretation has been published (Terebelo et al. 2017 AJH, https://doi.org/10.1002/ajh.24796 ). The late mortality prediction model has been presented as a poster at various conferences (see XXX et al ASH) and has been submitted for publication. We provide a screen-shot of the published early mortality matrix at this page. Both of these will be provided at the Connect® Registries website. Access is currently restricted to the sites and investigators on these registries. Note that the late survival version of the nomogram in our BASS Book Chapter is a Beta version. The version to be published uses data after an additional follow-up of more than a year and involves corrections to the model and enhancements to the methodology. While both versions are similar, any Myeloma physician, or a patient working with a physician, should consult the updated version.

Obtaining prognostic variables

We screened variables using univariate Cox and Logistic analyses in the two contexts. We followed it up by a multiple imputation procedure to impute the missing data. Multivariate model selection using the initial univariate screened variables is somewhat complicated with missing data which is multiply imputed, as each imputation can provide a different variable set in the model making it difficult to combine the inferences across imputations. We used an initial crude variable selection step using the methods in Wood et al (2008, Stats in Medicine, 27, 3227–3246) to find a smaller set of viable models likely to contain the most appropriate model. A more rigorous modelling approach using Rubin’s was possible for this more limited candidate set. Using the Bayesian Information Criteria (BIC) together with clinical judgements of the appropriateness of the models, the final variable set in the prediction models were determined. The discussions here are brief and the reader can refer to our BASS Chapter referenced earlier for further detail and code for this section and the ones that follow.

Obtaining the Prediction Heat-Map

Predictions for a logistic model are easy to obtain using the logit model and the intercept and parameter estimates. In Terebelo et al., we estimated probabilities of mortality within 180 days for Myeloma patients. A screen-shot of the heat-map prediction matrix presenting these estimates is shown below. It was designed to show less favorable outcomes in the bottom left corner and more favorable outcomes toward the top right corner of the matrix. We order the variables based on importances assessed by the magnitude of the odds ratios. We move to smaller column or row headers within the larger ones with decreasing magnitudes of effect. The row header predictors have the predictor level with the favorable outcome on top. The column header predictors have the predictor level with the favorable outcome to the right. This generates a blank matrix with column and row headers, which is then populated with the appropriate predictions. Note that the predictions are interpretable only in an aggregate sense. A 40% estimated probability of early mortality corresponding to a patient profile would imply that about 40 patients in 100 with that profile would die early give or take 10 patients – a 95% Confidence Interval of [30.4, 49.6].

For survival over the longer time frame we used a Cox Regression and generated the predictions using methods in Allison (2010). This involved using a dataset with all variable combinations in the baseline statement of the SAS PHREG procedure to generate survival probabilities at every event time in the registry. To obtain a survival probability beyond three years for instance, we would retain the data records corresponding to the event time closest to and less than the three-year time-point (1095 days). The prediction of survival beyond three years for each predictor combination is then estimated as the average of the corresponding 3-year survivals for that combination across imputations.

Internal and External Validation

Internal validation involves the splitting of the dataset into test and training samples. The model obtained in the training sample is evaluated in the test sample. Better estimates of validation indices are obtained when they are obtained through analysis of repeated random splits into test and training samples—a process called bootstrap re-sampling. The validation index we used to measure the predictive ability of our prognostic model was the Harrell’s C-Index (Harrell 2001). This index is interpretable as a concordance probability—the probability that a randomly selected pair of patients, one with a poorer survival outcome than the other, will be correctly differentially identified based on inputting the two patient’s baseline prognostic characteristics in the fitted model. The concordance index can be obtained using the ‘rms’ R Package for both logistic and Cox models. It is useful to report the inflation in the concordance index in the training samples compared to the test sample as it is indicative of an overfit when large. An adjusted C-Index removing the training effect is usually reported.

External validation is conducted using external independent studies. The prediction in the external study is computed using the model derived internally and then compared to actual outcome in the external study. A C-Index is reported as before and results can be depicted graphically by using sub-cohorts in the external study with increasing probabilities of survival and plotting actual probabilities versus predicted probabilities. As before details and R-Code are provided in the BASS book chapter.