A simplified clinical prediction rule for prognosticating independent walking after spinal cord injury: a prospective study from a Canadian multicenter spinal cord injury registry.

Traumatic spinal cord injury (SCI) is a debilitating condition with limited treatment options for neurologic or functional recovery. The ability to predict the prognosis of walking post injury with emerging prediction models could aid in rehabilitation strategies and reintegration into the community. To revalidate an existing clinical prediction model for independent ambulation (van Middendorp et al., 2011) using acute and long-term post-injury follow-up data, and to investigatethe accuracy of a simplified model using prospectively collected data from a Canadian multicenter SCI database, the Rick Hansen Spinal Cord Injury Registry (RHSCIR). Prospective cohort study. The analysis cohort consisted of 278 adult individuals with traumatic SCI enrolled in the RHSCIR for whom complete neurologic examination data and Functional Independence Measure (FIM) outcome data were available. The FIM locomotor score was used to assess independent walking ability (defined as modified or complete independence in walk or combined walk and wheelchair modality) at 1-year follow-up for each participant. A logistic regression (LR) model based on age and four neurologic variables was applied to our cohort of 278 RHSCIR participants. Additionally, a simplified LR model was created. The Hosmer-Lemeshow goodness of fit test was used to check if the predictive model is applicable to our data set. The performance of the model was verified by calculating the area under the receiver operating characteristic curve (AUC). The accuracy of the model was tested using a cross-validation technique. This study was supported by a grant from The Ottawa Hospital Academic Medical Organization ($50,000 over 2 years). The RHSCIR is sponsored by the Rick Hansen Institute and is supported by funding from Health Canada, Western Economic Diversification Canada, and the provincial governments of Alberta, British Columbia, Manitoba, and Ontario. ET and JP report receiving grants from the Rick Hansen Institute (approximately $60,000 and $30,000 per year, respectively). DMR reports receiving remuneration for consulting services provided to Palladian Health, LLC and Pacira Pharmaceuticals, Inc ($20,000-$30,000 annually), although neither relationship presents a potential conflict of interest with the submitted work. KEH received a grant for involvement in the present study from the Government of Canada as part of the Canada Summer Jobs Program ($3,000). JP reports receiving an educational grant from Medtronic Canada outside of the submitted work ($75,000 annually). TM reports receiving educational fellowship support from AO Spine, AO Trauma, and Medtronic; however, none of these relationships are financial in nature. All remaining authors have no conflicts of interest to disclose. The fitted prediction model generated 85% overall classification accuracy, 79% sensitivity, and 90% specificity. The prediction model was able to accurately classify independent walking ability (AUC 0.889, 95% confidence interval [CI] 0.846-0.933, p<.001) compared with the existing prediction model, despite the use of a different outcome measure (FIM vs. Spinal Cord Independence Measure) to qualify walking ability. A simplified, three-variable LR model based on age and two neurologic variables had an overall classification accuracy of 84%, with 76% sensitivity and 90% specificity, demonstrating comparable accuracy with its five-variable prediction model counterpart. The AUC was 0.866 (95% CI 0.816-0.916, p<.01), only marginally less than that of the existing prediction model. A simplified predictive model with similar accuracy to a more complex model for predicting independent walking was created, which improves utility in a clinical setting. Such models will allow clinicians to better predict the prognosis of ambulation in individuals who have sustained a traumatic SCI.

Hicks KE ，Zhao Y ，Fallah N ，Rivers CS ，Noonan VK ，Plashkes T ，Wai EK ，Roffey DM ，Tsai EC ，Paquet J ，Attabib N ，Marion T ，Ahn H ，Phan P ，RHSCIR Network ... -  《-》

Development of an unsupervised machine learning algorithm for the prognostication of walking ability in spinal cord injury patients.

Traumatic spinal cord injury can have a dramatic effect on a patient's life. The degree of neurologic recovery greatly influences a patient's treatment and expected quality of life. This has resulted in the development of machine learning algorithms (MLA) that use acute demographic and neurologic information to prognosticate recovery. The van Middendorp et al. (2011) (vM) logistic regression (LR) model has been established as a reference model for the prediction of walking recovery following spinal cord injury as it has been validated within many different countries. However, an examination of the way in which these prediction models are evaluated is warranted. The area under the receiver operators curve (AUROC) has been consistently used when evaluating model performance, but it has been shown that AUROC overemphasizes the most common event resulting in an inaccurate assessment when the data are imbalanced. Furthermore, there is evidence that the use of more advanced MLA, such as an unsupervised k-means model, may show superior performance compared to LR as they can handle a larger number of features. The first objective of the study was to assess the performance of both an unsupervised MLA and LR model with complete admission neurologic information against the vM and Hicks models. Second, a comparison between the accuracy of the AUROC and the F1-score will be made to determine which method is superior for the assessment of diagnostic performance of prediction models on large-scale datasets. Retrospective review of a prospective cohort study. The Rick Hansen Spinal Cord Injury Registry (RHSCIR) was used in this study. All patients enrolled between 2004 and 2017 with complete neurologic examination and Functional Independence Measure outcome data at ≥1 year follow-up or who could walk at discharge were included. The prognostic variables included age (dichotomized at ≥65 years old); American Spinal Injury Association Impairment Scale (AIS) grade; and individual motor, light touch, and pinprick score from L2 to S1. The Functional Independence Measure locomotor score was used to assess independent walking ability at discharge or 1-year follow-up. An unsupervised MLA with k=2 was chosen in order to identify a "walk" cluster and a "not walk" cluster. Model performance was assessed through the development of a receiver operating characteristic curve with associated AUROC and a precision-recall curve with associated F1-score. The study and the RHSCIR are supported by funding from Health Canada, Western Economic Diversification Canada, and the Governments of Alberta, British Columbia, Manitoba, and Ontario. These funders had no role in the study or study reporting and the authors have no conflicts of interest to report. No clinically relevant differences were found between with the use of an unsupervised MLA with a greater amount of initial neurologic information compared to the established standards for any AIS classification. Although demonstrated for all separate AIS classifications, most notably, the AUROC for the vM (0.78) and Hicks models (0.76) were found to be superior to that of the new LR model (0.72); however, the vM and Hicks models had more than double the amount of false negative classifications compared to the LR. The F1-scores between these three models were also found to be different but with the vM and Hicks models being lower than the LR (0.85, 0.81, and 0.89, respectively). No clinically relevant differences were found between the use of an unsupervised MLA with complete admission neurologic information compared to the previously validated standards; however, when comparing the performance of the AUROC and F1-score, the AUROC showed inaccurate prognostic performance when there was an imbalance toward a greater amount of false negatives. Importantly, the F1-score did not succumb to this imbalance. As AUROC has been used as the standard when evaluating performance of prediction models, consideration as to whether this is the most appropriate method is warranted. Future work should focus on comparing AUROC and F1-scores with other previously validated models.

DeVries Z ，Hoda M ，Rivers CS ，Maher A ，Wai E ，Moravek D ，Stratton A ，Kingwell S ，Fallah N ，Paquet J ，Phan P ，RHSCIR Network ... -  《-》

Highlighting discrepancies in walking prediction accuracy for patients with traumatic spinal cord injury: an evaluation of validated prediction models using a Canadian Multicenter Spinal Cord Injury Registry.

Models for predicting recovery in traumatic spinal cord injury (tSCI) patients have been developed to optimize care. Several models predicting tSCI recovery have been previously validated, yet recent findings question their accuracy, particularly in patients whose prognoses are the least predictable. To compare independent ambulatory outcomes in AIS (ASIA [American Spinal Injury Association] Impairment Scale) A, B, C, and D patients, as well as in AIS B+C and AIS A+D patients by applying two existing logistic regression prediction models. A prospective cohort study. Individuals with tSCI enrolled in the pan-Canadian Rick Hansen SCI Registry (RHSCIR) between 2004 and 2016 with complete neurologic examination and Functional Independence Measure (FIM) outcome data. The FIM locomotor score was used to assess independent walking ability at 1-year follow-up. Two validated prediction models were evaluated for their ability to predict walking 1-year postinjury. Relative prognostic performance was compared with the area under the receiver operating curve (AUC). In total, 675 tSCI patients were identified for analysis. In model 1, predictive accuracies for 675 AIS A, B, C, and D patients as measured by AUC were 0.730 (95% confidence interval [CI] 0.622-0.838), 0.691 (0.533-0.849), 0.850 (0.771-0.928), and 0.516 (0.320-0.711), respectively. In 160 AIS B+C patients, model 1 generated an AUC of 0.833 (95% CI 0.771-0.895), whereas model 2 generated an AUC of 0.821 (95% CI 0.754-0.887). The AUC for 515 AIS A+D patients was 0.954 (95% CI 0.933-0.975) with model 1 and 0.950 (0.928-0.971) with model 2. The difference in prediction accuracy between the AIS B+C cohort and the AIS A+D cohort was statistically significant using both models (p=.00034; p=.00038). The models were not statistically different in individual or subgroup analyses. Previously tested prediction models demonstrated a lower predictive accuracy for AIS B+C than AIS A+D patients. These models were unable to effectively prognosticate AIS A+D patients separately; a failure that was masked when amalgamating the two patient populations. This suggests that former prediction models achieved strong prognostic accuracy by combining AIS classifications coupled with a disproportionately high proportion of AIS A+D patients.

Phan P ，Budhram B ，Zhang Q ，Rivers CS ，Noonan VK ，Plashkes T ，Wai EK ，Paquet J ，Roffey DM ，Tsai E ，Fallah N ... -  《-》

The effect of age and injury severity on clinical prediction rules for ambulation among individuals with spinal cord injury.

While several models for predicting independent ambulation early after traumatic spinal cord injury (SCI) based upon age and specific motor and sensory level findings have been published and validated, their accuracy, especially in individual American Spinal Injury Association [ASIA] Impairment Scale (AIS) classifications, has been questioned. Further, although age is widely used in prediction rules, its role and possible modifications have not been adequately evaluated until now. To evaluate the predictive accuracy of existing clinical prediction rules for independent ambulation among individuals at spinal cord injury model systems (SCIMS) Centers as well as the effect of modifying the age parameter from a cutoff of 65 years to 50 years. Retrospective analysis of a longitudinal database. Adult individuals with traumatic SCI. The FIM locomotor score was used to assess independent walking ability at the 1-year follow-up. In all, 639 patients were enrolled in the SCIMS database between 2011 and 2015, with complete neurological examination data within 15 days following the injury and a follow-up assessment with functional independence measure (FIM) at 1-year post injury. Two previously validated logistic regression models were evaluated for their ability to predict independent walking at 1-year post injury with participants in the SCIMS database. Area under the receiver operating curve (AUC) was calculated for the individual AIS categories and for different age groups. Prediction accuracy was also calculated for a new modified LR model (with cut-off age of 50). Overall AUC for each of the previous prediction models was found to be consistent with previous reports (0.919 and 0.904). AUCs for grouped AIS levels (A+D, B+C) were consistent with prior reports, moreover, prediction for individual AIS grades continued to reveal lower values. AUCs by different age categories showed a decline in prognostication accuracy with an increase in age, with statistically significant improvement of AUC when age-cut off was reduced to 50. We confirmed previous results that former prediction models achieve strong prognostic accuracy by combining AIS subgroups, yet prognostication of the separate AIS groups is less accurate. Further, prognostication of persons with AIS B+C, for whom a clinical prediction model has arguably greater clinical utility, is less accurate than those with AIS A+D. Our findings emphasize that age is an important factor in prognosticating ambulation following SCI. Prediction accuracy declines for older individuals compared with younger ones. To improve prediction of independent ambulation, the age of 50 years may be a better cutoff instead of age of 65.

Engel-Haber E ，Zeilig G ，Haber S ，Worobey L ，Kirshblum S ... -  《-》

Predicting Recruitment Feasibility for Acute Spinal Cord Injury Clinical Trials in Canada Using National Registry Data.

Thibault-Halman G ，Rivers CS ，Bailey CS ，Tsai EC ，Drew B ，Noonan VK ，Fehlings MG ，Dvorak MF ，Kuerban D ，Kwon BK ，Christie SD ... -  《-》