Development and external validation of a machine learning model for predicting severe immediate postoperative complications in adults undergoing cardiac surgery

Abstract

Background Patients undergoing cardiac surgery are at high risk of severe immediate postoperative complications. Predicting immediate postoperative adverse events remains challenging owing to the complex and nonlinear interplay of numerous risk factors. This study aimed to develop and externally validate a machine-learning (ML) model to predict critical outcomes during the immediate postoperative period after cardiac surgery. Methods Adult patients who underwent cardiac surgery at Seoul National University Hospital (SNUH) between October 2004 and October 2021 were included for model development and internal validation. Thirty-seven preoperative and intraoperative variables were used as model inputs. The primary outcome was a composite of reoperation for bleeding, death, cardiac arrest, and initiation of mechanical circulatory support with extracorporeal membrane oxygenation (ECMO) or intra-aortic balloon pump (IABP) within 24 h after surgery. An extreme gradient boosting (XGBoost) model was developed as the primary model and compared with a least absolute shrinkage and selection operator (LASSO) regularized logistic regression model. An equal-weighted ensemble of the two models was additionally constructed. Model performance was assessed using the area under the receiver operating characteristic curve (AUROC) with 95% confidence intervals (CIs), and pairwise comparisons were performed using DeLong's test. For external validation, we analyzed data from Seoul National University Bundang Hospital (SNUBH) and Severance Hospital. Results A total of 7,946 patients from SNUH were used for model development, with external validation performed in 2,270 patients from SNUBH and 1,966 patients from Severance Hospital. The incidence of the composite outcome was 5.93%, 1.37%, and 2.49% at SNUH, SNUBH, and Severance Hospital, respectively. The XGBoost model achieved an AUROC of 0.804 (95% CI, 0.707-0.891) in internal validation, and 0.735 (95% CI, 0.636-0.829) and 0.712 (95% CI, 0.630-0.795) in external validations at SNUBH and Severance Hospital, respectively. The XGBoost model demonstrated the highest discrimination in internal validation among the XGBoost, LASSO and ensemble models; however, no statistically significant differences were observed among the three models in external validation. Conclusions The proposed ML model demonstrated acceptable discrimination in internal validation and maintained baseline discriminative capacity across two independent external cohorts in predicting severe immediate postoperative complications after cardiac surgery. However, precision-oriented metrics declined substantially in external settings, largely attributable to lower event rates, underscoring the need for institution-specific recalibration prior to clinical implementation.