Key points
|
Title : Quantifying the Prognostic Value of Preoperative Surgeon Intuition: Comparison of Surgeon Intuition and Clinical Risk Prediction Derived from the American College of Surgeons NSQIP Risk Calculator
Summary:
Background:
Surgical risk prediction models traditionally use patient attributes and measures of physiology to generate predictions about postoperative outcomes. However, the surgeon’s assessment of the patient may be a valuable predictor, given the surgeon’s ability to detect and incorporate factors that existing models cannot capture. We compared the predictive utility of surgeon intuition and a risk calculator derived from the American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP).
Study design:
From 10/1/2021 to 9/1/2022, surgeons were surveyed immediately before performing surgery to assess their perception of a patient’s risk of developing any postoperative complications. Clinical data were extracted from the ACS NSQIP. Both data sources were used independently to build models to predict the probability of a patient experiencing any postoperative complications at 30 days, as defined by the ACS NSQIP.
Results:
Surgeon preoperative evaluation was obtained from 216 patients. NSQIP data were available for 9,182 patients who underwent general surgery (1/1/17 to 9/1/22).
A binomial regression model trained on clinical data alone had an AUC of 0.83 (95% CI: 0.80-0.85) for predicting any complication. A model trained only on the surgeon’s preoperative intuition had an AUC of 0.70 (95% CI: 0.63-0.78). A model trained on surgeon intuition and a subset of clinical predictors had an AUC of 0.83 (95% CI: 0.77-0.89).
Conclusions:
The surgeon’s preoperative intuition alone is an independent predictor of patient outcomes; however, a risk calculator derived from the ACS NSQIP is a stronger predictor of postoperative complications. Combining intuition and clinical data did not strengthen the prediction.
Comments
The surgeon’s preoperative intuition is an independent predictor of 30-day postoperative complications; However, when compared to the standard risk calculator derived from the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP®), its predictive power is not as strong, according to a study published in the Journal of the American College of Surgeons . (JAC).
Surgeons weigh many different factors, such as medical history and current health, when deciding what type of surgery to perform. A doctor’s intuition (training, past experiences, and his or her “gut feeling” about a patient) also plays a role in the evaluation.
However, even when the diagnosis is the same, there is still significant variability in physician decision making. According to one study, patients who seek a second opinion only receive the same diagnosis from both doctors about 12% of the time.
“The integration of the explicit, the intangible and experience together form what we call the surgeon’s intuition. "Surgeons with a certain level of training and experience will have relatively similar intuition in certain cases," said the study’s senior author, Gabriel A. Brat, MD, MPH, FACS, trauma surgeon and assistant professor of surgery at Beth Medical Center. Israel Deaconess and Harvard. Medicine School. “However, intuition is dynamic. It depends on the characteristics of the provider. “One surgeon can look at one patient and believe one thing about that patient’s outcome, and another surgeon can look at the same patient and predict a different outcome.”
For the study, researchers sought to quantify the value of intuition in predicting outcomes among surgical patients. They investigated whether preoperative intuition could be used in risk prediction similar to how the ACS NSQIP Surgical Risk Calculator is currently used. The NSQIP Risk Calculator is a tool used to estimate patient-specific risks of postoperative complications for almost all operations and includes an adjustment for the surgeon’s intuition.
“We wanted to know if it is possible to fine-tune intuition in a more precise way,” said Jayson S. Marwaha, MD, MBI, lead author of the study and a general surgery resident at Georgetown University.
The researchers developed a new algorithm that predicts postoperative outcomes using only the surgeon’s preoperative intuition. To do this, researchers surveyed general surgeons between October 2021 and September 2022 at Beth Israel Deaconess Medical Center just before starting their surgery. A one-question text message asked them to predict the patient’s probability of having a negative outcome, specifically whether the patient had a below-average risk, an average risk, or an above-average risk of postoperative complications or death. In total, 216 patients were included in this analysis.
In a separate model, the research team collected NSQIP data on 9,182 patients who underwent general surgery between January 2017 and September 2022 at the medical center. They predicted patient outcomes by analyzing clinical data captured by the NSQIP risk calculator.
After comparing the two models, a third model was constructed, combining preoperative intuition and the NSQIP Risk Calculator, to determine whether this third model could outperform the other two models.
Key results
Nearly half of the surgeons who responded to the survey (45.4%) indicated that their patients’ risk of any complication was average, with 40.3% reporting a higher-than-average risk and 14.4% reporting a higher-than-average risk. responding a lower than average risk.
The surgeon’s preoperative intuition was an independent predictor of postoperative complications. A preoperative surgeon intuition model predicting complications had an area under the curve (AUC) of 0.70, where an AUC of 1.0 is a perfect prediction and 0.5 is a random outcome.
Surgeon intuition in predicting any complications was less accurate than the ACS NSQIP risk calculator, which had an AUC of 0.83.
A combined model using surgeon intuition and the NSQIP risk calculator performed no better, with an AUC of 0.83, than the NSQIP risk calculator alone.
A subset analysis showed that more experienced attending surgeons’ intuition in predicting patient outcomes was more accurate than less experienced residents.
“The value of surgical intuition for preoperative prediction was not improved by including human intuition in the model and this suggests that, at least for most presurgical predictions, the information collected by the NSQIP Risk Calculator is better at predicting those results than the feeling surgeons have when they look at patients,” Dr. Brat said.
“Human intuition takes into account a lot of information that is not available to the calculator, but it does not weigh it explicitly. We don’t have a weighting system in our head that says, "We know this information is more important than other information to some extent," whereas the NSQIP Risk Calculator has that explicit weighting system. So, in certain situations, it is the case that an explicit algorithm is going to be better at predicting. “The value of the physician is integrating information that is not available to the risk calculator.”
“NSQIP Risk Calculator data is among the best at predicting outcomes, demonstrably superior to administrative, billing, and claims data. However, what is done with the data is the next important step,” said Clifford Y. Ko, MD, MS, MSHS, FACS, FASCRS, director of the ACS Division of Research and Optimal Patient Care, who was not involved. in the study. "It will be the joint responsibility of the surgeon and the patient, based on the data, to decide whether to proceed with surgery and how best to prepare for the phases of care before, during and after surgery."
The main limitations of the study are the small size of the data set and the fact that the majority of surgeons who participated in the research were emergency and trauma surgeons. These results may not apply to other types of surgeons or settings.
Co-authors are Jayson S. Marwaha, MD, MBI; Brendin R. Beaulieu-Jones, MD, MBA; Margaret Berrigan, MD; Guillermo Yuan, PhD; Stephen R. Odom II, MD, FACS; Charles H. Cook, MD, FACS; Benjamin B. Scott, MD; Dr. Alok Gupta, FACS; Charles S. Parsons, MD, FACS; and Anupamaa J. Seshadri, MD. The study was supported by the National Library of Medicine.
Reference : Marwaha JS, Beaulieu-Jones BR, Berrigan M, et al. Quantifying the Prognostic Value of Preoperative Surgeon Intuition: Comparing Surgeon Intuition and Clinical Risk Prediction, as derived from the ACS NSQIP Risk Calculator. Journal of the American College of Surgeons. DOI: 10.1097/XCS.00000000000000658.
