Study Shows Analytic Developed at MCIRCC Outperforms Existing Model for Predicting Unexpected Deterioration of COVID-19 and Non-COVID-19 Patients
ANN ARBOR, MI – ICU triage decisions, outcomes management, and capacity planning for hospitals has always been a challenge but became a critical issue with the influx of patients with COVID-19 symptoms. Though notorious for its respiratory complications, symptoms can affect the entire body and patients can deteriorate quickly as their immune systems work overtime to combat the disease. Early intervention can prevent critical deterioration for these patients while helping hospitals better manage their resources.
In a study published in JMIR: Medical Informatics, a team of data scientists from the Michigan Center for Integrative Research in Critical Care (MCIRCC) demonstrated that a recently developed predictive analytic, called PICTURE (Predicting ICU Transfer and other Unforeseen Events), is significantly more accurate in predicting deterioration and need for life-saving interventions for both general ward patients and COVID-19 positive patients compared to the Epic Deterioration Index (EDI), an existing product which has recently been assessed for use in COVID-19 and non-COVID-19 patients. PICTURE also significantly outperformed another commonly used deterioration index, the National Early Warning Score (NEWS).
To learn more about the PICTURE suite of predictive analytics, download our informative brochure (PDF)
The MCIRCC team made a quick pivot in 2020 in response to the COVID-19 health crisis, rapidly adapting research and lessons learned from other critical illnesses to develop an enhanced version of the PICTURE platform - a machine learning algorithm utilizing electronic health record (EHR) data. Designed for seamless integration into any hospital, the customized algorithm crunches an array of data, such as vital signs, lab results, and demographic information, to flag the patients that are at highest risk of deterioration.
“The electronic health record holds a vast wealth of patient data, but clinicians might not always be able to put it all together and see the big picture.” said Dr. Michael Sjoding, a pulmonary critical physician at Michigan Medicine and member of the team that developed PICTURE. “Our hope is the PICTURE model will ultimately help clinicians do that so they can provide better patient care.”
In addition to giving advanced warning about these high-risk patients, the model is also able to explain to the clinician why it thought the alert should fire. PICTURE can display a list of factors that the model believes are most important to its prediction, which allows the clinician to quickly adjudicate and respond to the alarm.
“The PICTURE model is able to integrate data from the electronic health record, and transform it into meaningful predictions based on the patient’s risk of experiencing an adverse outcome,” Brandon Cummings, a data scientist at MCIRCC, explained. “Trained on data from more than 100,000 patients, it has learned physiologic signatures that indicate a patient may be at risk of deterioration. This is especially important in the case of COVID-19 patients, who can deteriorate rapidly and unexpectedly. By predicting these events before they occur, PICTURE can give clinicians time to react and stabilize the patient before more drastic measures are required.”
In the published study, the PICTURE model was trained and validated on a cohort of hospitalized non-COVID-19 patients using electronic health record data from 2014-2018. It was then applied to two hold-out test sets: non-COVID-19 patients from 2019 and patients testing positive for COVID-19 in 2020. PICTURE results were aligned to Epic’s EDI scores for head-to-head comparison of the models’ ability to predict an adverse event (defined as ICU transfer, mechanical ventilation use, or death).
“In our head-to-head comparison, PICTURE significantly outperformed the EDI and NEWS for both COVID-19 and non-COVID patients.” said Cummings. “This means we catch more deteriorating patients, and also have fewer false alarms which, over time, can lead to alert fatigue. In addition, the alerts happened sooner than those generated by the EDI, giving the clinician more time to react. Combined with PICTURE’s ability to explain its predictions, the system can provide powerful insight regarding a patient’s status and potential risk.”
The study was the first PubMed indexed paper to report a direct head-to-head comparison with Epic’s EDI in COVID-19 patients, and one of the first to do an external, direct comparison.
“The comparison between PICTURE and Epic’s Deterioration Index is of practical importance because the Epic tool is using the same source data for their algorithm (local EHR data) and the Epic tool is readily available to any institution using Epic,” said Dr. Richard Medlin, assistant professor of emergency medicine and associate chief medical information officer at Michigan Medicine and collaborator on the project. “PICTURE’s superior performance demonstrates that careful algorithm development can significantly improve predictive value from the exact same data and lead to both earlier indications of deterioration and fewer false alarms.”
The MCIRCC Data Science team is currently working to test PICTURE in other health care systems and is also developing a full suite of specialized versions of PICTURE adapted for various real-world patient populations, including rehabilitation, pediatrics, and sepsis patients.
“The PICTURE product line is another great example of MCIRCC using multidisciplinary innovation and integration to develop scalable platforms that span the spectrum of critical illness and injury,” said Dr. Kevin Ward, MCIRCC Executive Director. “The ability to anticipate these events will be valuable when considering potential future waves of COVID-19 infections. However, the real value will be the continued use of PICTURE in all hospitalized patients no matter what the situation is.”
Cummings BC, Ansari S, Motyka JR, Wang G, Medlin RP Jr, Kronick SL, Singh K, Park PK, Napolitano LM, Dickson RP, Mathis MR, Sjoding MW, Admon AJ, Blank R, McSparron JI, Ward KR, Gillies CE
JMIR Medical Informatics. 03/04/2021:25066 (forthcoming/in press)
PMID: 33818393
About MCIRCC
The team at the Michigan Center for Integrative Research in Critical Care (MCIRCC) is dedicated to pushing the leading edge of research to develop new technologies and novel therapies for the most critically ill and injured patients. Through a unique formula of innovation, integration, and entrepreneurship, their multi-disciplinary teams of health providers, basic scientists, engineers, and data scientists, commercialization coaches, donors and industry partners are taking a boundless approach on re-imagining every aspect of critical care medicine. For more information, visit www.mcircc.umich.edu.