PICTURE predictive analytic tool proves adaptable and reliable in hospital outside of U-M
Michigan Medicine study shows model that predicts patient deterioration defies traditional challenges associated with utilization in a new clinical environment.
Contact:
Megan VanStratt, Marketing Communications Director, Weil Institute
vanstrat@umich.edu
ANN ARBOR, MI – Prediction models, risk scores and decision tools are becoming a more integral part of modern medical practice. Current systems have demonstrated that early detection of patient deterioration can lead to reduced mortality risk, reduced length-of-stay, and decreased hospital costs. But to implement these systems seamlessly within hospitals outside of where they were originally developed, validation at multiple institutions with significantly different patient populations is critical.
In a recent study published in Critical Care Medicine, researchers from Michigan Medicine, the Weil Institute for Critical Care Research & Innovation, Precision Health, and Hurley Medical Center (Flint, MI) analyzed data from more than 11,000 hospital encounters to externally validate a predictive analytic originally developed at the Weil Institute, called PICTURE (Predicting Intensive Care Transfers and other UnfoReseen Events). PICTURE is a machine learning algorithm that utilizes electronic health record (EHR) data to predict ICU transfer or death as a proxy for patient deterioration in hospital settings.
In the study, the team examined whether the predictions made by PICTURE, which was developed at a single academic medical center, would generalize to a second community-oriented hospital that had significantly different patient demographics.
“Imagine a student that went to the University of Michigan for medical school, followed by a Michigan Medicine residency and eventually staying to work as a clinician,” said Brandon Cummings, Senior Data Scientist at the Weil Institute, and lead author of the study. “Now imagine that same student got a new job at a different hospital. It would take time and effort to figure out the new hospital’s systems, methodologies, and organizational practices. The same is true for predictive analytic models that are developed in one medical setting and used in hospitals all over the world.”
"A common hurdle in developing a tool that can predict clinical deterioration is showing that the tool can be used across multiple institutions. PICTURE performed very well at both institutions [Hurley Medical Center and Michigan Medicine] at predicting meaningful clinical deterioration."
Mike Roebuck, MD
Chief Medical Informatics Officer and Emergency Physician, Hurley Medical Center
Clinical Assistant Professor of Emergency Medicine, University of Michigan
According to the study, moving a predictive model to a new clinical environment outside that which it was trained is fraught with challenges. Every hospital is unique and will have elements that can’t be controlled for or replicated. Just because the predictive model showed excellent performance at the institution where it was originally trained, doesn’t mean that same model will perform equally well in a different environment.
“In the paper, we wanted to investigate the similarities and differences between Hurley Medical Center and Michigan Medicine, and show how PICTURE responds,” said Cummings.
Michigan Medicine is a large academic research institution and medical center where many patients are referred for advanced specialty care. In contrast, Hurley Medical Center is a large 443-bed community hospital providing medical services to Flint, MI and Genessee county. Although both hospitals use the same EHR vendor (Epic Systems, Verona WI), differences in the patient population, clinical care guidelines, demographic makeup, and many other factors led to changes in feature distribution, deterioration rate, and missingness that had the potential to impact the model.
“It was really challenging,” noted Cummings. “To protect privacy, all of the data had to be fully de-identified before it could be sent to us. This meant we really had to rely on the team at Hurley to help us out and synchronize a tremendous number of underlying differences in the data.”
“A common hurdle in developing a tool that can predict clinical deterioration is showing that the tool can be used across multiple institutions,” said Dr. Mike Roebuck, MD, Chief Medical Informatics Officer and Emergency Physician at Hurley Medical Center, Clinical Assistant Professor of Emergency Medicine at U-M, and co-author of the study. “PICTURE performed very well at both institutions at predicting meaningful clinical deterioration.”
Despite the clinical and information-technology differences, PICTURE was able to consistently predict deterioration events and outperform existing metrics at both institutions, including a well-known proprietary system called the EDI (Epic Deterioration Index), supporting its suitability as an early-warning reminder tool to predict deterioration in general ward patients across different clinical settings.
The research team said they are encouraged by the results and are looking forward to validation across further hospitals to ensure portability across a wider variety of clinical settings.
PICTURE has already been licensed to AirStrip®, one of the world’s leading health system solution providers. “We are thrilled to have the opportunity to offer PICTURE to our network of over 500 hospitals,” said J.F. Lancelot, Chief Technology Innovation Officer at AirStrip®. “Validated predictive tools like this are going to make a major difference in measurably improving the clinical, operational, and financial bottom lines in healthcare.”
At University of Michigan, the Weil Institute is rolling out PICTURE to the Rapid Response Team – a group of nurses and other health care professionals who bring early intervention critical care to the bedside of patients who demonstrate acute changes and/or are progressively deteriorating. They are also working closely with Precision Health at U-M to carry out clinical studies on additional iterations of PICTURE predictive models.
“Conducting high-quality studies of predictive models is difficult and requires coordination across many different stakeholders and tackling of technical barriers. Precision Health is excited to collaborate and provide resources to help teams like the Weil Institute answer important questions around the implementation of predictive models,” said Dr. Karandeep Singh, MD, MMSc, Assistant Professor of Learning Health Sciences, Internal Medicine, Urology, and Information at U-M and Associate Chief Medical Information Officer for Artificial Intelligence.
Additional PICTURE products are in various stages of development – including specific PICTURE analytics for pediatrics, rehabilitation units, and even an adaptation of the models that will integrate waveforms and numerics to provide faster and more accurate predictions.
Citation
External Validation and Comparison of a General Ward Deterioration Index Between Diversely Different Health Systems. Critical Care Medicine ():10.1097/CCM.0000000000005837, March 16, 2023. | DOI: 10.1097/CCM.0000000000005837, https://journals.lww.com/ccmjournal/Fulltext/9900/External_Validation_and_Comparison_of_a_General.108.aspx
Authors
Cummings, Brandon C. MHI (1,2); Blackmer, Joseph M. BS (1,2); Motyka, Jonathan R. MS (1,2); Farzaneh, Negar PhD (1,2); Cao, Loc MS (1,2); Bisco, Erin L. BA (1,2); Glassbrook, James D. AAS RT (3); Roebuck, Michael D. MD (2,4); Gillies, Christopher E. PhD (1,2); Admon, Andrew J. MD, MPH, MS (1,5,6); Medlin, Richard P. Jr MD, MSIS (1,2); Singh, Karandeep MD, MMS (5,7,8); Sjoding, Michael W. MD (1,5,8); Ward, Kevin R. MD (1,2,9); Ansari, Sardar PhD (1,2)
Affiliations at Time of Publication
1. The Max Harry Weil Institute of Critical Care Research & Innovation, University of Michigan, Ann Arbor, MI.
2. Department of Emergency Medicine, University of Michigan Medical School, Ann Arbor, MI.
3. Information Technology, Hurley Medical Center, Flint, MI.
4. Department of Emergency Medicine, Hurley Medical Center, Flint, MI.
5. Department of Internal Medicine, University of Michigan Medical School, Ann Arbor, MI.
6. Medicine Service, LTC Charles S. Kettles VA Medical Center, Ann Arbor, MI.
7. Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor, MI.
8. Precision Health, University of Michigan, Ann Arbor, MI.
9. Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI.
Disclosures
Mr. Cummings’s, Mr. Blackmer’s, Dr. Farzaneh’s, Ms. Cao’s, and Dr. Ansari’s institutions received funding from the Michigan Institute for Data Science and Airstrip Technologies. Mr. Cummings, Mr. Blackmer, Dr. Farzaneh, Ms. Cao, and Drs. Gillies, Medlin, Ward, Sjoding, and Ansari have disclosed that multiple patents have been filed for this work. Their invention disclosures have been submitted with the Office of Technology Transfer, University of Michigan, Ann Arbor, and Airstrip Technologies has a license option for Predicting Intensive Care Transfers and other UnfoReseen Events (PICTURE) from the University of Michigan. Mr. Motyka disclosed that he is currently employed by at Strata Oncology (Precision Oncology). Dr. Gillies disclosed that he is employed at Regeneron Pharmaceuticals. Dr. Admon’s institution received funding from the National Heart, Lung, and Blood Institute. Drs. Admon and Sjoding received support for article research from the National Institutes of Health (NIH). Dr. Singh’s institution received funding from Blue Cross Blue Shield of Michigan and Teva Pharmaceuticals; he received funding from Flatiron Health. Dr. Sjoding’s institution received funding from the NIH. The remaining authors have disclosed that they do not have any potential conflicts of interest.
About the Weil Institute, formerly MCIRCC
The team at the Max Harry Weil Institute for Critical Care Research and Innovation (formerly the Michigan Center for Integrative Research in Critical Care) is dedicated to pushing the leading edge of research to develop new technologies and novel therapies for the most critically ill and injured patients. Through a unique formula of innovation, integration and entrepreneurship that was first imagined by Weil, their multi-disciplinary teams of health providers, basic scientists, engineers, data scientists, commercialization coaches, donors and industry partners are taking a boundless approach to re-imagining every aspect of critical care medicine. For more information, visit weilinstitute.med.umich.edu.
About Hurley Medical Center
From its founding in 1908, Hurley Medical Center has devoted itself to bringing innovative, leading-edge technology and medical services to Flint and Genesee County. From our expertly trained physicians and nurses to our highly innovative technology to our state-of-the art facilities, Hurley is widely recognized as a medical center of exceptional excellence.
About Precision Health
Precision Health at the University of Michigan expands far beyond medical treatment to include discovery, implementation, and community health. Using our unique breadth and depth of excellence across disciplines, U-M offers three complementary components of a comprehensive initiative: discovery, treatment, and health.
Spanning across 19 colleges and schools, Precision Health brings the best and brightest together to 1) develop fundamental social, medical, computational, and engineering science; 2) translate these basic science discoveries into promising treatments that are evaluated in partnership with Michigan Medicine patients and regional health systems; and 3) evaluate and increase the public health impact of effective therapies, working with community health systems, policy makers, and payers to implement these therapies nationally.
https://precisionhealth.umich.edu/
About AirStrip
AirStrip ONE® offers solutions that allow hundreds of health systems to unlock the full potential of their existing medical technology infrastructure investments with an interoperability platform that provides seamless access to clinical data and mobile actionable insights across the care continuum using remote access or web access granted by the health system. It features mobile diagnostic quality cardiac waveform viewing and sophisticated mobile fetal surveillance, providing clinicians with near-real-time contextual and clinically relevant data for situational awareness.
AirStrip
Wesley Hartline
SVP of Business Development, Revenue Growth
wesleyhartline@airstrip.com