Model Performance Diagnostics Suite
Value Proposition
The FDA has called for proper post-marked surveillance for Machine Learning and AI tools in healthcare, but no one has a solution as well-equipped to address this call to action as the Model Performance Diagnostics Suite (MPD).
The MPD detects data shift and model degradation by comparing the distribution of the real-time data that is fed to the model to the data on which the model was trained. MPD also applies a causal inference framework to estimate the performance of the model post-deployment, a unique feature that is currently not offered by any other tool. Once hooked up to a model, interactive analytics are visualized via any web browser. The module also provides customized, timely alerts when changes are detected so that a team managing the model can retune it.
The MPD is currently operationalized at Michigan Medicine to track performance of PICTURE, the Weil Institute’s deterioration prediction model that is in active use by the hospital’s Rapid Response team.
Competitive Advantage
Estimates the model performance after deployment even when the model targets are impacted as a result of the model predictions (e.g., adverse events being prevented because they were predicted by the model)
Helps in minimizing false positives or other intrusive model behavior for clinicians and staff
Prevents worsening of clinical outcomes over time due to model performance decline
Enables teams to ensure their models perform equally well across all demographics
Monitor performance of multiple models simultaneously
Unique Features
Interactive Drift Report
Scientifically Validated “True Performance Monitoring”
Detect changes in the model input and output
Data visualization and cross-sections for:
Model Predictions, Performance & Utility
Data Missingness, Outliers & Bias
Drift in Covariates (Input), Targets, Patient Characteristics, Explanations, Practice Patterns, EHR Changes,
Principal Investigators
Sardar Ansari, PhD
Licensing Manager
Drew Bennett
Intellectual Property
Invention Disclosure # 2024-043
Solution Sheet
Available Soon
New clinical predictive models are being released every month in this growing, multibillion dollar industry. The FDA’s strong, public push for post-market surveillance to ensure these models work appropriately and equally across demographics has created a demand for a tool like the Model Performance Diagnostics Suite (MPD).
It’s inevitable that model performance declines over time due to changes in the care environment. In turn, false positives (and false negatives) become more frequent, reducing clinician trust and therefore the efficacy of a model. This puts model administrators in a bind. How can they measure performance accurately and justify retraining to the FDA? The MPD addresses the critical need for researchers and analytic operators to detect and prevent drift and degradation as well as accurately estimate the model’s true performance in a care environment.
Please contact the Licensing Manager, Drew Bennett, for more information.
Funding History
$206,685 in non-dilutive funding
$206,685 in industry partner funding
Substantial (additional) departmental, school and center based support
Next Steps
Realtime integration and monitoring with live Models at Michigan Medicine
Use of MPD for Prospective validation
Use of MPD in a Clinical trial (Q1 2023)
Add customizable alerts by feature or demographic
Add Generalization features to accommodate additional diverse ML models
Add ability to capture and analyze Provider Feedback and EHR Integration for models
Funding Organizations
Publications
None at this time
Media Coverage
None at this time