Enhancing the Reliability of Real-World Evidence and Clinical Decision-Making: Robust, Calibrated, and Uncertainty-Aware Methods for Observational Healthcare Research

Loading...
Thumbnail Image
Degree type
PhD
Graduate group
Applied Mathematics and Computational Science
Discipline
Mathematics
Subject
Model Misspecification
Real-world Evidence
Systematics Bias
Trustworthy Prediction
Funder
Grant number
License
Copyright date
01/01/2025
Distributor
Related resources
Author
Shen, Yishan
Contributor
Abstract

Real-world data (RWD) has become instrumental in clinical and public health research, providing opportunities for more generalizable and practical insights compared to traditional randomized controlled trials (RCTs). Nevertheless, observational studies inherently encounter critical challenges, such as model misspecification, systematic biases, and uncertainties in predictions, potentially compromising the reliability and validity of generated evidence. Ensuring reliability is particularly vital for real-world decision-making, where trustworthy evidence must explicitly account for model uncertainty. This work presents novel methodologies specifically designed to enhance the reliability and robustness of evidence derived from RWD across diverse healthcare research contexts. In Chapter 2, we introduce the Assumption-lean Projection-based Heterogeneity-aware Algorithm (ALPHA), a robust method specifically designed to mitigate risks arising from model misspecification in large-scale distributed research settings. ALPHA efficiently estimates treatment effects without stringent modeling assumptions and accommodates site-specific heterogeneity across decentralized data networks without sharing individual-level information. Applying ALPHA to electronic health record data from eight national pediatric hospitals, we quantify the impact of COVID-19 infection on healthcare utilization outcomes, demonstrating ALPHA’s effectiveness in consistently reducing variance and systematic errors while accurately capturing substantial heterogeneity across clinical sites. Building upon the need to address systematic biases in multi-site observational research, Chapter 3 introduces a novel multi-site empirical calibration framework. This framework employs negative control outcomes (NCOs) to correct residual biases effectively, explicitly accounting for heterogeneity across both clinical sites and outcomes. By assessing cardiovascular outcomes associated with second-line diabetes therapies across five federated institutions, this method significantly improves type-I error control, achieves nominal statistical coverage, and delivers credible bias-adjusted estimates, thus enhancing the validity and generalizability of real-world evidence. Recognizing the importance of reliability in high-stakes clinical decisions, Chapter 4 presents SAFER, a calibrated risk-aware recommendation model explicitly designed for dynamic treatment regimes. SAFER uniquely combines structured electronic health record data and unstructured clinical notes, quantifies model uncertainty via per-label risk assignment, incorporating conformal selection framework to control prediction uncertainty rigorously. Validated on publicly available sepsis datasets, SAFER significantly surpasses existing models, markedly reduces counterfactual mortality, and delivers trustworthy, clinically actionable recommendations, thereby strengthening decision-making reliability in critical care scenarios. Collectively, these methodological advancements systematically enhance the reliability of real-world evidence by addressing core challenges, including bias reduction, robustness to model misspecification, and comprehensive uncertainty quantification. Ultimately, this work supports more informed and effective healthcare decisions, reinforcing trust in real-world evidence applications.

Advisor
Chen, Yong, Y.C.
Date of degree
2025
Date Range for Data Collection (Start Date)
Date Range for Data Collection (End Date)
Digital Object Identifier
Series name and number
Volume number
Issue number
Publisher
Publisher DOI
Journal Issue
Comments
Recommended citation