Real-world data (RWD) has become instrumental in clinical and public health research, providing opportunities for more generalizable and practical insights compared to traditional randomized controlled trials (RCTs). Nevertheless, observational studies inherently encounter critical challenges, such as model misspecification, systematic biases, and uncertainties in predictions, potentially compromising the reliability and validity of generated evidence. Ensuring reliability is particularly vital for real-world decision-making, where trustworthy evidence must explicitly account for model uncertainty. This work presents novel methodologies specifically designed to enhance the reliability and robustness of evidence derived from RWD across diverse healthcare research contexts. In Chapter 2, we introduce the Assumption-lean Projection-based Heterogeneity-aware Algorithm (ALPHA), a robust method specifically designed to mitigate risks arising from model misspecification in large-scale distributed research settings. ALPHA efficiently estimates treatment effects without stringent modeling assumptions and accommodates site-specific heterogeneity across decentralized data networks without sharing individual-level information. Applying ALPHA to electronic health record data from eight national pediatric hospitals, we quantify the impact of COVID-19 infection on healthcare utilization outcomes, demonstrating ALPHA’s effectiveness in consistently reducing variance and systematic errors while accurately capturing substantial heterogeneity across clinical sites. Building upon the need to address systematic biases in multi-site observational research, Chapter 3 introduces a novel multi-site empirical calibration framework. This framework employs negative control outcomes (NCOs) to correct residual biases effectively, explicitly accounting for heterogeneity across both clinical sites and outcomes. By assessing cardiovascular outcomes associated with second-line diabetes therapies across five federated institutions, this method significantly improves type-I error control, achieves nominal statistical coverage, and delivers credible bias-adjusted estimates, thus enhancing the validity and generalizability of real-world evidence. Recognizing the importance of reliability in high-stakes clinical decisions, Chapter 4 presents SAFER, a calibrated risk-aware recommendation model explicitly designed for dynamic treatment regimes. SAFER uniquely combines structured electronic health record data and unstructured clinical notes, quantifies model uncertainty via per-label risk assignment, incorporating conformal selection framework to control prediction uncertainty rigorously. Validated on publicly available sepsis datasets, SAFER significantly surpasses existing models, markedly reduces counterfactual mortality, and delivers trustworthy, clinically actionable recommendations, thereby strengthening decision-making reliability in critical care scenarios. Collectively, these methodological advancements systematically enhance the reliability of real-world evidence by addressing core challenges, including bias reduction, robustness to model misspecification, and comprehensive uncertainty quantification. Ultimately, this work supports more informed and effective healthcare decisions, reinforcing trust in real-world evidence applications.