Life-critical medical devices require robust safety and efficacy to treat patient populations with potentially large patient heterogeneity. Today, the de facto standard for evaluating medical devices is the randomized controlled trial. However, even after years of device development many clinical trials fail. For example, in the Rhythm ID Goes Head to Head Trial (RIGHT) the risk for inappropriate therapy by implantable cardioverter defibrillators (ICDs) actually increased relative to control treatments. With recent advances in physiological modeling and devices incorporating more complex software components, population-level device outcomes can be obtained with scalable simulations. Consequently, there is a need for data-driven approaches to provide early insight prior to the trial, lowering the cost of trials using patient and device models, and quantifying the robustness of the outcome. This work presents a clinical trial modeling and statistical framework which utilizes simulation to improve the evaluation of medical device software, such as the algorithms in ICDs. First, a method for generating virtual cohorts using a physiological simulator is introduced. Next, we present our framework which combines virtual cohorts with real data to evaluate the efficacy and allows quantifying the uncertainty due to the use of simulation. Results predicting the outcome of RIGHT and improving statistical power while reducing the sample size are shown. Finally, we improve device performance with an approach using Bayesian optimization. Device performance can degrade when deployed to a general population despite success in clinical trials. Our approach improves the performance of the device with outcomes aligned with the MADIT-RIT clinical trial. This work provides a rigorous approach towards improving the development and evaluation of medical treatments.