This thesis comprises three research topics: multitask learning, uncertainty quantification, andlanguage model alignment. In the first chapter, we investigate the challenges and opportunities in learning from large, heterogeneousdatasets, where data is collected from multiple sources with sparse heterogeneity, meaning that the source/task-associated model parameters are equal to a global parameter plus a sparse task-specific term. Traditional methods struggle to address this data heterogeneity, leading to inefficient or biased models. To overcome these limitations, through the lens of linear models (offline regression, online contextual bandits) and communication-constrained distributional estimation, we introduce two methods: MOLAR and SHIFT, which leverage shared structures across datasets with a (weighted) median while accounting for sparse heterogeneity via task-wise debiasing. Our approaches provide principled solutions for dealing with sparse heterogeneity, offer minimax optimal guarantees, and demonstrate practical improvements through experiments with both synthetic and empirical datasets. The second and third chapters are devoted to uncertainty quantification, focusing on general machinelearning classifiers (e.g., neural networks) and large language models (LLMs), respectively. Specifically, the second chapter targets the discrepancy between the probability scores produced by probabilistic classifiers and the actual frequencies of the labels, known as miscalibration. We frame the task of detecting miscalibration as a hypothesis-testing problem. Drawing inspiration from nonparametric hypothesis testing, we propose T-Cal, a minimax optimal test for calibration based on a debiased plug-in estimator of the ℓ2-Expected Calibration Error (ECE). T-Cal offers a principled and statistically sound approach for assessing the calibration of machine learning classifiers. The third chapter concerns Rank-Calibration, a novel framework proposed for evaluating uncertainty measures of language models in natural language generation. Rank-Calibration motivates a new metric, the Rank-Calibration Error (RCE), which quantifies how well the levels of an uncertainty measure align with the quality of generated outputs. Both parts are validated through comprehensive experiments, demonstrating their effectiveness in improving models’ reliability. In the fourth chapter, we address the critical challenge of aligning LLMs with safety preferences, focusingon reducing computational complexity while ensuring safety in model responses. We propose a novel dualization approach that transforms the problem of constrained alignment, traditionally solved through computationally expensive primal-dual policy optimization, into a more efficient unconstrained alignment task. This is achieved by pre-optimizing a smooth, convex dual function, eliminating the need for iterative updates between policy and dual variables. Our method leads to two practical algorithms, MoCAN and PeCAN, tailored for model-based and preference-based scenarios, respectively. Extensive experiments demonstrate that our approach significantly reduces the computational burden while maintaining alignment with diverse safety constraints, offering a more scalable solution to safe reinforcement learning from human feedback (RLHF) in LLMs.