This dissertation explores issues of trustworthy machine learning in algorithmic decision-making contexts. We study problems related to algorithmic fairness and downstream decision-making when using machine learning. A theme in the work is using the theory of calibration in sequential prediction as both a desirable notion of reliability and of algorithmic fairness, and as a tool to achieve "trustworthy" guarantees in the decision-making pipelines we investigate. We begin by investigating problems related to algorithmic fairness. We first show how to make mean predictions with conditional guarantees without making any assumptions about the data. We show that we can efficiently make predictions satisfying strong conditional guarantees - a finer, "multigroup" measure of algorithmic fairness than standard, coarse statistical notions. Next, we study the problem of training a model when the downstream fairness constraints are not known at training time. We show how to flexibly train a model such that it can be easily adapted downstream to (simultaneously) satisfy many desirable fairness constraints. We then proceed to studying how to use predictions from machine learning models in pipelines of decision-making. First, we study the problem of decision-making with uncertain objective functions. We imagine that we are given a collection of models we use in a downstream optimization problem and give two ensembling methods that result in ``transparent'' decisions which improve on those from all initial models. Second, we study a setting where a human and algorithm jointly make predictions based on information they each have available to them, which may be inaccessible to the other party. We study the problem of reaching accuracy-improving agreement, generalizing results of Aumann’s agreement theorem with Bayesian agents using computationally tractable behavioral assumptions.