The challenge of ensuring that tools for data science and machine learning enforce ethical notions like privacy and fairness is one of the most important facing modern computer scientists. While the last decade has seen a flurry of research in this area, there are still significant challenges to using existing algorithms and definitions in practice. This thesis considers the theoretical questions arising from practical considerations, with an emphasis on machine learning applications. In particular, we make crucial definitions and obtain new results towards answering the following questions: • How can we learn optimally private classifiers subject to a hard accuracy constraint? • How can we leverage heuristic optimization oracles for private learning while still maintaining rigorous privacy guarantees? • How can we extend the coarse fairness protections provided by statistical notions of fairness to richer subgroup classes? • How can we learn subject to an individual fairness notion whose metric is not provided, but is instead learned from a panel of experts? Behavioral subject experiments validate theoretical results.