This dissertation studies the trade-off between differential privacy and statistical accuracy in parameter estimation problems. We understand the privacy-accuracy trade-off by finding the best achievable accuracy of any differentially private algorithm, also known as the "privacy-constrained minimax risk", in a series of statistical problems: Gaussian mean estimation and linear regression, estimation in general parametric models, and non-parametric function estimation. The increasing difficulty and generality of this series is matched by the development of differentially private algorithms such as noisy iterative hard thresholding, and of minimax lower bound techniques such as the score attack.