Differential privacy has seen remarkable success as a rigorous and practical for- malization of data privacy in the past decade. But it also has some well known weaknesses, lacking comprehensible interpretation and an accessible and precise toolkit. This is due to the inappropriate (ε, δ) parametrization and the frequent approximation in the analysis. We overcome the difficulties by 1. relaxing the traditional (ε, δ) notion to the so-called f -differential privacy from a decision theoretic viewpoint, hencing giving it strong interpretation, and 2. with the relaxed notion, perform exact analysis without unnecessary approx- imation. Miraculously, with the relaxation and exact analysis, the theory is endowed with various algebraic structures, and enjoys a central limit theorem. The central limit theorem highlights the role of a specific family of DP notion called Gaussian Dif- ferential Privacy. We demonstrate the use of the tools we develop by giving an improved analysis of the privacy guarantees of noisy stochastic gradient descent.