Deep learning has indeed become the workhorse of artificial intelligence, achieving state-of-the-art performance across a wide range of domains, from computer vision to natural language processing. With the widespread deployment of deep learning and the rapid growth of model sizes, two fundamental challenges have emerged: interpretability and efficient hyperparameter optimization of learning rate. The lack of interpretability in deep learning models hinders their adoption in safety-critical applications, while the reliance on manually tuned hyperparameters, such as the learning rate, adds substantial computational overhead. A detailed review of these challenges is discussed in Chapter 1. This dissertation addresses both challenges by advancing interpretable deep learning (Chapter 2) and learning-rate-free optimization (Chapter 3 & 4), paving the way for more transparent and automatic deep learning systems. Interpretable machine learning has shown remarkable performance while maintaining explainability. In particular, Neural Additive Models (NAMs) provide interpretability to traditionally black-box deep learning models and achieve state-of-the-art accuracy within the broader class of generalized additive models. In the second chapter, we propose the sparse neural additive models (SNAM) to empower NAM with feature selection and improve the generalization. It employs the group sparsity regularization (e.g. Group LASSO), where each feature is learned by a sub-network whose trainable parameters are clustered as a group. We study the theoretical properties for SNAM with novel techniques to tackle the non-parametric truth, thus extending from classical sparse linear models such as the LASSO, which only works on the parametric truth. Specifically, we show that SNAM with subgradient and proximal gradient descents provably converges to zero training loss as $t\to\infty$, and that the estimation error of SNAM vanishes asymptotically as $n\to\infty$. We also prove that SNAM, similar to LASSO, can have exact support recovery, i.e. perfect feature selection, with appropriate regularization. Moreover, we show that the SNAM can generalize well and preserve the `identifiability', recovering each feature's effect. We validate our theories via extensive experiments and further testify to the good accuracy and efficiency of SNAM. Differential learning rate (DLR), a technique that applies different learning rates to different model parameters, has been widely used in deep learning and achieved empirical success via its various forms. For example, parameter-efficient fine-tuning (PEFT) applies zero learning rates to most parameters so as to significantly save the computational cost. At the core, DLR leverages the observation that different parameters can have different loss curvature, which is hard to characterize in general. Meanwhile, the computational cost of grid-searching for optimal DLR grows exponentially with the increase of the number of parameter groups. In the third chapter, we propose the Hessian-informed differential learning rate (Hi-DLR), an efficient approach that solves the hyperparameter optimization (HPO) of learning rates and captures the loss curvature for any model and optimizer adaptively. Given a proper grouping of parameters, we empirically demonstrate that Hi-DLR can improve the convergence by dynamically determining the learning rates during the training. Furthermore, we can quantify the influence of different parameters and freeze the less-contributing parameters, which leads to a new PEFT that automatically adapts to various tasks and models. Lastly, adversarial attacks are important methods to evaluate model robustness and enhance it through adversarial training. However, the effectiveness of adversarial traininig significantly depends on the choice of learning rate of adversarial attack, which can be costly to tune in practice. Specifically, in Projected Gradient Descent (PGD) -- a strong adversarial attack method -- a too large learning rate can produce unnatural, noisy adversarial images that may destabilize model training, whereas a too small learning rate generates weak adversarial examples and results in poor robustness. In the fourth chapter, we propose to equip PGD with the Hessian-informed learning-rate-free method. Our method exhibits comparable model performance on both clean and adversarial test data without excessive computational cost, while tackling the constraint in the adversarial problems with careful algorithmic designs. In the end, we summarize our contributions and propose future directions in Chapter 5.