Imperfect, useful dynamical models have enabled significant progress in planning and controlling robotic locomotion and manipulation. Traditionally, these models have been physics-based, with accuracy relying upon manual calibration only feasible in laboratory environments. As robotics expands into complex real-world applications, models of unknown environments must instead be automatically fit to limited data. One major challenge is modeling frictional contact, especially during collisions involved in common robotics tasks. Rapid deformation under impact manifests as extreme sensitivity to initial conditions and material properties. Thus, even slight errors in state estimation and system identification can lead to significant prediction errors. Consequently, model inaccuracy or the sim-to-real gap often hinders the development of performant robotics algorithms. Physical models can be optimized using advanced techniques to overcome these challenges, but such methods have limited tractability when a large number of parameters are unknown. Furthermore, even given accurate parameters, roboticists often make inaccurate rigid-body approximations to reduce the computational burdens of physical simulation to meet faster-than-real-time requirements. An alternative black-box approach, in which models are learned from scratch, has attempted to address these issues for instance using deep neural networks (DNN's). While DNNs in theory can capture any dynamical behavior, they empirically struggle with the stiff behaviors associated with contact. This dissertation instead focuses on scaling physical model identification to the high-dimensional setting and quantifying the limited accuracy of low-fidelity models. We consider rigid bodies undergoing rigid contact, for which infinite stiffness is represented as constrained optimization. By careful treatment of these constraints, we demonstrate that infinitely-stiff dynamics can be identified by optimizing a non-stiff objective. In conjunction, we use DNN's in a white-box setting to model physical quantities, specifically reconstructing geometries from scratch. We then consider how rigid-body collision models lack the fidelity to correctly predict outcomes of nearly-simultaneous impacts---such as heel and toe strikes during a footstep. We develop a theoretical basis to capture partial knowledge of impact events as uncertain set-valued outcomes, and again use numerical optimization to compute approximations of such sets.