Marr’s tri-level hypothesis marked a turning point for systems neuroscience and hasconsistently influenced thinking in the field since its proposal nearly fifty years ago by David Marr and Tomaso Poggio. Its assertion that any system could be understood on three distinct but equally valid levels - implementation, representational/algorithmic, and computational - provided a unifying framework into which researchers could organize their work and the work of their peers. Neuroscience appears to again be at a turning point. Recent technological, computational, and theoretical developments offer the possibility of making significant progress in understanding neural systems on the representational /algorithmic level. Thus, once again, a unifying framework is needed for researchers to organize this new wave of work. In this thesis, I humbly take a bold stab at proposing such a unifying framework. This framework, Representational Inference through Structure, or Inference through Structure for brevity, argues that researchers make inferences about the representations and algorithms used by neural systems by leveraging three different types of observed structure. Specifically: structure in the stimuli/inputs, structure in the neural activity itself, and/or structure in the task. In the introductory chapter I argue that this framework allows us to view approaches sometimes seen as disparate or even antagonistic as instead leveraging a different type of structure and thus part of a unified effort to understand the brain. In the next chapters I then take my own work across three different domains of systems neuroscience as examples of leveraging the different types of structure. In the final chapter I conclude by demonstrating the broader utility of the framework and discussing the potential next steps in the field of systems neuroscience.