How does the brain represent the enormous variety of the visual world? An approach to this question recognizes the types of information that visual representations maintain. The work in this thesis begins by investigating the neural correlates of perceptual similarity & distinctiveness, using EEG measurements of the evoked response to faces. In considering our results, we recognized that the effects being measured shared intrinsic relationships, both in measurement and in their theoretic basis. Using carry-over fMRI designs, we explored this relationship, ultimately demonstrating a new perspective on stimulus relationships based around sensory history that best explains the modulation of brain responses being measured. The result of this collection of experiments is a unified model of neural response modulation based around the integration of recent sensory history into a continually-updated reference; a "drifting-norm." With this novel framework for understanding neural dynamics, we tested whether cognitive theories of autism spectrum disorder (ASD) might have a foundation in altered neural coding for perceptual information. Our results suggest ASD brain responses depend on a more moment-to-moment understanding of the visual world relative to neurotypical controls. This application both provides an exciting foothold in the brain for future investigations into the etiology of ASD, and validates the importance of sensory history as a dimension of visual representation.