The limited resources available to the visual system must be allocated efficiently to support its function. To achieve this, our brain needs to take advantage of the statistical regularities of our visual environment. In this thesis, I systematically explore how different aspects of natural stimulus statistics can impact and determine perceptual behavior and sensory representation in both biological and artificial systems. In Chapter 1, I provide a brief review of the theory of efficient coding, models of natural image statistics, and the interplay between these two fields. In Chapter 2, based on a Bayesian ideal observer model that is constrained by efficient coding, I show how simple stimulus priors can provide a quantitative link between psychophysics and neurophysiology in the domain of speed perception. In Chapter 3, I extend these ideas to the domain of sensory adaptation. In particular, I develop a method to quantify changes in sensory encoding in a tilt illusion experiment, and find that these changes are consistent with an efficient coding account for which the encoding is optimized toward the conditional statistics of orientation based on the surrounding context. In Chapter 4, I generalize the efficient coding principle to fully naturalistic stimuli by building models of natural image statistics and image-computable ideal observers to quantify the information encoded by the early stages of visual encoding. I show how features of the retinal encoding can be explained by an optimal design principle. In Chapter 5, I present a novel algorithm for directly solving the linear optimal coding problem by finding the set of linear measurements that minimize error in a Bayesian image reconstruction problem. This approach improves upon established methods such as principal component analysis and compressed sensing, and provides a unifying perspective. Lastly, in Chapter 6, I discuss open questions and future directions.