One of the most common features in fluvial environments is the systematic downstream decline in grain size, which is usually attributed to either abrasion - the reduction in sediment size due to attrition of mass - or selective sorting - the size segregation of grains due to their relative transport mobility. Despite the ubiquity of this grain pattern and the extensive research on both of these processes, there remains questions regarding the underlying principles driving abrasion and sorting, as well as the relative contribution of these processes to grain fining. Therefore, a mechanistic understanding of these processes is necessary to observe their direct effect on pattern formation. This dissertation investigates the controls and limits on abrasion and sorting through field studies and laboratory experiments. First, using the well-defined boundary conditions of an alluvial fan, we examine how grain hiding limits gravel sorting by tracking changes in the grain size distribution measured using a novel image-based technique. Further downfan, we compare surface sand fractions measured in the field with those from the lab and show that the gravel-sand sorting profiles are self-similar, suggesting generality in their development. In a second field study, using detailed hand and image-based measurements characterizing size and shape of thousands of grains throughout a watershed, we are able to directly observe the effectiveness of abrasion. We then input these measurements into a simple numerical model to tease apart the contribution of abrasion and sorting to downstream grains size and shape evolution. Finally, we conduct laboratory experiments to isolate the effects of impact energy on abrasion rates and use material properties of the grains to collapse mass loss curves between different lithologies. We measure the grain size distribution of the products of abrasion to show that they are in agreement with expectations from brittle fracture theory. The results from this work indicate that both sorting and abrasion are effective mechanisms in producing downstream grain size patterns. Because grain size exerts a strong control on channel morphology, understanding the controls on particle size change fosters a more complete picture of the fluvial system.