Understanding the neural organization of semantic memory - our shared general knowledge - has the potential to uncover the neural mechanisms by which we give meaning to the endless array of objects that we encounter in the world. One prominent set of theories posits a distributed organization of semantic memory - remembering object features activates brain regions overlapping with or adjacent to regions involved in perceiving and acting on those features (e.g., Allport, 1985). Despite accumulated evidence favoring these theories, it is important to understand both the commonalities in the mapping between perception and memory, as well as meaningful variability across sources such as contexts, people, and use. In three studies, we found evidence that while conceptual knowledge is grounded in neural substrates, several factors contribute to variations in semantic memory retrieval. In Chapter 2, we used the logic often used in neuroimaging studies of semantic memory by demonstrating overlapping chromaticity effects (e.g., greater response to colored than grayscale stimuli) in the left lingual gyrus for both color perception and color knowledge. Chapter 3 investigated whether the mapping between perception and memory varied across contexts and participants. Whereas context (here, fidelity of color information as manipulated through task demand) varied the extent to which the left fusiform gyrus was active during a color similarity judgment, individual differences in cognitive style predicted activity in the left lingual gyrus. We replicated these results in a second experiment that controlled for stimulus modality and anticipatory strategies. In Chapter 4, we used a training paradigm to investigate the role of feature diagnosticity (i.e., features that best distinguish between two otherwise similar categories) in semantic representations. Whereas subjects had knowledge of feature importance in novel object categorization, whether they used this information affected neural representations. Ventral temporal brain regions were more active during a separate retrieval task for subjects who learned and used the diagnostic feature for object categorization. Additionally, behavioral ratings of similarity predicted multivariate neural similarity. Collectively, this work suggests that semantic representations, integral to a memory system often thought of as free of contextual constraints, contain meaningful variations across contexts, people, and use.