Finding a specific visual target, such as your car keys, requires the brain to combine visual information about objects in the currently viewed scene with working memory information about your target to determine whether your target is in view. This combination of context-specific signals with visual information is thought to happen via feedback of target information from higher brain areas to the ventral visual pathway. However, exactly where and how these signals are combined remains unknown. To investigate, we recorded neural responses in V4 and inferotemporal cortex (IT) while monkeys performed an invariant object search task, where targets could appear across variation in their size, position and background context. We applied two complementary approaches to this data to investigate the neural mechanisms underlying target search. The first approach (Chapter 2) is from a computational perspective: where and how are visual and target signals combined when searching for a target? Specifically, we found that while task-relevant modulations in V4 were large, they were larger in IT, suggesting that top-down context-specific modulations are integrated into the ventral visual pathway at multiple stages. In Chapter 3, we focused on the neural responses recorded from IT from the perspective of neural coding: we sought to understand how signal and noise combine to determine task performance. We found that while signals that report the solution for object search were much smaller than signals that act as noise for the task (nuisance modulations) in IT cortex, nuisance modulations had a small effect on task performance. This counterintuitive finding was due to large trial variability constrained by short, behaviorally relevant spike counting windows. Together, this body of work provides insight into where and how the brain combines context-specific signals with visual information during invariant object search.