The ability to recognize actions is important for cognitive development and social cognition. Areas in the lateral occipitotemporal cortex show increased activity when subjects view action sequences; however, whether this activity distinguishes between specific actions as necessary for action recognition is unclear. We used a functional magnetic resonance imaging adaptation paradigm to test for brain regions that exhibit action-specific activity. Subjects watched a series of action sequences in which the action performed or the person performing the action could be repeated from a previous scan. Three regions—the superior temporal sulcus (pSTS), human motion-sensitive cortex (MT/MST), and extrastriate body area (EBA)—showed decreased activity for previously seen actions, even when the actions were novel exemplars because the persons involved had not been seen previously. These action-specific adaptation effects provide compelling evidence that representations in the pSTS, MT/MST, and EBA abstract actions from the agents involved and distinguish between different particular actions.