We introduce a method for tracking multiple people in a cluttered street scene. We use global context to address the challenge of long occlusion by endowing each tracked object with a planning agent. This planner uses context of the street scene, people and other moving objects to reason about pedestrian intended behavior for tracking under occlusion and ambiguity. We extract short but robust trajectories called tracklets by tracking people with a simple appearance model. We formulate the tracking problem as a batch mode optimization, linking tracklets into paths, each with supporting evidence by an agent’s goal directed behavior, and image partial matching along the trajectory gap. We propose a global criteria for consistent linking of the tracklet with planning that can correct local ambiguity in linking. We test our algorithm in a challenging real world setting, where we automatically estimate scene context and intended goals, then track multiple people from a moving camera.