Data-driven techniques have become synonymous with replication of real-world phenomena. Efforts have been underway to use these techniques in crowd simulation through a mapping of pedestrian trajectories onto virtual agents using a similarity of circumstance. These works have exposed two fundamental issues with data-driven crowds. First, robust real-world data is logistically difficult to accurately collect and filled with unknown variables, such as a person's mental state, which change behavior without providing a means to replicate their effects. Second, current data-driven approaches store and search the entire set of training data to decide the next course of action for each agent. A straightforward single-model system would alleviate the burden of storing and searching the data. The problem with a monolithic model, though, is that a single steering policy cannot handle all possible scenarios. To counter this we propose the splitting of possible scenarios into separable contexts, with each context in turn learning a model. The model used by an agent can then be dynamically swapped at runtime based on the evolving conditions around the agent. This results in a more scalable approach to data-driven simulation. In lieu of tracked data from real pedestrians, we propose the use of an oracle steering algorithm. This algorithm stands in for real data and can be queried for a steering decision for any combination of factors. This allows us to more thoroughly explore the problem space as needed. Furthermore, we can control all variables and capture behavior from scenarios that are otherwise infeasible to adequately sample in reality. This synthetic source of training data allows for a scalable and structured approach to training machine-learned models which virtual agents can use to navigate at runtime.