Bidisperse colloidal suspensions of temperature-sensitive microgel spheres were quenched from liquid to glass states by a rapid temperature drop, and then the glass was permitted to age. Irreversible rearrangements, events that dramatically change a particle’s local environment, were observed to be closely related to dynamic heterogeneity. The rate of these irreversible events decreased during aging and the the number of particles required to move as part of these irreversible rearrangements increased. Thus, the slowing dynamics of aging were governed by growing, correlated domains of particles. Additionally, short-range order developed, and a spatial decay length scale associated with orientational order was found to grow during aging.