We present an experimental spectroscopic study of large random colloidal aggregates of silver nanoparticles undergoing local restructuring. We argue that such well-known phenomena as strong fluctuation of local electromagnetic fields, appearance of “hot spots” and enhancement of nonlinear optical responses depend on the local structure on the scales of several nanosphere diameters, rather than the large-scale fractal geometry of the sample.