We investigate the effect of particle polydispersity on the optical extinction spectra of colloidal aggregates of spherical metallic (silver) nanoparticles, taking into account the realistic interparticle gaps caused by layers of stabilizing polymer adsorbed on the metal surface (adlayers). The spectra of computer-generated aggregates are computed using two different methods. The coupled-multipole method is used in the quasistatic approximation and the coupled-dipole method beyond the quasistatics. The latter approach is applicable if the interparticle gaps are sufficiently wide relative to the particle radii. Simulations are performed for two different particle size distribution functions (bimodal and Gaussian), varying the number of particles per aggregate, and different distribution functions of the interparticle gap width. The strong influence of the latter factor on the spectra is demonstrated and investigated in detail.