We calculate numerically the normal modes of vibrations in three-dimensional jammed packings of soft spheres as a function of the packing fraction and obtain the energy diffusivity, a spectral measure of transport that controls sound propagation and thermal conductivity. The crossover frequency between weak and strong phonon scattering is controlled by the coordination and shifts to zero as the system is decompressed toward the critical packing fraction at which rigidity is lost. We present a scaling analysis that relates the packing fraction dependence of the crossover frequency to the anomalous scaling of the shear modulus with compression. Below the crossover, the diffusivity displays a power-law divergence with inverse frequency consistent with Rayleigh law, which suggests that the vibrational modes are primarily transverse waves, weakly scattered by disorder. Above it, a large number of modes appear whose diffusivity plateaus at a nearly constant value before dropping to zero above the localization frequency. The thermal conductivity of a marginally jammed solid just above the rigidity threshold is calculated and related to the one measured experimentally at room temperature for most glasses.