Flocks of birds, schools of fish, and jams in traffic surprisingly mirror the collective motion observed in the microscopic wet worlds of living microbes, such as bacteria. While these small organisms were discovered centuries ago, scientists have only recently examined the dynamics and mechanics of suspensions that contain these swimming particles. I conduct experiments with the model organism and active colloid, the bacterium Escherichia coli, and use polymers, particles, and phase-separated mixtures to probe the non-equilibrium dynamics of bacterial suspensions. I begin by examining the hydrodynamic interactions between swimming E. coli and particles. For dilute suspensions of bacteria in Newtonian fluids, I find that larger particles can diffuse faster than smaller particles - a feature absent in passive fluids, which may be important in particle transport in bio- and geo-physical settings populated by microbes. Next, I investigate E. coli dynamics in non-Newtonian polymeric solutions. I find that cells tumble less and move faster in polymeric solutions, enhancing cell translational diffusion. I show that tumbling suppression is due to fluid viscosity while the enhancement in swimming speed is due to fluid elasticity. Visualization of single fluorescently-labeled DNA polymers reveals that the flow generated by individual E. coli is sufficiently strong that polymers can stretch and induce elastic stresses in the fluid. These, in turn, can act on the cell in such a way to enhance its transport. Lastly, I probe the interplay between kinetics, mechanics, and thermodynamic of active fluids by examining the evolution of an active-passive phase interphase. I create this interface by exposing regions of a dense bacterial swarm to UV light, which locally immobilizes the bacteria. Vortices etch the interface, setting interface curvature and speed. The local interface curvature correlates with the interface velocity, suggesting an active analog of the Gibbs-Thomson boundary condition. My results have implications for the burgeoning field of active soft matter, including insight into their bulk rheology, how material properties are defined and measured, and their thermodynamics and kinetics.