A bacterial bath is a model active system consisting of a population of rodlike motile or self-propelled bacteria suspended in a fluid environment. This system can be viewed as an active, nonequilibrium version of a lyotropic liquid crystal or as a generalization of a driven diffusive system. We derive a set of phenomenological equations, which include the effects of internal force generators in the bacteria, describing the hydrodynamic flow, orientational dynamics of the bacteria, and fluctuations induced by both thermal and nonthermal noises. These equations violate the fluctuation dissipation theorem and the Onsager reciprocity relations. We use them to provide a quantitative account of results from recent microrheological experiments on bacterial baths.