The ability of a virus to infect a cell type is at least in part determined by the presence of host factors required for the viral life cycle. However, even within cell types that express known factors needed for infection, not every cell is equally susceptible, suggesting that our knowledge of the full spectrum of factors that promote infection is incomplete. Here, we used single-cell clone tracing to retrospectively identify and characterize lung epithelial cells that are highly susceptible to infection with SARS-CoV-2. The transcriptional state of these highly susceptible cells includes markers of retinoic acid signaling and epithelial differentiation. Loss of candidate factors identified by our approach revealed that many of these factors play roles in viral entry. Moreover, a subset of these factors exert control over the infectable cell state itself, regulating the expression of key factors associated with viral infection and entry. Analysis of patient samples revealed the heterogeneous expression of these factors across both cells and patients in vivo. We further broaden our experimental framework to influenza A virus, demonstrating that a distinct subset of cells is intrinsically susceptible to infection as compared to SARS-CoV-2. Altogether, our results show that the variable expression of intrinsic cell states is a major determinant of whether a cell can be infected by a particular virus.