Resistance to immunotherapies is a significant problem for current clinical care of cancer patients. While tumor infiltrating T cells are essential for sensitivity of tumors to immunotherapies, factors that modulate T cell abundance in tumor microenvironments are not fully understood. To study tumor cell-intrinsic factors regulating anti-tumor immunity and sensitivity to immunotherapy, we established a novel experimental system by generating a library of congenic pancreatic tumor cell clones from a genetic mouse model driven by mutant Kras and p53. These tumor cell clones robustly formed implanted tumors that recapitulated T-cell-inflamed and non-T-cell-inflamed tumor microenvironments, associated with distinct features of T cells and myeloid cells. Importantly, we found that the non-T-cell-inflamed phenotype was dominant over the T-cell-inflamed phenotype in tumor microenvironments. Furthermore, we performed transcriptional and epigenetic profiling, genetic screening, as well as functional validation, and found that variation in the abundance of immune cell is dictated by molecular features of tumor cells. Mechanistically, tumor cells actively recruit suppressive myeloid cells to repress the infiltration and activation of T cells. Specifically, we identified multiple molecular mechanisms underlying this process and found that a closely connected network of epigenetic and transcriptional regulators (e.g. KDM3A, KLF5 SMAD4, and USP22) modulate the expression of signaling factors (e.g. EGFR and EPHA2) as well as immune modulating molecules (e.g. CXCL1, PTGS2 and CSF2) to control the interaction between tumor cells and immune cells. These results demonstrated that heterogeneity of tumor immune phenotypes can be driven by heterogeneous tumor-cell-intrinsic mechanisms.