Myeloid-derived suppressor cells (MDSCs) play a prominent role in the tumor microenvironment. A quantitative understanding of the tumor–MDSC interactions that influence disease progression is critical, and currently lacking. We developed a mathematical model of metastatic growth and progression in immune-rich tumor microenvironments. We modeled the tumor–immune dynamics with stochastic delay differential equations and studied the impact of delays in MDSC activation/recruitment on tumor growth outcomes. In the lung environment, when the circulating level of MDSCs was low, the MDSC delay had a pronounced impact on the probability of new metastatic establishment: blocking MDSC recruitment could reduce the probability of metastasis by as much as 50%. To predict patient-specific MDSC responses we fit to the model individual tumors treated with immune checkpoint inhibitors via Bayesian parameter inference. We reveal that control of the inhibition rate of natural killer (NK) cells by MDSCs had a larger influence on tumor outcomes than controlling the tumor growth rate directly. Posterior classification of tumor outcomes demonstrates that incorporating knowledge of the MDSC responses improved predictive accuracy from 63% to 82%. Investigation of the MDSC dynamics in an environment low in NK cells and abundant in cytotoxic T cells revealed, in contrast, that small MDSC delays no longer impacted metastatic growth dynamics. Our results illustrate the importance of MDSC dynamics in the tumor microenvironment overall and predict interventions promoting shifts toward less immune-suppressed states. We propose that there is a pressing need to consider MDSCs more often in analyses of tumor microenvironments.