Tumor-associated inflammation (TAI) is a feature of essentially all cancers and can confer both tumor-promoting and -suppressive functions. Cancer-associated fibroblasts (CAF) comprise one very heterogeneous cellular component of the tumor microenvironment characterized by a high degree of plasticity. Recent single-cell sequencing analyses revealed distinct CAF populations in various human cancers and helped to define key CAF subtypes, such as myofibroblastic, inflammatory, and antigen-presenting CAFs, with the first two being present in virtually all tumors. Importantly, these three CAF populations are involved in and modulate the positive and negative consequences of TAI. The remarkable plasticity of CAFs allows them to shift phenotypically and functionally in response to environmental changes. In this review, we describe how CAFs nurture tumor-promoting inflammation and suppress adaptive immunity. We also summarize the recently emerging evidence pertaining to tumor-suppressive CAF functions in the context of TAI. Finally, we summarize therapeutic concepts that aim at modulating CAF functions or depleting immunosuppressive CAFs to synergize with immunotherapy.