Sex hormones, such as estrogens and androgens, regulate genomic and cellular processes that contribute to sex-specific disparities in the pathophysiology of various cancers. Sex hormones can modulate the immune signals and activities of tumor cells and tumor-associated leukocytes to support or suppress cancer progression. Therefore, hormonal differences between males and females play a crucial role in cancer immunity and in the response to therapies that exploit the intrinsic immune system to eliminate malignant cells. In this review, we summarize the impact of sex hormones in the breast cancer microenvironment, with a focus on how the hormonal environment affects tumor immunity. We also discuss the potential benefits of endocrine therapy used in combination with immunotherapy to strengthen the antitumor immune response.