Primary resistance to CD19-directed chimeric antigen receptor T-cell therapy (CART19) occurs in 10% to 20% of patients with acute lymphoblastic leukemia (ALL); however, the mechanisms of this resistance remain elusive. Using a genome-wide loss-of-function screen, we identified that impaired death receptor signaling in ALL led to rapidly progressive disease despite CART19 treatment. This was mediated by an inherent resistance to T-cell cytotoxicity that permitted antigen persistence and was subsequently magnified by the induction of CAR T-cell functional impairment. These findings were validated using samples from two CAR T-cell clinical trials in ALL, where we found that reduced expression of death receptor genes was associated with worse overall survival and reduced T-cell fitness. Our findings suggest that inherent dysregulation of death receptor signaling in ALL directly leads to CAR T-cell failure by impairing T-cell cytotoxicity and promoting progressive CAR T-cell dysfunction.

Significance:

Resistance to CART19 is a significant barrier to efficacy in the treatment of B-cell malignancies. This work demonstrates that impaired death receptor signaling in tumor cells causes failed CART19 cytotoxicity and drives CART19 dysfunction, identifying a novel mechanism of antigen-independent resistance to CAR therapy.

See related commentary by Green and Neelapu, p. 492.

Engineered cellular immunotherapy is a novel platform in which autologous immune cells are collected from patients, genetically manipulated in vitro to redirect antigen specificity toward tumor-associated antigens, and then reinfused. T cells engineered to express chimeric antigen receptors targeting the B-cell surface protein CD19 (CART19) have generated substantial enthusiasm in the treatment of B-cell leukemias and lymphomas (1–5), and are now emerging as a standard approach for patients with relapsed and refractory disease. The broader efficacy of this therapy, however, is limited in a significant fraction of patients with B-cell malignancies by two forms of therapeutic failure: lack of disease response after cell infusion (primary resistance) and disease relapse after initial remission (acquired resistance). Several biological mechanisms have been identified that lead to acquired resistance by antigen loss (6–8), whereas the mechanisms responsible for primary resistance remain poorly understood.

Clinical experience has shown that up to 1 in 5 patients with CD19+ acute lymphoblastic leukemia (ALL) and approximately half of patients with diffuse large B-cell lymphoma do not achieve remission after CART19 treatment (1–3), indicating that a sizable proportion of CD19+ malignancies are intrinsically resistant to this therapy. Failed antitumor responses are often accompanied by loss of infused CAR T cells, which has led to the prevailing assumption that these therapeutic failures occur due to a poorly characterized inherent T-cell defect (9, 10). Understanding the molecular basis of resistance will not only illuminate the biology underlying this form of CART19 failure, but also identify strategies to overcome these barriers as we further refine cell-based immunotherapies.

To approach the problem of primary resistance from an unbiased perspective, we performed a CRISPR-based genome-wide loss-of-function screen in the Nalm6 ALL cell line under selective pressure from CART19. We identified that impairments in leukemia and lymphoma death receptor signaling lead to primary CART19 resistance, which in turn mediates progressive CAR T-cell functional impairment. Interrogation of clinical samples revealed a correlation between death receptor gene expression in pretreatment samples and response after CART19. Together, these data identify a novel mechanism of antigen-independent tumor resistance to CAR therapy.

Death Receptor Signaling Regulates Primary Resistance to CAR T-cell Immunotherapy

To identify the pathways that enable primary resistance to CART19, we performed an unbiased, genome-wide CRISPR/Cas9-based knockout (KO) screen (11) in the CD19+ human ALL cell line Nalm6 (Fig. 1A). Using the Brunello short guide RNA (sgRNA) library (12), we generated a pool of Nalm6 cells in which each cell had been edited for loss of function of a single gene, and combined these cells with CART19 for 24 hours to model primary resistance. In this short-term screening system, sgRNA sequencing demonstrated significant enrichment of guides targeting proapoptotic death receptor signaling molecules (FADD, BID, CASP8, and TNFRSF10B), and depletion of guides targeting antiapoptotic molecules (CFLAR, TRAF2, and BIRC2; Fig. 1B). Gene set enrichment analysis (GSEA) confirmed a strong association between sequenced guides and apoptosis pathway signaling [Kyoto Encyclopedia of Genes and Genomes (KEGG) pathway hsa04210], with a false discovery rate of 1.44 × 10−8 (Fig. 1C). Notably, all sequenced guide targets associated with the apoptosis gene set belonged to the death receptor–driven apoptotic signaling pathway, suggesting that death receptor signaling is a key regulator of primary resistance to CART19 in ALL.

Figure 1.

Death receptor signaling is a key mediator of resistance to CAR T-cell therapy. A, Schematic of genome-wide KO screen of Brunello library-edited Nalm6 ALL cells and CART19. B, Scatter plot of normalized MAGeCK beta scores representing enriched and depleted sgRNAs after Brunello screen. C, GSEA of the most-enriched and most-depleted sgRNAs identified after screen. D, Survival over time of WT, FADDKO, and BIDKO Nalm6 cells combined with CART19. E, GFP+ Nalm6 FADDKO or (F) BIDKO cells were combined with GFP-negative WT Nalm6 cells (25% GFP+ KO with 75% GFP-negative WT cells) and cocultured with either control of CART19 cells. Proportion of GFP+ cells over time is shown. G, GFP+ WT Nalm6 cells were combined with either control or CART19 cells in the presence of birinipant or vehicle and Nalm6 survival was measured over time. H–J, Survival over time of (H) WT, FADDKO, and BIDKO Nalm6 cells combined with CART22, (I) OCI-Ly10 cells combined with CART19, and (J) Nalm6 cells combined with an alternative CD19-targeted CAR bearing the CD28 costimulatory domain. K and L, Survival of immunodeficient mice engrafted with (K) FADDKO or (L) BIDKO Nalm6 cells after treatment with control or CART19 cells. **, P < 0.01; ***, P < 0.001; ****, P < 0.0001; ns, not significant.

Figure 1.

Death receptor signaling is a key mediator of resistance to CAR T-cell therapy. A, Schematic of genome-wide KO screen of Brunello library-edited Nalm6 ALL cells and CART19. B, Scatter plot of normalized MAGeCK beta scores representing enriched and depleted sgRNAs after Brunello screen. C, GSEA of the most-enriched and most-depleted sgRNAs identified after screen. D, Survival over time of WT, FADDKO, and BIDKO Nalm6 cells combined with CART19. E, GFP+ Nalm6 FADDKO or (F) BIDKO cells were combined with GFP-negative WT Nalm6 cells (25% GFP+ KO with 75% GFP-negative WT cells) and cocultured with either control of CART19 cells. Proportion of GFP+ cells over time is shown. G, GFP+ WT Nalm6 cells were combined with either control or CART19 cells in the presence of birinipant or vehicle and Nalm6 survival was measured over time. H–J, Survival over time of (H) WT, FADDKO, and BIDKO Nalm6 cells combined with CART22, (I) OCI-Ly10 cells combined with CART19, and (J) Nalm6 cells combined with an alternative CD19-targeted CAR bearing the CD28 costimulatory domain. K and L, Survival of immunodeficient mice engrafted with (K) FADDKO or (L) BIDKO Nalm6 cells after treatment with control or CART19 cells. **, P < 0.01; ***, P < 0.001; ****, P < 0.0001; ns, not significant.

Close modal

To confirm these observations and elucidate the functional significance of impaired tumor death receptor signaling, we focused on two of the most significantly enriched targets: FAS-associated protein with death domain (FADD), which serves as a membrane-proximal adaptor to all proapoptotic death receptors, and BH3 interacting-domain death agonist (BID), which is activated downstream of FADD and initiates mitochondrial activity that induces apoptosis (13). Nalm6 cells lacking either FADD or BID were generated using CRISPR/Cas9-based genome editing. Wild-type (WT) Nalm6 cells were combined with varying percentages of FADDKO or BIDKO Nalm6 cells (80%, 40%, or 20% KO) and then cultured with CART19. Cocultures with higher fractions of KO cells were more resistant to death in both short-term (Supplementary Fig. S1A and S1B) and long-term cultures (Fig. 1D). Long-term cocultures demonstrated a progressive enrichment of KO Nalm6 cells at the expense of WT Nalm6 cells over time (Fig. 1E and F). Notably, disruption of FADD or BID did not protect Nalm6 cells from chemotherapy-mediated killing (Supplementary Fig. S1C), indicating specific resistance to T cell–mediated cytotoxicity.

To validate the significance of death receptor signaling in CAR T-cell function, we combined WT Nalm6 cells with birinapant, a small-molecule antagonist of BIRC2, a suppressor of death receptor–driven apoptosis encoded by one of our most-depleted gRNAs (Fig. 1B). Although birinipant alone had no antileukemic activity, the combination of birinapant with CART19 led to synergistic killing (Fig. 1G), confirming the importance of this pathway in CART19 cytotoxic function. To further substantiate that death receptor signaling is essential for CAR T-cell cytotoxicity, we impaired death receptor ligand activity in T cells, beginning with disruption of the genes encoding FAS ligand (FASLG) or TRAIL in CART19 cells, and combined them with WT Nalm6 cells. Although the degree of functional impairment differed, loss of either FAS ligand or TRAIL significantly reduced CART19 cytotoxic activity (Supplementary Fig. S1D and S1E). To inhibit the activation of tumor necrosis factor receptor–driven apoptotic signaling, we combined CART19 and Nalm6 cocultures with a tumor necrosis factor α (TNFα) inhibitory antibody, and found that this also significantly impaired CART19 cytotoxicity (Supplementary Fig. S1F).

We extended this observation in several distinct models, including an additional CAR model (CART22; Fig. 1H; Supplementary Fig. S1G and S1H), two additional CD19+ ALL models (Supplementary Fig. S1I and S1J), and a model of diffuse large B-cell lymphoma (Fig. 1I; Supplementary Fig. S1K and S1L). We also found that FADD and BID loss led to resistance against CAR T cells engineered with a CD19-targeted CAR bearing the CD28 costimulatory domain (Fig. 1J). To evaluate the impact of impaired death receptor signaling on CART19 antitumor activity in vivo, we engrafted NOD/SCID/γc−/− (NSG) mice with WT, FADDKO, or BIDKO Nalm6 cells, and then delivered either control (nonengineered) T cells or a “stress” dose of CART19 (14). Despite the incomplete resistance to CART19-driven killing observed in our in vitro cytotoxicity assays, animals with KO leukemias demonstrated clear resistance to CART19 in vivo, with rapid disease progression after CART19 delivery (Supplementary Fig. S1M and S1N). Remarkably, CART19 treatment led to only a marginal improvement in median survival (4–5 days) in animals with KO leukemias (Fig. 1K and L). Together, these findings confirm that intact cancer cell death receptor signaling is broadly necessary for successful CAR T-cell cytotoxic function and that disruption of a single death receptor signaling gene in lymphoid cancer cells enables resistance to CAR T cells.

Impaired Tumor Death Receptor Signaling Drives Progressive CAR T-cell Dysfunction

T cells can kill target cells by ligation of surface death receptors to initiate the extrinsic apoptotic pathway or by secretion of cytolytic molecules (perforin and granzyme) that activate the mitochondrial-mediated intrinsic apoptotic pathway (13). We were intrigued by the observation that impairment of only extrinsic apoptosis enabled long-term tumor cell survival, suggesting a dominant-negative effect on T cells. To investigate this, we evaluated how loss of BID in Nalm6 cells affected broad CART19 functions. We conducted long-term coculture experiments and again demonstrated that BIDKO Nalm6 cells were resistant to CART19 whereas WT Nalm6 cells were eradicated (Supplementary Fig. S2A). Persistence of BIDKO Nalm6 cells was associated with impaired CART19 expansion (Fig. 2A), reduced perforin and granzyme production (Fig. 2B and C), and lower cytokine secretion over time (Supplementary Fig. S2B–S2D).

Figure 2.

Persistent exposure to ALL promotes progressive CART19 dysfunction. A, T-cell expansion over time after combination with Nalm6 cells. B and C, Perforin (B) and granzyme B (C) concentrations in culture supernatants over time after combination of CART19 and Nalm6 cells. D, Schematic of functional evaluation studies, in which CART19 cells were sorted after short-term (5-day) or long-term (15-day) initial coculture with either WT or BIDKO Nalm6 cells, and then reexposed to WT Nalm6 cells in secondary cultures. E, CART19 expansion after 48 and 120 hours of secondary culture after short-term initial culture. F, Cytokine secretion after 48 hours of secondary culture after short-term initial culture. G, Nalm6 survival after 48 and 120 hours of secondary culture after short-term initial culture. H, CART19 expansion after 48 and 120 hours of secondary culture after long-term initial culture. I, Cytokine secretion after 48 hours of secondary culture after long-term initial culture. J, Nalm6 survival after 48 and 120 hours of secondary culture after long-term initial culture. K and L, CART19 cells were combined with WT or BIDKO Nalm6 cells either once at the beginning of coculture or repeatedly and then sorted and recultured with WT Nalm6 cells as described. CART19 expansion was measured after 48 and 120 hours of secondary culture after (K) short-term and (L) long-term initial coculture. M and N, WT, PRF1KO, or TRAILKO CART19 cells were combined with WT Nalm6 cells and sorted as described. CART19 expansion was measured after 48 and 120 hours of secondary culture after (M) short-term and (N) long-term initial coculture. *, P < 0.05; **, P < 0.01; ***, P < 0.001; ****, P < 0.0001; ns, not significant.

Figure 2.

Persistent exposure to ALL promotes progressive CART19 dysfunction. A, T-cell expansion over time after combination with Nalm6 cells. B and C, Perforin (B) and granzyme B (C) concentrations in culture supernatants over time after combination of CART19 and Nalm6 cells. D, Schematic of functional evaluation studies, in which CART19 cells were sorted after short-term (5-day) or long-term (15-day) initial coculture with either WT or BIDKO Nalm6 cells, and then reexposed to WT Nalm6 cells in secondary cultures. E, CART19 expansion after 48 and 120 hours of secondary culture after short-term initial culture. F, Cytokine secretion after 48 hours of secondary culture after short-term initial culture. G, Nalm6 survival after 48 and 120 hours of secondary culture after short-term initial culture. H, CART19 expansion after 48 and 120 hours of secondary culture after long-term initial culture. I, Cytokine secretion after 48 hours of secondary culture after long-term initial culture. J, Nalm6 survival after 48 and 120 hours of secondary culture after long-term initial culture. K and L, CART19 cells were combined with WT or BIDKO Nalm6 cells either once at the beginning of coculture or repeatedly and then sorted and recultured with WT Nalm6 cells as described. CART19 expansion was measured after 48 and 120 hours of secondary culture after (K) short-term and (L) long-term initial coculture. M and N, WT, PRF1KO, or TRAILKO CART19 cells were combined with WT Nalm6 cells and sorted as described. CART19 expansion was measured after 48 and 120 hours of secondary culture after (M) short-term and (N) long-term initial coculture. *, P < 0.05; **, P < 0.01; ***, P < 0.001; ****, P < 0.0001; ns, not significant.

Close modal

We hypothesized that the prolonged exposure to Nalm6 cells that results from resistance to cytotoxicity was responsible for the observed CAR T-cell functional impairment. We investigated the kinetics of T-cell dysfunction by sorting CART19 cells after short (5 days) or long (15 days) exposure to either WT or BIDKO Nalm6 cells, and evaluated their ability to initiate effector functions upon reexposure to WT Nalm6 cells (Fig. 2D). CART19 cells isolated after short initial exposures to either WT or BIDKO Nalm6 cells demonstrated similar expansion, cytokine production, and antitumor cytotoxicity upon reexposure (Fig. 2EG; Supplementary Fig. S2E–S2H). In contrast, long initial exposures resulted in divergent functional abilities: CART19 cells initially cultured with BIDKO Nalm6 cells expanded poorly, produced less IFNγ, TNFα, and MIP1α, and were unable to kill WT Nalm6 cells upon reexposure (Fig. 2HJ; Supplementary Fig. S2I–S2L). Notably, similar defects were induced when CART19 cells were continuously reexposed to WT Nalm6 cells (Fig. 2K and L; Supplementary Fig. S3A–S3D), suggesting that dysfunction was not a result of impaired tumor death receptor signaling itself, but rather a consequence of the resultant antigen persistence. To further explore the role of persistent antigen exposure in a tumor-agnostic system, we combined WT Nalm6 cells with TRAILKO or PRF1KO CART19 cells. Loss of either molecule significantly impaired cytotoxic function (Supplementary Fig. S4A), consistent with our previous findings that CAR T cells can use multiple modalities to kill target cells (Supplementary Fig. S1D–S1E). Failure to eliminate target leukemia in initial cultures significantly impaired CART19 expansion (Supplementary Fig. S4B). CART19 cells were similarly sorted after short-term and long-term cultures (Supplementary Fig. S4C), and we again observed progressive loss of expansion potential over time (Fig. 2M and N). Effector cytokine production was also preserved after short initial cultures, and impaired after prolonged cultures for both TRAILKO and PRF1KO CART19 cells (Supplementary Fig. S4D and S4E). Collectively, these data demonstrate that impaired tumor death receptor signaling leads to the acquisition of CART19 dysfunction as a result of antigen persistence.

Dysfunctional CAR T Cells Demonstrate Molecular Features of Exhaustion

To identify the molecular programs responsible for CAR-driven T-cell dysfunction, we performed phenotypic, transcriptomic, and epigenomic interrogation of CART19 cells exposed to WT and BIDKO Nalm6 cells. Persistent exposure to antigen and prolonged activation of the endogenous T-cell antigen receptor (TCR) is the central driver of T-cell exhaustion in human and murine models of chronic viral infection and cancer (15–21), and thus we first evaluated expression of the exhaustion-related immunosuppressive surface proteins PD-1 and TIM3 and the exhaustion-related transcription factor Tbet. We observed an initial increase in expression of both surface markers on CART19 cells exposed to BIDKO Nalm6 cells or continuously exposed to WT Nalm6 cells (Fig. 3A and B; Supplementary Fig. S5A–S5D) and persistent elevations in Tbet expression (Supplementary Fig. S5E and S5F). Intriguingly, unlike the pattern seen in classic T-cell exhaustion, PD-1 levels returned to normal during the course of continuous antigen exposure (Fig. 3A and B). We next analyzed gene-expression profiles of CART19 cells at rest or after 15 days of exposure to WT or BIDKO Nalm6 cells. These three groups clustered independently by principal component analysis (PCA; Fig. 3C), confirming divergent transcriptional activity. We identified 1,285 differentially expressed genes in CART19 cells exposed to WT or BIDKO Nalm6 cells; BIDKO-exposed CART19 cells had a gene signature enriched in cell-cycle genes and metabolic regulators, accompanied by decreased expression of immunostimulatory genes (Supplementary Fig. S6A and S6B). More specifically, CART19 cells exposed to BIDKO Nalm6 cells demonstrated increased expression of known dysfunction-associated genes (EGR2, CTLA4, IRF4, BTLA, TOX2, and NR4A2; refs. 22–26) and decreased expression of T cell–activating genes, including TCR signaling molecules (SLAMF6, LCK, JAK1, and SOS1), stimulatory surface receptors (IL7R and ICOS), and transcriptional activators (STAT1, ID2, and NFATC3; Fig. 3D). Together, these data reflect that exposure to apoptosis-resistant tumors drives a dysfunctional transcriptional program in CART19.

Figure 3.

Persistent exposure to Nalm6 cells drives molecular programs of dysfunction in CART19. A and B, Cell-surface expression of (A) PD-1 and (B) TIM3 over time on CART19 cells combined with WT or BIDKO Nalm6 cells either once at the beginning of culture or repeatedly. C, PCA of gene-expression profiles of CART19 cells at rest or after 15 days of coculture with WT or BIDKO Nalm6 cells. D, Volcano plot of differentially expressed genes in CART19 cells exposed to either WT or BIDKO Nalm6 cells for 15 days. Transcripts with log-fold change >0.5 and FDR <0.05 are shown in red. E, Heat map of differentially accessible chromatin sites in CART19 cells at rest or after 15 days of coculture with WT or BIDKO Nalm6 cells. F, Exemplary ATAC-seq tracks indicating increased chromatin accessibility (highlighted in gray) in T cell–inhibitory transcription factors (noted in red) or decreased chromatin accessibility in proinflammatory transcription factor KLF2 (noted in green). *, P < 0.05; **, P < 0.01; ***, P < 0.001; ****, P < 0.0001; ns, not significant. MFI, median fluorescence intensity.

Figure 3.

Persistent exposure to Nalm6 cells drives molecular programs of dysfunction in CART19. A and B, Cell-surface expression of (A) PD-1 and (B) TIM3 over time on CART19 cells combined with WT or BIDKO Nalm6 cells either once at the beginning of culture or repeatedly. C, PCA of gene-expression profiles of CART19 cells at rest or after 15 days of coculture with WT or BIDKO Nalm6 cells. D, Volcano plot of differentially expressed genes in CART19 cells exposed to either WT or BIDKO Nalm6 cells for 15 days. Transcripts with log-fold change >0.5 and FDR <0.05 are shown in red. E, Heat map of differentially accessible chromatin sites in CART19 cells at rest or after 15 days of coculture with WT or BIDKO Nalm6 cells. F, Exemplary ATAC-seq tracks indicating increased chromatin accessibility (highlighted in gray) in T cell–inhibitory transcription factors (noted in red) or decreased chromatin accessibility in proinflammatory transcription factor KLF2 (noted in green). *, P < 0.05; **, P < 0.01; ***, P < 0.001; ****, P < 0.0001; ns, not significant. MFI, median fluorescence intensity.

Close modal

Given that CART19 dysfunction developed over time, we next interrogated CART19 chromatin accessibility to determine if persistent antigen exposure results in epigenetic imprinting associated with dysfunction. Concurrently with RNA sequencing (RNA-seq), we profiled CART19 cells using an assay for transposase-accessible chromatin with high-throughput sequencing (ATAC-seq). We observed significant chromatin remodeling in CART19 cells after exposure to BIDKO Nalm6 cells, with more modest changes in CART19 cells exposed to WT Nalm6 cells (Fig. 3E). Comparison of global promoter site accessibility revealed increased accessibility of immune activation genes and decreased accessibility of genes regulating metabolic processes and intracellular signaling in the BIDKO-exposed cells (Supplementary Fig. S6C and S6D), patterns that have been previously reported as an exhaustion-specific epigenetic signature (15). Notably, we did not observe changes in chromatin accessibility at “classic” exhaustion-defining loci observed in nonengineered T cells (i.e., PDCD1, CTLA4, TIGIT, EOMES, CD38, or PTGER2; Supplementary Table S1; refs. 15, 17, 21, 27). Given this distinction, we sought to identify the characteristic gene-regulation programs associated with CAR T-cell dysfunction resulting from persistent antigen exposure. We integrated our transcriptomic and epigenomic data to identify genes with differential expression and concordant promoter accessibility changes. We found increased expression and enhanced promoter accessibility of several key immunosuppressive transcription factors (TOX2, IRF8, and PRDM1) in BIDKO-exposed CART19 (Fig. 3F). Similarly, we found decreased expression and suppressed promoter accessibility of several immunostimulatory genes (LCK, DGKA, and KLF2; Fig. 3F; Supplementary Table S1). Together, these findings suggest that persistent exposure to leukemia leads to global programmatic changes in CART19 cells that are similar but not identical to classic, nonengineered T-cell exhaustion (15, 17, 21).

Death Receptor Gene Expression Correlates with Clinical Outcomes after CAR T-cell Therapy in ALL

In light of these preclinical observations, we investigated the association between leukemia expression of death receptor genes and clinical outcomes after treatment with tisagenlecleucel, a CD19 CAR T-cell clinical product (1). Using samples from two multicenter clinical trials of tisagenlecleucel for relapsed pediatric ALL (ClinicalTrials.gov identifiers NCT02435849 and NCT02228096), we evaluated all available leukemia-infiltrated bone marrow samples collected prior to treatment from patients who had durable (>1 year) complete remissions (CR; n = 17) and from patients who did not respond to treatment and had no evidence of CD19 antigen loss (NR; n = 8). Using the KEGG pathway database, we identified the proapoptotic genes specifically involved in either intrinsic (death receptor–independent) or extrinsic (death receptor–dependent) apoptosis to generate two distinct gene sets (Supplementary Table S2 and Methods). Using normalized expression data from patient samples, we generated two gene set scores (“death receptor signature” and “intrinsic apoptosis signature”) that represented overall expression of proapoptotic genes in each pathway. We found no difference in the intrinsic apoptosis signature between CRs and NRs (Supplementary Fig. S7A and S7B). In contrast, nonresponders had lower expression of death receptor genes and significantly lower death receptor signature scores than durable responders (P = 0.0126; Fig. 4A and B). Using ROC analysis, we identified a gene set score cutoff that provided high positive (85.7%; 95% CI, 70.6–94.4) and negative (88.9%; 95% CI, 53.4–98.2) predictive values of durable remission after tisagenlecleucel (Fig. 4B). We explored clinical outcomes in this patient cohort and found striking differences in overall survival between patients with high and low death receptor signature scores (Fig. 4C; P < 0.0001).

Figure 4.

Death receptor gene expression correlates with outcomes after tisagenlecleucel. A, Heat map of RNA expression in pretreatment leukemia-infiltrated bone marrow collected from patients treated with tisagenlecleucel (CR, n = 18; NR, n = 8). B, Integrated gene set expression score of death receptor–associated proapoptotic genes (“death receptor signature”) in CRs and NRs from pediatric patients with ALL. High and low scores were defined using ROC analysis. C, Overall survival of all pediatric patients analyzed. D and E, Pharmacokinetic analysis of tisagenlecleucel in peripheral blood over the first (D) 90 days and (E) 1,000 days after infusion, as measured by qPCR of CAR transcripts in patients with high (n = 20) or low (n = 6) death receptor signature scores. F, Death receptor signature scores in CRs and NRs from adult patients with ALL. High and low scores were again defined by ROC analysis. G, Overall survival of all adult patients analyzed. PPV, positive predictive value; NPV, negative predictive value.

Figure 4.

Death receptor gene expression correlates with outcomes after tisagenlecleucel. A, Heat map of RNA expression in pretreatment leukemia-infiltrated bone marrow collected from patients treated with tisagenlecleucel (CR, n = 18; NR, n = 8). B, Integrated gene set expression score of death receptor–associated proapoptotic genes (“death receptor signature”) in CRs and NRs from pediatric patients with ALL. High and low scores were defined using ROC analysis. C, Overall survival of all pediatric patients analyzed. D and E, Pharmacokinetic analysis of tisagenlecleucel in peripheral blood over the first (D) 90 days and (E) 1,000 days after infusion, as measured by qPCR of CAR transcripts in patients with high (n = 20) or low (n = 6) death receptor signature scores. F, Death receptor signature scores in CRs and NRs from adult patients with ALL. High and low scores were again defined by ROC analysis. G, Overall survival of all adult patients analyzed. PPV, positive predictive value; NPV, negative predictive value.

Close modal

CAR T-cell expansion and persistence are functional surrogates that are highly associated with clinical response (28, 29). Based on our preclinical observations, we hypothesized that exposure to death receptor–impaired leukemia may correlate with these markers of T-cell dysfunction in patients. Indeed, pharmacokinetic analysis revealed reduced tisagenlecleucel expansion and shorter persistence in patients with low death receptor signature scores as compared with patients with high scores (Fig. 4D and E). To account for varying leukemic infiltration in patient bone marrow, we adjusted gene set scores for the extent of leukemic blast involvement (see Methods). These adjusted death receptor signature scores were also highly associated with clinical outcome, tisagenlecleucel expansion and persistence, and overall patient survival (Supplementary Fig. S7C–S7F). Multivariate logistic regression analysis identified that the association between death receptor signature score and clinical response was mediated by the association between death receptor signature score and T-cell expansion, mirroring our in vitro observations of impaired T-cell expansion upon exposure to death receptor–impaired Nalm6 cells.

To validate the role of death receptor gene expression as a predictive marker of response and survival after tisagenlecleucel, we used the gene set and score cutoff developed using this pediatric sample set and applied it to an independent set of pretreatment bone marrow samples from adult patients enrolled on a phase I clinical trial at the University of Pennsylvania (NCT02030847; ref. 30). Consistent with our previous results, death receptor signature scores were significantly lower in nonresponders as compared with patients achieving CR (Fig. 4F) and was again associated with overall survival (Fig. 4G, P = 0.0079) in this second cohort.

scRNA Sequencing Reveals Dynamic Development of CAR T-cell Dysfunction in Clinical Samples

Based on our in vitro and in vivo observations that CAR T-cell dysfunction can be acquired over time during persistent exposure to leukemic cells and ultimately leads to therapeutic failure, we pursued a granular assessment of how CAR T-cell dysfunction phenotypes evolved in patients. Using stored tissue from the initial phase I trial of CD19 CAR T cells in pediatric ALL (NCT01626495), we analyzed T cells from the originally manufactured infusion product and from patient peripheral blood at time of peak CAR T-cell expansion from a nonresponder and complete responder using single-cell RNA-seq (scRNA-seq). We focused our analysis on exhaustion-defining genes (ref. 15; Supplementary Table S3). T cells in the infusion products (Fig. 5A) from the responder and nonresponder demonstrated nearly identical expression of exhaustion-associated genes (Fig. 5B and C). However, 10 days after infusion, T cells collected from the peripheral blood of these two patients (Fig. 5D) acquired considerably different phenotypes: T cells from the patient who went on to have a complete response maintained the preinfusion transcriptional profile, whereas T cells from the patient who went on to demonstrate no response expressed much higher levels of exhaustion-associated genes (Fig. 5E and F). This pattern was also seen when analyzed specifically in CAR+ T cells (Fig. 5GL). These findings demonstrate acquisition of a dysfunctional CAR T-cell transcriptional program in a nonresponding patient not seen in a patient who experienced durable CR.

Figure 5.

Single-cell analysis reveals the development of CAR T-cell dysfunction in patients. A, UMAP projection of T cells contained in tisagenlecleucel infusion products prepared for pediatric patients who would go on to have either a complete response or no response after treatment. Cells are color-coded to differentiate the responding (gray) and nonresponding (gold) patients. B, T cells from the infusion products are color-coded to indicate expression of exhaustion-related gene transcripts (see Supplementary Table S3 for list of genes). C, Bar graphs reflecting proportion of T cells from the infusion products that express 0, 1, 2, 3, or ≥4 exhaustion-related genes. Similar representations are shown for (D–F) peripheral blood T cells collected at peak expansion after tisagenlecleucel infusion, (G–I) CAR+ T cells in the infusion products, and (J–L) peripheral blood CAR+ T cells collected at peak expansion after tisagenlecleucel infusion.

Figure 5.

Single-cell analysis reveals the development of CAR T-cell dysfunction in patients. A, UMAP projection of T cells contained in tisagenlecleucel infusion products prepared for pediatric patients who would go on to have either a complete response or no response after treatment. Cells are color-coded to differentiate the responding (gray) and nonresponding (gold) patients. B, T cells from the infusion products are color-coded to indicate expression of exhaustion-related gene transcripts (see Supplementary Table S3 for list of genes). C, Bar graphs reflecting proportion of T cells from the infusion products that express 0, 1, 2, 3, or ≥4 exhaustion-related genes. Similar representations are shown for (D–F) peripheral blood T cells collected at peak expansion after tisagenlecleucel infusion, (G–I) CAR+ T cells in the infusion products, and (J–L) peripheral blood CAR+ T cells collected at peak expansion after tisagenlecleucel infusion.

Close modal

Until now, primary resistance to CART19 was presumed to be a result of intrinsic T-cell functional impairments present at the time of product infusion (9, 10). Our data offer a novel model in which tumor-intrinsic biology contributes to observed therapeutic failure, wherein T cells instead develop dysfunction as a result of persistent exposure to cancer cells that are resistant to T cell–mediated killing (Fig. 6).

Figure 6.

Proposed model for the mechanism of impaired tumor death receptor signaling that leads to CAR T-cell dysfunction and therapeutic failure.

Figure 6.

Proposed model for the mechanism of impaired tumor death receptor signaling that leads to CAR T-cell dysfunction and therapeutic failure.

Close modal

Through an unbiased, genome-wide loss-of-function screen, we identified the critical importance of death receptor signaling in tumor cells on CAR T-cell cytotoxic function. We further validated the role of death receptor signaling in additional ALL and non-Hodgkin lymphoma cell lines, using two different B cell–directed CARs, and in both 4-1BB and CD28 costimulated CARs. This mechanism appears to rely on two phases: an initial resistance to death receptor–driven killing, followed by an antigen-driven, progressive impairment in CAR T-cell function. Together, this leads to CAR T-cell failure that perpetuates disease progression.

To explain the observed functional impairments, we conducted RNA-seq and ATAC-seq and found significant transcriptional and epigenetic reprogramming induced by persistent antigen exposure, suggesting that persistent CAR engagement has a broad impact on cellular activity and identity. Interestingly, although similar in many ways, these dysfunctional CAR T cells did not recapitulate the epigenetic hallmarks of classic T-cell exhaustion. We speculate that these distinctions may arise from differences in signaling that result from 4-1BB costimulation, and further work is needed to test this hypothesis and determine its functional relevance.

Primary resistance to CD19 CAR therapy is also a significant problem in non-Hodgkin lymphoma and chronic lymphocytic leukemia (2, 31). In this study, we used models of ALL for mechanistic studies and correlated our findings using clinical studies of CAR T cells in ALL. A recent study identified the role of the extrinsic apoptotic signaling pathway in regulating the antitumor T-cell response after checkpoint blockade in solid tumors (32), suggesting that this pathway may be a central regulator of antitumor responses to distinct modalities of immunotherapy.

A possible implication of our observations is that the use of heathy donor (i.e., allogeneic donor or “universal” donor) T cells as a substrate for CAR T-cell manufacturing may face the same barriers as autologous products. Understanding how intrinsic and acquired T-cell dysfunction cooperate to cause therapeutic failure will be critical to the design of the next generation of cellular therapies.

Brunello Guide Library Screens

The Brunello sgRNA KO plasmid library was designed and produced as previously described (11, 12). For generation of the lentiviral KO library, four 15-cm dishes of HEK293T cells (ATCC ACS-4500) were seeded at ∼40% confluence the day before transfection in D10 media (DMEM supplemented with 10% fetal bovine serum). One hour prior to transfection, media were removed, and 13 mL of prewarmed reduced serum OptiMEM media was added to each flask. For each dish, 20 μg of plasmid library, 10 μg of pVSVg, and 15 μg of psPAX2 (Addgene) were diluted in 2 mL OptiMEM. Lipofectamine 2000 (100 μL; Invitrogen) was diluted in 2 mL OptiMEM, and after a 5-minute incubation was added to the diluted DNA mixture. The complete mixture was incubated for 20 minutes before being added to cells. After 6 hours, the media were changed to 15 mL D10. Media containing lentiviral particles were removed after 48 hours and stored in 1 mL aliquots at −80°C. To find optimal virus volumes to achieve a multiplicity of infection (MOI) of 0.2 to 0.3, each viral production was individually titrated. Briefly, 2 × 106 cells were plated into each well of a 12-well plate, and varying volumes of virus (between 0 and 160 μL) were added. Cells were counted, and each well was split into two plates and diluted 1:10; one plate underwent selection with puromycin (1 μg/mL). After 48 hours, each well was counted, and viability was assessed to determine percent transduction (calculated as cell count from the replicate with puromycin divided by cell count from the replicate without puromycin). The virus volume yielding an MOI closest to 0.2 was chosen for large-scale screening. For long-term depletion screening, 2 × 108 Nalm6 cells (edited as described above) were combined with 5 × 107 CART19 cells or control T cells [effector:target (E:T) ratio 1:4] and cocultured in standard culture media; 5 × 107 control Nalm6 cells were frozen for genomic DNA analysis. Each condition was established in duplicate, and cocultures were monitored by flow cytometry over time and maintained at a concentration of 1 × 106 cells/mL. Cultures were monitored until Nalm6 cells recurred in selection cultures, at which time dead cells were removed using a dead cell removal magnetic bead kit (Miltenyi), and living cells were frozen for genomic DNA analysis. Short-term screening cocultures were established in the same manner; however, dead cell removal and live-cell collection were performed after 24 hours of coculture.

Genomic DNA Extraction and Guide Sequencing

From each screening culture, 3 × 107 to 5 × 107 were flash-frozen as dry cell pellets. At time of DNA extraction, 6 mL of NK lysis buffer (50 mmol/L Tris, 50 mmol/L EDTA, 1% SDS, pH 8) and 30 μL of 20 mg/mL Proteinase K were added to the frozen cell sample and incubated at 55°C overnight. The following day, 30 μL of 10 mg/mL RNase A, diluted in NK lysis buffer to 10 mg/mL and then stored at 4°C, was added to the lysed sample, which was then inverted 25 times and incubated at 37°C for 30 minutes. Samples were cooled on ice before the addition of 2 mL of prechilled 7.5 M ammonium acetate to precipitate proteins. The samples were vortexed at high speed for 20 seconds and then centrifuged at ≥ 4,000 × g for 10 minutes. After the spin, a tight pellet was visible in each tube, and the supernatant was carefully decanted into a new 15-mL tube. Six milliliters 100% isopropanol was added to the tube, inverted 50 times, and centrifuged at ≥ 4,000 × g for 10 minutes. Genomic DNA (gDNA) was visible as a small white pellet in each tube. The supernatant was discarded, and 6 mL of freshly prepared 70% ethanol was added to the tube, inverted 10 times, and centrifuged at ≥4,000 × g for 1 minute. The supernatant was discarded by pouring; the tube was briefly spun, and remaining ethanol was removed using a pipette. After air drying for 10 to 30 minutes, 500 μL of 1× TE buffer was added, the tube was incubated at 65°C for 1 hour and at room temperature overnight to fully resuspend the DNA. The next day, the gDNA samples were vortexed briefly, and gDNA concentration was measured.

To measure the distribution of sgRNA within each screen arm, we used Illumina next-generation sequencing applied to an amplicon generated from a single targeted PCR of the integrated sgRNA cassette (11). Briefly, we used all collected gDNA (1,000× coverage) divided into 100 μL PCR reactions with 5 μg of DNA per reaction. We used Takara ExTaq DNA Polymerase and the default mix protocol with the following PCR program: [95° 2 minutes (98° 10 seconds, 60° 30 seconds, 72° 30 seconds) × 24, 72° 5 minutes]. PCR products were gel-purified using the QiaQuick Gel Extraction Kit (Qiagen). The purified pooled library was then sequenced on a HiSeq4000 with ∼5% PhiX added to the sequencing lane. Quality assessment was done by qubit (for concentration), bioAnalyzer (for size distribution), and Kapa Library Quantification (for clusterable molarity).

Genome-Wide Screening Data Analysis

To count the number of reads associated with each sgRNA in each Fastq file, we first extracted the sgRNA targeting sequencing using a regular expression containing the three nucleotides flanking each side of the sgRNA 20 bp target. sgRNA spacer sequences were then aligned to a preindexed Brunello library (Addgene) using the short-read aligner “bowtie” using parameters (-v 0 -m 1). Data analysis was performed using custom R scripts. Using the guide sequencing data, we analyzed enrichment and depletion of guide RNAs with the Model-based Analysis of Genome-wide CRISPR/Cas9 Knockout (MAGeCK) algorithm using the maximum likelihood estimation module (35, 36).

CRISPR/Cas9 Guide Design, Genomic Engineering, and Indel Detection

CRISPR sgRNAs were designed using software integrated into Benchling (http://Benchling.com). For each target gene, six sgRNA sequences were designed to target early exon sequences, and in vitro transcribed using the GeneArt Precision gRNA Synthesis Kit (Invitrogen) for screening. Cells were electroporated using the Lonza 4D-Nucleofector Core/X Unit. Lymphoid cancer cells were electroporated using the SF Cell Line 4-D Nucleofector Kit, and primary T cells were electroporated using the P3 Primary Cell 4D Kit (Lonza). For Cas9 and sgRNA delivery, the ribonucleoprotein (RNP) complex was first formed by incubating 10 μg of TrueCut Cas9 Protein V2 (Lonza) with 5 μg of sgRNA for 10 to 30 minutes at room temperature. Cells were spun down at 300 × g for 10 minutes and resuspended at a concentration of 3 × 106 to 5 × 106 cells/100 μL in the specified buffer. The RNP complex, 100 μL of resuspended cells, and 4 μL of 100 μmol/L IDT Electroporation Enhancer (IDT) were combined and electroporated. Pulse codes EO-115 and CV-104 were used for primary T cells and lymphoid cancer cells, respectively. After electroporation, the cells were incubated in standard media at a 5 × 106 cells/mL at 30°C for 48 hours, then cultured at 37°C for the duration of experimental procedures. Tracking of Indels by DEcomposition (TIDE) analysis was used to detect KO efficiency at the genomic level (37). Genomic DNA from electroporated cells was isolated (Qiagen DNeasy Blood and Tissue Kit), and 200 to 300 ng was PCR amplified using Accuprime Pfx SuperMix and 10 μmol/L forward/reverse primers flanking the region of interest. Primers were designed such that the amplicon was at a target size ∼1 kb. PCR products were gel-purified and sequenced, and trace files were analyzed using Desktop Genetics software (tide.deskgen.com, Desktop Genetics) to determine KO efficiency. R2 values were calculated, reflecting goodness of fit after nonnegative linear modeling by TIDE software (37). See Supplementary Table S4 for sgRNA sequences and sequencing primer sequences.

General Cell Culture

Unless otherwise specified, cells were grown and cultured at a concentration of 1 × 106 cells/mL of standard culture media (RPMI1640 + 10% FCS, 1% penicillin/streptomycin, 1% HEPES, 1% nonessential amino acids) at 37°C in 5% ambient CO2.

Lentiviral Vector Production and Transduction of CAR-Engineered Human T Cells

Replication-defective, third-generation lentiviral vectors were produced using HEK293T cells (ATCC ACS-4500). Approximately 8 × 106 cells were plated in T150 culture vessels in standard culture media and incubated overnight at 37°C. Eighteen to 24 hours later, cells were transfected using a combination of Lipofectamine 2000 (96 μL, Invitrogen), pMDG.1 (7 μg), pRSV.rev (18 μg), pMDLg/p.RRE (18 μg) packaging plasmids and 15 μg of expression plasmid (either CD19 or CD22-targeted CAR). Lipofectamine and plasmid DNA was diluted in 4 mL Opti-MEM media prior to transfer into lentiviral production flasks. At both 24 and 48 hours following transfection, culture media were isolated and concentrated using high-speed ultracentrifugation (25,000 × g for 2.5 hours). Human T cells were procured through the University of Pennsylvania Human Immunology Core. CD4+ and CD8+ cells were combined at a 1:1 ratio and activated using CD3/CD28 stimulatory beads (Thermo-Fisher) at a ratio of 3 beads/cell and incubated at 37°C overnight. The following day, CAR lentiviral vectors were added to stimulatory cultures at an MOI between 4 and 6. Beads were removed on day 6 of stimulation, and cells were counted daily until growth kinetics and cell size demonstrated they had rested from stimulation. All experiments used a CAR encoding the CTL019 chimeric antigen receptor, composed of the FMC63 scFv, 4-1BB, and CD3ζ costimulatory domains, unless otherwise noted.

Bioluminescence-Based Cell Survival Assays

Cell lines (Nalm6, REH, RS4;11, and OCI-Ly10) were engineered to express click beetle green, and cell survival was measured using bioluminescence quantification. D-luciferin potassium salt (Perkin-Elmer) was added to cell cultures (final concentration 15 μg/mL) and incubated at 37°C for 10 minutes. Bioluminescence signal was detected using a BioTek Synergy H4 imager, and signal was analyzed using BioTek Gen5 software. Percent specific lysis was calculated using a control of target cells without effectors. For cultures including birinapant, cytotoxicity assays were established as described with the addition of either 500 or 1,000 nmol/L birinapant.

Long-Term Coculture Assays

CAR T cells were combined with target leukemia cells at an E:T ratio of 1:4, and cocultures were evaluated for absolute count of T cells and leukemia cells by flow cytometry using CountBright absolute counting beads (Thermo-Fisher). Cultures were maintained at a concentration of 1 × 106 total cells/mL, and CAR+ T cells were sorted on days 5 and 15 as described. T cells were then recombined with target leukemia cells at an E:T ratio of 1:8 for reexposure studies, and effector functions were measured as described. For continuous exposure cocultures, cells were counted every day, and cultures were re-fed with leukemia cells to maintain a constant and equivalent number of leukemia cells across cocultures.

Flow Cytometry

Cells were resuspended in FACS staining buffer (PBS+ 3% fetal bovine serum) using the following antibodies: CD3 (clone OKT3, eBioscience), CD4 (clone OKT4, eBioscience), CD8 (clone SK1, BD), PD-1 (clone J105, eBioscience), TIM3 (clones F38-2E2, BioLegend), and Tbet (clone 4B10, eBioscience). CD19 CAR was detected using an anti-idiotype antibody provided by Novartis Pharmaceuticals. All changes in overall tumor or T-cell counts reflect changes in absolute cell counts, which were determined using CountBright absolute counting beads (Thermo-Fisher). Cell viability was established using Live/Dead Aqua Fixable Staining Kit (Thermo-Fisher), and data were acquired on an LSRII Fortessa Cytometer (BD). All data analyses were performed using FlowJo 9.0 software (FlowJo, LLC).

Cytokine and Cytolytic Molecule Quantification

Human cytokine quantification was performed using a custom 31-plex Luminex panel (EMD Millipore) containing the following analytes: EGF, FGF2, Eotaxin, sIL2Ra, G-CSF, GM-CSF, IFNα2, IFNγ, IL10, IL12P40, IL12P70, IL13, IL15, IL17A, IL1RA, HGF, IL1β, CXCL9/MIG, IL2, IL4, IL5, IL6, IL7, CXCL8/IL8, CXCL10/IP10, CCL2/MCP1, CCL3/MIP1α, CCL4/MIP1β, RANTES, TNFα, and VEGF. Cell culture supernatants were flash-frozen on dry ice and thawed at the time of cytokine analysis. Assays were established per manufacturer recommendations. Data were acquired on a FlexMAP 3D quantification instrument, and analysis was done using xPONENT software. Data quality was determined by ensuring the standard curve for each analyte had a 5P R2 value > 0.95 with or without minor fitting using xPONENT software. To pass assay technical quality control, the results for two controls in the kit were required to be within the 95% confidence interval provided by the vendor for >25 of the tested analytes. We established in vitro quantification of granzyme B (antibody sets from R&D Systems) and perforin (antibody sets from MABTECH), using ELISA substrate ADHP from Cayman Chemical, assay plates from E&K Scientific, and substrate plates from Greiner Bio-One. All ELISA reagents were prepared according to the protocols for DuoSet ELISA except for color reagent B, which was supplemented with ADHP at 100 μmol/L. Instead of coating the capture antibody to the wells of the ELISA plate, it was coated on the surfaces of macrospheres, which enabled the measurement of both analytes using 100 μL of cell supernatant (collected as described above). Assays were set up using bead-strips in assay plates based on an assay map following the protocol for antibody sets. At the end of the assay, one substrate plate per 12 bead-strips for each analyte was prepared by adding 100 μL/well of substrate solution (1:1 of color reagent A and ADHP). Each bead-strip was placed in one column of the substrate plate according to the assay map. Color development was for 10 to 30 minutes. Plates were read on a FLUO STAR OMEGA instrument using fluorescence channel at 530 nm (excitation) and 590 nm (emission). Data were analyzed using Omega Data Analysis software.

Xenograft Mouse Models

Six- to 10-week-old NOD/SCIDγc−/− (NSG) mice were obtained from the Jackson Laboratory and maintained in pathogen-free conditions. Animals were injected via tail vein with 1 × 106 WT, BIDKO, or FADDKO Nalm6 cells in 0.2 mL sterile PBS. On day 7 after tumor delivery, 5 × 105 T cells (control or CAR+) were injected via tail vein in 0.2 mL sterile PBS. Animals were monitored for signs of disease progression and overt toxicity, such as xenogeneic graft-versus-host disease, as evidenced by >10% loss in body weight, loss of fur, diarrhea, conjunctivitis, and disease-related hind limb paralysis. Disease burdens were monitored over time using the Xenogen IVIS bioluminescence imaging system.

RNA-seq and Analysis

RNA-seq libraries were made following the previously established SMARTseq2 protocol (38). Briefly, total RNA was extracted using Qiazol (Qiagen) from 300 CART19 cells and recovered by RNA Clean and Concentrator spin columns (Zymo) followed by incubation with Oligo-dT. The transcription reaction was carried out on 100 pg of cDNA for 1 minute at 55°C. Libraries were uniquely barcoded (39) and amplified for 14 cycles. Fragment size distribution was verified, and paired-end sequencing (2 × 75 bp reads) was carried out on an Illumina NextSeq 500.

Raw reads were mapped to the GRCh37/hg19 genome assembly using the RNA-seq aligner STAR (version 2.5.4b; ref. 40). The algorithm was given known gene models provided by GENCODE (release_27_hg19, www.gencodegenes.org) to achieve higher mapping accuracy. Quantification was also performed by STAR using the –quantMode GeneCounts flag. Raw counts were normalized (TMM normalization implemented in edgeR, followed by the voom transformation). Lastly, the Bioconductor package limma was used to assess differential expression using linear models. Genes with differential expression at a false discovery rate (FDR) < 0.05 and a log-fold change (LFC) > 0.5 between replicates were considered for further analysis. Visualization was performed using the R package ggplot2.

ATAC-seq

Omni ATAC-seq libraries were made as previously described (41). Briefly, nuclei were isolated from 50,000 sorted CART19 cells, followed by the transposition reaction using Tn5 transposase (Illumina) for 30 minutes at 37°C with 1000rp mixing. Purification of transposed DNA was completed with DNA Clean and Concentrator (Zymo), and fragments were barcoded with ATAC-seq indices (39). Final libraries were double size selected using AMPure beads prior to sequencing. Paired-end sequencing (2 × 75 bp reads) was carried out on an Illumina NextSeq 500 platform. Adapters were trimmed using atactk (version 0.1.5, https://atactk.readthedocs.io/en/latest/index.html), and raw reads were aligned to the GRCh37/hg19 genome using bowtie (42) with the following flags: –chunkmbs 2000 –sam –best –strata -m1 -X 2000. MACS2 (43) was used for peak calling with an FDR cutoff of 0.05. Downstream analysis and visualization was done using HOMER (44) and deepTools2 (45).

Apoptotic Pathway Gene Set Development

Using the KEGG “Apoptosis” pathway (hsa04210), we established two distinct gene set signatures. We determined to which pathway each gene belonged, either extrinsic (death receptor–dependent) or intrinsic (death receptor–independent) apoptosis. Genes that did not fall into either pathway (i.e., nuclear factor kappa-light-chain-enhancer of activated B cell–associated, nerve-growth factor–associated; ref. 13) were excluded. Genes were then classified for activity, either proapoptotic or antiapoptotic, and proapoptotic genes were included in gene sets.

Evaluation of Clinical Specimens

Bone marrow (BM) specimens were collected from patients enrolled in three multicenter clinical trials (NCT02435849/ELIANA, NCT02228096/ENSIGN, and NCT02030847). The study protocols and their amendments were approved by the Institutional Review Board at each participating institution and were conducted in accordance with the principles of the Declaration of Helsinki and the International Conference on Harmonization of Good Clinical Practice guidelines including written informed consent.

All relevant ethical regulations were complied with in this study, and informed consent obtained at time of study enrollment allowed for RNA-seq analysis on all patient samples evaluated. For purposes of this analysis, patients were categorized as achieving complete remission (CRs) or nonresponders (NRs). CRs were defined as patients who achieved either complete remission or complete remission with incomplete hematologic recovery within 3 months and stayed in complete remission for at least 1 year after CAR infusion (1) for the pediatric trials, and as achieving morphologic remission within 3 months after CAR infusion for adult patients. The cellular kinetics of tisagenlecleucel were determined as previously described (28). Briefly, measurement of the synthetic CAR transgene by quantitative PCR. The cellular kinetic parameter of expansion was derived from peripheral blood samples and estimated by noncompartmental methods using Phoenix (Pharsight, Version 6.4). Graphical analysis of transgene persistence was performed and categorized by high and low scores. BM blast percentage quantification for each pediatric patient was assessed by the Minimal Residual Disease (MRD) assay. Briefly, BM aspirates were collected in sodium heparin vacutainer tubes, maintained at room temperature, and tested within 5 days after collection. Immunophenotyping was performed starting with approximately 2 × 107 total white blood cells (WBC). WBC gate was adjusted to include WBCs and exclude red blood cells, dead/dying cells, and debris. The number of viable WBCs identified by 7-AAD was used for calculating the MRD as a percentage of total WBCs. A 4-tube, 8-color flow cytometry assay was performed using the BD FACSCanto II instrument to determine B-ALL MRD and leukemia-associated immunophenotypes.

RNA-seq and Bioinformatics Analysis on Clinical Trial Samples

Total RNA was extracted from BM cells stored in PAXgene tubes according to the manufacturer's instructions (Qiagen). Integrity was checked on the Agilent TapeStation (RIN), followed by preparation for sequencing using the TruSeq RNA v2 prep (Illumina). High-throughput sequencing was performed on an Illumina HiSeq 2500 platform to a target depth of 50 million paired-end reads per sample. Fastq files were processed for data quality control, read mapping, transcript assembly, and transcript abundance estimation. A number of quality control metrics were assessed including data quality and guanine and cytosine content on per base and sequence levels, sequence length distribution and duplication levels, and insert size distribution. Finally, HTSeq was used to count the number of reads mapping to each gene (46). Data normalization for differential expression analysis was carried out with the edgeR R package (47). The logCPM value of each gene was z-normalized and gene set scores were calculated as the mean of the normalized gene expression for each gene in a given gene set. In order to account for the variability of the marrow blast count, the analysis was repeated, and the gene set score was normalized to regress out the blast percentage. To identify a threshold gene set score with high positive and negative predictive value of clinical outcomes in our pediatric sample set, we performed ROC analysis on the cohort of gene set scores.

scRNA-seq and Data Analysis

One nonresponding and one responding patient from a previous clinical trial (NCT01626495) were selected for single-cell analysis. Frozen vials of CAR T-cell infusion product and peripheral blood were gently thawed and counted, and dead cells were removed (Dead Cells Removal Kit, Miltenyi, #130-090-101). Resulting cell samples had viabilities of 92% to 98% and were processed using the 10× Genomics Chromium Single-Cell V(D)J Reagent Kits (PN-1000006, PN-1000020, and PN-120262) to generate single-cell emulsions for barcoding, reverse transcription, and cDNA amplification. Immediately following these steps, 10× barcoded fragments were pooled and attached to standard Illumina adaptors to generate a barcoded single-cell RNA library. Sequencing libraries were quantified by qPCR before sequencing on the Illumina platform using the HiSeq 4000 instrument.

For each sample, raw sequencing reads from all sequence runs were mapped to the human reference genome (GRCH38) using CellRanger V2.1.1 (10X Genomics). Gene-expression profiles were further processed using Seurat package v3 (48). To remove doublets and broken cells, any cells with fewer than 500 expressed genes and with unusually high (above 3 median absolute deviations) UMI counts or unusually high percentages of mitochondrial gene transcripts were discarded from downstream analysis. Clustering was performed using the standard Seurat package pipeline. Gene-expression profiles were transformed using the SCTransform function, followed by PCA for dimensionality reduction, and the number of significant PCs was determined using an elbow plot. Next, cells were clustered using the “FindClusters” function with 0.6 resolution and visualized using the uniform manifold approximation and projection (UMAP) algorithm. To compare T cells from infusion products and peak expansion samples between patients, cells from both samples were first merged, clustered, and visualized using the UMAP algorithm. Then, T cells, defined as CD3ϵ+ and CD79A cells, were isolated and reclustered. For CAR T-cell comparison, CAR T cells, defined as CD3ϵ+, CAR+, and CD79A cells, were isolated and used for cluster analysis. To map the number of exhaustion genes expressed, we first identified cells that expressed any of the exhaustion-associated genes, and then further identified cells that expressed combinations of 2, 3, or ≥4 of these marker genes. Finally, cells were color-coded based on the number of genes expressed.

Statistical Analysis

All in vitro data presented are representative of at least three independent experiments, except for the short-term KO screens (performed once each with four biological replicates), RNA-seq, and ATAC-seq (performed once each with three biological triplicates). In vivo animal studies were performed twice. All comparisons between two groups were performed using either a two-tailed unpaired Student t test or Mann–Whitney test, depending on normality of distribution. Comparisons between more than two groups were performed by two-way ANOVA with Bonferroni correction for multiple comparisons. All results are represented as mean ± SEM. Survival data were analyzed using the log-rank (Mantel–Cox) test.

Data Availability

Guide library sequencing, RNA-seq, and ATAC sequencing data are available from the Gene Expression Omnibus using the accession number GSE130663 for all data sets. RNA-seq and ATAC-seq expression data are included in Supplementary Table 1. Other data generated from this study are available from the corresponding authors upon reasonable request.

N. Singh holds patents related to CAR T-cell therapy at the University of Pennsylvania. E.J. Orlando is an investigator at Novartis and has ownership interest (including patents) in the same. K.T. Mueller is a clinical pharmacologist at Novartis Pharmaceuticals. S.L. Berger has received honoraria from the speakers bureaus of University of North Carolina and LMU Munich. S.L. Maude is a consultant for Novartis Pharmaceuticals and Kite Pharma and reports receiving commercial research support from Novartis Pharmaceuticals. S.A. Grupp is a SAB/consultant for Novartis, a consultant for CBMG and TCR2, a CAB for Cellectis, and a SAB for Adaptimmune and reports receiving commercial research grants from Novartis and Servier. C.H. June has ownership interest (including patents) in Tmunity and Novartis. S. Gill reports receiving a commercial research grant from Novartis. M. Ruella is an advisory board member for AbClon and a consultant for NanoString, reports receiving a commercial research grant from AbClon, has received honoraria from the speakers bureau of NanoString, and has ownership interest in patents in the CAR T field. No potential conflicts of interest were disclosed by the other authors.

Conception and design: N. Singh, S. Gill, M. Ruella

Development of methodology: N. Singh, O. Shestova, P. Ravikumar, K.E. Hayer, E.J. Orlando, K.T. Mueller, C.R. Good, O. Shalem, S. Gill, M. Ruella

Acquisition of data (provided animals, acquired and managed patients, provided facilities, etc.): N. Singh, Y.G. Lee, O. Shestova, P. Ravikumar, S.J. Hong, R. Pajarillo, E.J. Orlando, C.R. Good, N.V. Frey, S.L. Maude, S.A. Grupp, C.H. June, S. Gill, M. Ruella

Analysis and interpretation of data (e.g., statistical analysis, biostatistics, computational analysis): N. Singh, K.E. Hayer, X.M. Lu, E.J. Orlando, K.T. Mueller, C.R. Good, S.L. Berger, M.D. Weitzman, C.H. June

Writing, review, and/or revision of the manuscript: N. Singh, Y.G. Lee, O. Shestova, P. Ravikumar, K.E. Hayer, S.J. Hong, X.M. Lu, R. Pajarillo, S. Agarwal, S. Kuramitsu, E.J. Orlando, K.T. Mueller, C.R. Good, S.L. Berger, O. Shalem, M.D. Weitzman, N.V. Frey, S.L. Maude, S.A. Grupp, C.H. June, S. Gill, M. Ruella

Administrative, technical, or material support (i.e., reporting or organizing data, constructing databases): S. Agarwal, S. Kuramitsu, K.E. Hayer, X.M. Lu

The authors thank A. Hoshino, A. Green, and I. Maillard for valuable discussions and intellectual input, S. Lacey, F. Chen, and N. Kengle for technical assistance with cytokine quantification assays, and J. Schug for informative discussions about guide library sequencing. One of the chimeric antigen receptors used in this study was obtained under a material transfer agreement from Dr. Dario Campana, Dr. Chihaya Imai, and St. Jude Children's Research Hospital (33) and was subsequently modified by cloning into a lentiviral vector and expressed with a eukaryotic promoter (34). The research was supported by the Society of Immunotherapy for Cancer Holbrook Kohrt Immunotherapy Translational Fellowship (N. Singh), Breakthrough Bike Challenge Buz Cooper Scholarship (N. Singh), NCI K08CA194256 (S. Gill), American Society of Hematology Scholar Award, NCI 1K99CA212302, and R00CA212302 (M. Ruella), Mark Foundation ASPIRE award (M. Ruella), University of Pennsylvania-Novartis Alliance (S. Gill and C.H. June), and NCI 1P01CA214278 and R01CA226983 (C.H. June).

The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.

1.
Maude
SL
,
Laetsch
TW
,
Buechner
J
,
Rives
S
,
Boyer
M
,
Bittencourt
H
, et al
Tisagenlecleucel in children and young adults with B-cell lymphoblastic leukemia
.
N Engl J Med
2018
;
378
:
439
48
.
2.
Schuster
SJ
,
Bishop
MR
,
Tam
CS
,
Waller
EK
,
Borchmann
P
,
McGuirk
JP
, et al
Tisagenlecleucel in adult relapsed or refractory diffuse large B-cell lymphoma
.
N Engl J Med
2019
;
380
:
45
56
.
3.
Park
JH
,
Rivière
I
,
Gonen
M
,
Wang
X
,
Sénéchal
B
,
Curran
KJ
, et al
Long-term follow-up of CD19 CAR therapy in acute lymphoblastic leukemia
.
N Engl J Med
2018
;
378
:
449
59
.
4.
Turtle
CJ
,
Hanafi
L-A
,
Berger
C
,
Hudecek
M
,
Pender
B
,
Robinson
E
, et al
Immunotherapy of non-Hodgkin's lymphoma with a defined ratio of CD8+ and CD4+ CD19 specific chimeric antigen receptor-modified T cells
.
Sci Transl Med
2016
;
8
:
355ra116
.
5.
Lee
DW
,
Kochenderfer
JN
,
Stetler-Stevenson
M
,
Cui
YK
,
Delbrook
C
,
Feldman
SA
, et al
T cells expressing CD19 chimeric antigen receptors for acute lymphoblastic leukaemia in children and young adults: a phase 1 dose-escalation trial
.
Lancet North Am Ed
2015
;
385
:
517
28
.
6.
Orlando
EJ
,
Han
X
,
Tribouley
C
,
Wood
PA
,
Leary
RJ
,
Riester
M
, et al
Genetic mechanisms of target antigen loss in CAR19 therapy of acute lymphoblastic leukemia
.
Nat Med
2018
;
24
:
1504
6
.
7.
Sotillo
E
,
Barrett
DM
,
Black
KL
,
Bagashev
A
,
Oldridge
D
,
Wu
G
, et al
Convergence of acquired mutations and alternative splicing of CD19 enables resistance to CART-19 immunotherapy
.
Cancer Discov
2015
;
5
:
1282
95
.
8.
Ruella
M
,
Xu
J
,
Barrett
DM
,
Fraietta
JA
,
Reich
TJ
,
Ambrose
DE
, et al
Induction of resistance to chimeric antigen receptor T cell therapy by transduction of a single leukemic B cell
.
Nat Med
2018
;
24
:
1499
503
.
9.
Fraietta
JA
,
Lacey
SF
,
Orlando
EJ
,
Pruteanu-Malinici
I
,
Gohil
M
,
Lundh
S
, et al
Determinants of response and resistance to CD19 chimeric antigen receptor (CAR) T cell therapy of chronic lymphocytic leukemia
.
Nat Med
2018
;
24
:
563
71
.
10.
Finney
OC
,
Brakke
HM
,
Rawlings-Rhea
S
,
Hicks
R
,
Doolittle
D
,
Lopez
M
, et al
CD19 CAR T cell product and disease attributes predict leukemia remission durability
.
J Clin Invest
2019
;
129
:
2123
32
.
11.
Shalem
O
,
Sanjana
NE
,
Hartenian
E
,
Shi
X
,
Scott
DA
,
Mikkelsen
TS
, et al
Genome-scale CRISPR-Cas9 knockout screening in human cells
.
Science
2014
;
343
:
84
7
.
12.
Doench
JG
,
Fusi
N
,
Sullender
M
,
Hegde
M
,
Vaimberg
EW
,
Donovan
KF
, et al
Optimized sgRNA design to maximize activity and minimize off-target effects of CRISPR-Cas9
.
Nat Biotechnol
2016
;
34
:
184
91
.
13.
Messmer
MN
,
Snyder
AG
,
Oberst
A
. 
Comparing the effects of different cell death programs in tumor progression and immunotherapy
.
Cell Death Differ
2019
;
26
:
115
29
.
14.
Feucht
J
,
Sun
J
,
Eyquem
J
,
Ho
Y-J
,
Zhao
Z
,
Leibold
J
, et al
Calibration of CAR activation potential directs alternative T cell fates and therapeutic potency
.
Nat Med
2019
;
25
:
82
8
.
15.
Bengsch
B
,
Ohtani
T
,
Khan
O
,
Setty
M
,
Manne
S
,
O'Brien
S
, et al
Epigenomic-guided mass cytometry profiling reveals disease-specific features of exhausted CD8 T cells
.
Immunity
2018
;
48
:
1029
45
.
16.
Doering
TA
,
Crawford
A
,
Angelosanto
JM
,
Paley
MA
,
Ziegler
CG
,
Wherry
EJ
. 
Network analysis reveals centrally connected genes and pathways involved in CD8+ T cell exhaustion versus memory
.
Immunity
2012
;
37
:
1130
44
.
17.
Sen
DR
,
Kaminski
J
,
Barnitz
RA
,
Kurachi
M
,
Gerdemann
U
,
Yates
KB
, et al
The epigenetic landscape of T cell exhaustion
.
Science
2016
;
354
:
1165
9
.
18.
Wherry
EJ
,
Ha
S-J
,
Kaech
SM
,
Haining
WN
,
Sarkar
S
,
Kalia
V
, et al
Molecular signature of CD8+ T cell exhaustion during chronic viral infection
.
Immunity
2007
;
27
:
670
84
.
19.
Baitsch
L
,
Baumgaertner
P
,
Devêvre
E
,
Raghav
SK
,
Legat
A
,
Barba
L
, et al
Exhaustion of tumor-specific CD8(+) T cells in metastases from melanoma patients
.
J Clin Invest
2011
;
121
:
2350
60
.
20.
Schietinger
A
,
Philip
M
,
Krisnawan
VE
,
Chiu
EY
,
Delrow
JJ
,
Basom
RS
, et al
Tumor-specific T cell dysfunction is a dynamic antigen-driven differentiation program initiated early during tumorigenesis
.
Immunity
2016
;
45
:
389
401
.
21.
Philip
M
,
Fairchild
L
,
Sun
L
,
Horste
EL
,
Camara
S
,
Shakiba
M
, et al
Chromatin states define tumour-specific T cell dysfunction and reprogramming
.
Nature
2017
;
545
:
452
6
.
22.
Khan
O
,
Giles
JR
,
McDonald
S
,
Manne
S
,
Ngiow
SF
,
Patel
KP
, et al
TOX transcriptionally and epigenetically programs CD8(+) T cell exhaustion
.
Nature
2019
;
571
:
211
8
.
23.
Seo
H
,
Chen
J
,
González-Avalos
E
,
Samaniego-Castruita
D
,
Das
A
,
Wang
YH
, et al
TOX and TOX2 transcription factors cooperate with NR4A transcription factors to impose CD8(+) T cell exhaustion
.
Proc Natl Acad Sci U S A
2019
;
116
:
12410
5
.
24.
Scott
AC
,
Dündar
F
,
Zumbo
P
,
Chandran
SS
,
Klebanoff
CA
,
Shakiba
M
, et al
TOX is a critical regulator of tumour-specific T cell differentiation
.
Nature
2019
;
571
:
270
4
.
25.
Alfei
F
,
Kanev
K
,
Hofmann
M
,
Wu
M
,
Ghoneim
HE
,
Roelli
P
, et al
TOX reinforces the phenotype and longevity of exhausted T cells in chronic viral infection
.
Nature
2019
;
571
:
265
9
.
26.
Chen
J
,
López-Moyado
IF
,
Seo
H
,
Lio
C-WJ
,
Hempleman
LJ
,
Sekiya
T
, et al
NR4A transcription factors limit CAR T cell function in solid tumours
.
Nature
2019
;
567
:
530
4
.
27.
Pauken
KE
,
Sammons
MA
,
Odorizzi
PM
,
Manne
S
,
Godec
J
,
Khan
O
, et al
Epigenetic stability of exhausted T cells limits durability of reinvigoration by PD-1 blockade
.
Science
2016
;
354
:
1160
5
.
28.
Mueller
KT
,
Maude
SL
,
Porter
DL
,
Frey
N
,
Wood
P
,
Han
X
, et al
Cellular kinetics of CTL019 in relapsed/refractory B-cell acute lymphoblastic leukemia and chronic lymphocytic leukemia
.
Blood
2017
;
130
:
2317
25
.
29.
Mueller
KT
,
Waldron
E
,
Grupp
SA
,
Levine
JE
,
Laetsch
TW
,
Pulsipher
MA
, et al
Clinical pharmacology of tisagenlecleucel in B-cell acute lymphoblastic leukemia
.
Clin Cancer Res
2018
;
24
:
6175
84
.
30.
Frey
NV
,
Shaw
PA
,
Hexner
EO
,
Pequignot
E
,
Gill
S
,
Luger
SM
, et al
Optimizing chimeric antigen receptor T-cell therapy for adults with acute lymphoblastic leukemia
.
J Clin Oncol
2019
;
JCO1901892
.
doi:10.1200/JCO.19.01892
.
31.
Porter
DL
,
Hwang
W-T
,
Frey
NV
,
Lacey
SF
,
Shaw
PA
,
Loren
AW
, et al
Chimeric antigen receptor T cells persist and induce sustained remissions in relapsed refractory chronic lymphocytic leukemia
.
Sci Transl Med
2015
;
7
:
303ra139
.
32.
Vredevoogd
DW
,
Kuilman
T
,
Ligtenberg
MA
,
Boshuizen
J
,
Stecker
KE
,
de Bruijn
B
, et al
Augmenting immunotherapy impact by lowering tumor TNF cytotoxicity threshold
.
Cell
2019
;
178
:
585
99
.
33.
Imai
C
,
Mihara
K
,
Andreansky
M
,
Nicholson
IC
,
Pui
C-H
,
Geiger
TL
, et al
Chimeric receptors with 4-1BB signaling capacity provoke potent cytotoxicity against acute lymphoblastic leukemia
.
Leukemia
2004
;
18
:
676
84
.
34.
Milone
MC
,
Fish
JD
,
Carpenito
C
,
Carroll
RG
,
Binder
GK
,
Teachey
D
, et al
Chimeric receptors containing CD137 signal transduction domains mediate enhanced survival of T cells and increased antileukemic efficacy in vivo
.
Mol Ther
2009
;
17
:
1453
64
.
35.
Li
W
,
Köster
J
,
Xu
H
,
Chen
C-H
,
Xiao
T
,
Liu
JS
, et al
Quality control, modeling, and visualization of CRISPR screens with MAGeCK-VISPR
.
Genome Biol
2015
;
16
:
281
.
36.
Li
W
,
Xu
H
,
Xiao
T
,
Cong
L
,
Love
MI
,
Zhang
F
, et al
MAGeCK enables robust identification of essential genes from genome-scale CRISPR/Cas9 knockout screens
.
Genome Biol
2014
;
15
:
554
.
37.
Brinkman
EK
,
Chen
T
,
Amendola
M
,
van Steensel
B
. 
Easy quantitative assessment of genome editing by sequence trace decomposition
.
Nucleic Acids Res
2014
;
42
:
e168
.
38.
Picelli
S
,
Faridani
OR
,
Björklund
AK
,
Winberg
G
,
Sagasser
S
,
Sandberg
R
. 
Full-length RNA-seq from single cells using Smart-seq2
.
Nat Protoc
2014
;
9
:
171
81
.
39.
Buenrostro
JD
,
Giresi
PG
,
Zaba
LC
,
Chang
HY
,
Greenleaf
WJ
. 
Transposition of native chromatin for fast and sensitive epigenomic profiling of open chromatin, DNA-binding proteins and nucleosome position
.
Nat Methods
2013
;
10
:
1213
8
.
40.
Dobin
A
,
Davis
CA
,
Schlesinger
F
,
Drenkow
J
,
Zaleski
C
,
Jha
S
, et al
STAR: ultrafast universal RNA-seq aligner
.
Bioinformatics
2013
;
29
:
15
21
.
41.
Corces
MR
,
Trevino
AE
,
Hamilton
EG
,
Greenside
PG
,
Sinnott-Armstrong
NA
,
Vesuna
S
, et al
An improved ATAC-seq protocol reduces background and enables interrogation of frozen tissues
.
Nat Methods
2017
;
14
:
959
62
.
42.
Langmead
B
,
Trapnell
C
,
Pop
M
,
Salzberg
SL
. 
Ultrafast and memory-efficient alignment of short DNA sequences to the human genome
.
Genome Biol
2009
;
10
:
R25
.
43.
Zhang
Y
,
Liu
T
,
Meyer
CA
,
Eeckhoute
J
,
Johnson
DS
,
Bernstein
BE
, et al
Model-based analysis of ChIP-Seq (MACS)
.
Genome Biol
2008
;
9
:
R137
.
44.
Heinz
S
,
Benner
C
,
Spann
N
,
Bertolino
E
,
Lin
YC
,
Laslo
P
, et al
Simple combinations of lineage-determining transcription factors prime cis-regulatory elements required for macrophage and B cell identities
.
Mol Cell
2010
;
38
:
576
89
.
45.
Ramírez
F
,
Ryan
DP
,
Grüning
B
,
Bhardwaj
V
,
Kilpert
F
,
Richter
AS
, et al
deepTools2: a next generation web server for deep-sequencing data analysis
.
Nucleic Acids Res
2016
;
44
:
W160
5
.
46.
Anders
S
,
Pyl
PT
,
Huber
W
. 
HTSeq–a Python framework to work with high-throughput sequencing data
.
Bioinformatics
2015
;
31
:
166
9
.
47.
Robinson
MD
,
McCarthy
DJ
,
Smyth
GK
. 
edgeR: a Bioconductor package for differential expression analysis of digital gene expression data
.
Bioinformatics
2010
;
26
:
139
40
.
48.
Stuart
T
,
Butler
A
,
Hoffman
P
,
Hafemeister
C
,
Papalexi
E
,
Mauck
WM
, et al
Comprehensive integration of single-cell data
.
Cell
2019
;
177
:
1888
902
.