Radiomics, Tumor Volume, and Blood Biomarkers for Early Prediction of Pseudoprogression in Patients with Metastatic Melanoma Treated with Immune Checkpoint Inhibition

Immune checkpoint inhibitors (ICI) have revolutionized the treatment of patients with metastatic melanoma and are guideline-recommended treatment standards (1–3). However, more than 50% of patients do not respond to ICI (4). Predictive biomarkers are needed for better patient selection. Lactate dehydrogenase (LDH) is one of the only established melanoma biomarkers with prognostic value for overall survival (OS; ref. 5). S100 is associated with OS and response to ipilimumab, whereas other blood-markers and clinical markers yielded mixed results (6–10). PD-L1 expression is insufficient for patient selection, as both PD-L1–positive and negative patients benefit from ICI (3). Tumor mutational burden (TMB), T-cell clonality, circulating tumor DNA (ctDNA), immune gene signatures, as well as T-cell–inflamed gene expression profile are promising, but currently unavailable in routine practice (11–19). Most of these factors do not have an application in response assessment and require at least minimally invasive diagnostics, rendering them challenging for repeated analysis.

ICI response assessment is especially challenging, as the response evaluation criteria in solid tumors (RECIST 1.1) do not consider novel response patterns, such as pseudoprogression (PP; refs. 20, 21). In the setting of ICI, a lesion enlargement could be caused by either true progressive disease (TPD) because of tumor growth or an influx of immune-competent cells, representing an effective antitumor immune response or pseudoprogression (22). Early differentiation of PP and TPD is highly relevant in daily clinical decision-making. The immune-related response criteria (irRC; ref. 23) were the first to include these phenomena, followed by iRECIST (24). Both rely on confirmation follow-up imaging with potentially negative consequences. A treatment might be changed early because PP could be misinterpreted as TPD using only RECIST criteria or a switch to an effective treatment alternative could be delayed while waiting for confirmatory follow-up imaging. With newly proposed PECRIT (25) and PERCIMT (26) criteria, lesions increasing in size or the appearance of new lesions do not necessarily imply true progression, but may also be attributed to PP. However, these criteria have not been validated in larger cohorts.

Noninvasive imaging is performed repetitively during the treatment course for continuous response assessment. Manual image assessment is, however, characterized by suboptimal accuracy as well as intraobserver and interobserver variability (27, 28). Consequently, there is a strong rationale for quantitative medical imaging analysis (i.e., radiomics). Recently, Sun and colleagues demonstrated a correlation between CT radiomic signatures and the molecular CD8 cell expression in patients with different solid tumors treated with ICI, discriminating inflamed tumors from noninflamed tumors, which was associated with higher response rates at 3 and 6 months, as well as higher OS (29).

Our study aimed to identify novel imaging markers and blood markers for early differentiation of PP from TPD in patients with metastatic melanoma treated with anti–PD-1 antibodies. The predictive potential of early single-timepoint radiomics and lesion volume as well as multi-timepoint delta radiomics and delta volume was assessed on a lesion-individual level using positron emission tomography (PET)/CT imaging. In addition, the routine blood markers LDH and S100 were assessed for PP prediction on a patient-individual level.

This is a single institution analysis of a deeply characterized cohort of 190 patients with metastatic melanoma treated with either single checkpoint inhibition (anti–PD-1) or dual checkpoint inhibition (anti–PD-1/anti–CTLA-4) between 2013 and 2019. Written informed consent was obtained from all patients and the study was approved by the local ethics committee (Kantonale Ethikkommission Zürich, approval number 2019-01012) in accordance with “good clinical practice” (GCP) guidelines and the Declaration of Helsinki.

The following exclusion criteria were applied to allow for a standardized radiomics and outcome analysis: lack of follow-up/baseline imaging; patients with only contrast-enhanced CT imaging (as most patients were staged/followed with nonenhanced PET/CT imaging); patients with only brain metastases; and patients presenting with only very small metastases at baseline (all baseline lesions <0.5 cc). The last exclusion criterion is based on multiple factors. Statistical comparisons between individual voxels of a defined lesion (e.g., radiomics analysis) can only be performed if an adequate number of voxels is present in the defined volume. This is limited by the resolution of the underlying imaging technology and the voxel size itself. In addition, all imaging-based analysis methods do require a minimum lesion size to perform measurements with sufficient accuracy and reliability.

On a lesion-individual level, response was defined using RECIST 1.1 criteria, comparing lesion diameter at three different timepoints (TP): baseline (TP0), first follow-up at 3 months (TP1), and second follow-up at 6 months (TP2). PP was defined as a diameter increase by ≥20% at TP1, followed by a decrease to <20% at TP2 compared with TP0. TPD was defined as an increase by ≥20% on both TP1 and TP2 compared with TP0.

The distribution of PP and TPD lesions was analyzed on a patient-level and all patients were classified into: (i) patients with ≥1 PP lesion and no TPD lesions (PP only); (ii) patients with ≥1 TPD and no PP lesions (TPD only); (iii) patients presenting with both PP lesions and TPD lesions (mixed PP and TPD); and (iv) all other patients, who did not have a progressive lesion at TP1. For the lesion-level analysis and radiomics analysis, the appearance of new lesions was not taken into account in the patient stratification, as only lesions available at all three timepoints were considered for these analyses. OS was defined as the time from ICI treatment initiation to the date of death. Progression-free survival (PFS) and immune PFS (iPFS) were defined as the time from ICI treatment initiation to the date of first progression/appearance of new lesions (PFS, RECIST; ref. 20), or confirmed progression/appearance of new lesions (iPFS, iRECIST; ref. 24).

LDH and S100 levels were measured at TP0 and TP1 and their relative change between TP1 and TP0 was calculated.

All imaging was performed at a single institution using standardized imaging protocols. All subjects were injected with a body weight–dependent and/or BMI-adapted 2[18F]fluoro-2-deoxy-D-glucose (FDG) dose (2.0–3.5 MBq per kg). Scanning was performed on different scanners, partly with time-of-flight acquisition. PET image reconstructions used ordered subset expectation maximization together with point spread function modeling where available. The CT acquisition parameters were almost identical for all scanners and have been described previously in detail (30).

On the basis of exclusion criteria, 112 patients with CT imaging for all three timepoints were included. PET imaging for all timepoints was available for 90 of these patients (80%). All lesions were manually segmented by two experienced clinicians based on a common protocol and consistent quality control at all timepoints (Fig. 1). A validation was performed for about 10% of all lesions, visually comparing the independently contoured lesions, demonstrating reproducibility across all lesion locations.

CT and PET images were coregistered rigidly and CT-based contours were propagated to the PET images. Any spatial geographic mismatch was manually corrected via shifting the CT-based contours to the corresponding lesion location in PET images. All lesions were classified into liver, lung, bone, or soft tissue (including lymph nodes, cutaneous/subcutaneous, and muscular lesions) metastases.

The in-house developed radiomics software Z-Rad (31) written in Python programming language was used for preprocessing and extraction of radiomic features from medical images in agreement with the image biomarker standardization initiative (32). CT and PET images were resized to isotropic voxels of size 3.75 mm and 5 mm, respectively, corresponding to the lowest image resolution (image slice thickness) of the whole cohort. Three types of features, describing shape, intensity, and texture were extracted. No Hounsfield unit range limits were applied. As a preprocessing step before texture features extraction, image intensity values were discretized using a quantization step of 5 HU for CT and 0.25 SUV for PET. In total, 172 features per lesion were extracted for each imaging modality. The features were extracted for images taken at TP0 and TP1. The reproducibility of radiomic features was independently confirmed for a subset of lesions. Next, the relative (%) change in feature values between these two timepoints was calculated which gave rise to delta radiomic features.

To differentiate progressive lesions at TP1 into PP and TPD, a total of seven model classes were considered, including multimodality approaches:

• (i) Blood: models based only on blood markers (LDH and S100).

• (ii) Volume: models based only on the volume of individual metastases.

• (iii) Radiomics: (including volume-related features)—models based on radiomic features.

• (iv) Radiomics: (excluding volume-related features)—models based on radiomic features. Lesion volume and radiomic features correlated to lesion volume above Pearson r = 0.5 were excluded.

• (v) Blood and volume: models based on blood markers and lesion volume.

• (vi) Blood and radiomics: (excluding volume-related features)—models based on blood markers and on radiomic features. Lesion volume and radiomic features correlated to lesion volume above Pearson r = 0.5 were excluded.

• (vii) Blood and radiomics: (including volume-related features)—models based on blood markers and on radiomic features.

The model building procedure consisted of five steps. First, an unsupervised feature selection with the Pearson correlation coefficient was performed. In this procedure, pairwise correlation coefficients were calculated for all features and the correlated features were removed. Second, all features were scaled by subtracting the mean and dividing by the SD. Third, a univariate supervised feature selection was done. For each feature, an F-test was performed; only features significant after the false discovery rate correction with the Benjamini–Hochberg (33) procedure were kept. This step was followed by feature selection based on feature weights from a fitted model. For this reason, a logistic regression model regularized with elastic net was fit to the data. Features with the highest weights were selected. Finally, the classifier was fit to the data. This final model was based on logistic regression and regularized with L₂ penalty. The hyperparameters that were tuned are listed in Supplementary Table S1.

Our models were trained, optimized, and tested in a setting of a nested cross-validation. The inner loop, used for model tuning, was a three times repeated stratified 10-fold cross-validation. In total, 96 randomly generated hyperparameter samples were evaluated in model tuning with random search optimization (34). The outer loop, used for model testing, was a three times repeated stratified 5-fold cross-validation. The performance metric used was the area under the receiver operating characteristic curve (AUC).

The optimal cutoff was estimated for every model from the corresponding averaged ROC curve. The averaged ROC curves were calculated on the basis of 15 ROC curves (3 × 5) from the outer loop of the nested cross-validation. The optimal cutoff was defined as the sensitivity and specificity at which the Youden index (sensitivity + specificity − 1) was maximal (35). For the optimal cutoff, sensitivity, specificity, predictive values, and likelihood ratios with corresponding confidence intervals (CI) were estimated.

OS, PFS, and iPFS were assessed and compared among PP-only, TPD-only, mixed PP and TPD groups, and all patients that did not progress at TP1. The landmark analysis method (36) has been used to avoid the guarantee-time bias. The landmark was set at the determination timepoint of PP and true progression (TP2). To avoid excluding patients who had their TP2 follow-up examination slightly earlier than 6 months after the onset of treatment, we chose 5 months for the landmark. Family-wise error rate (FWER) was controlled at the 0.05 level with the Holm–Bonferroni method for each type of survival (37).

Tumor burden, approximated by total tumor volume of metastatic lesions in a patient, was estimated at TP0, TP1, and TP2. New lesions that appeared at TP1 or TP2 were not considered in the estimation. Statistical significance of the differences between the groups was estimated with Mann–Whitney U test.

For visualization, statistical analysis, model building, and model testing, the following open-source Python packages were used: HoloViews, Lifelines, Matplotlib, NumPy & SciPy, Pandas, and Scikit-learn.

On the basis of our exclusion criteria, 112 patients with a total of 716 metastases were included. A total of 2,061 metastases for both CT and PET imaging were individually segmented and analyzed. 645 (90%) of the 716 baseline lesions were either visible at all three timepoints or had a complete remission at TP1 or TP2. The remaining 71 lesions had to be excluded due to either surgical removal or lack of follow-up imaging.

The analysis of the lesion locations, as well as the patient characteristics analysis, was performed at baseline (TP0) and is therefore based on the 716 baseline metastases. The most frequent location was in soft tissue with 378 (52.8%) of all 716 baseline lesions. Other locations included 161 (22.5%) lung, 128 (17.9%) liver/spleen, 47 (6.6%) bone, and 2 (0.3%) myocardial lesions. Table 1 describes the patient characteristics for all groups.

The number of lesions per patient and lesion location were distributed equally between PP-only and TPD-only patients. Patients with mixed PP/TPD presented a higher mean number of 9.2 metastases per patient and had overall less soft tissue lesions compared with the other two groups, resembling a distribution closer to that of all patients combined. The percentage of patients receiving dual checkpoint inhibition and prior anti–CTLA-4 treatment was evenly distributed among all groups. The overall percentage of patients having new lesions at TP1 and TP2 was 41% and 26%, respectively. PP-only patients presented with comparable numbers of 55% and 22%, respectively. TPD-only patients and mixed PP and TPD patients had a higher percentage of new lesions with 75%/73% (TP1) and 62%/64% (TP2), respectively.

To classify a single lesion into either pseudoprogression or true progression, a total of 3 timepoints are necessary: baseline lesion diameter (TP0), diameter increase by ≥20% at 3 months (TP1), and confirmation of either true progressive disease, or pseudoprogression at 6 months (TP2). Therefore, the analysis of the individual lesion response per timepoint was limited to the 645 lesions that were available at all 3 timepoints.

Of these 645 lesions, 82 (13%) showed complete remission at the first follow-up at 3 months (TP1). 122 (19%) showed partial remission and 335 (52%) lesions were stable. Importantly, 106 (16%) lesions showed a progression by ≥20% at TP1, of which 30 lesions (4.7% of all lesions, 28.3% of progressive lesions) were defined as PP at TP2. 76 lesions (11.8% of all lesions, 71.7% of progressive lesions) remained progressive throughout TP1 and TP2 and were classified as TPD. The development of PP was not associated with metastasis location (P = 0.40) or treatment type (single vs. dual ICI, P = 0.12). Figure 2 illustrates the changes in response between TP1 and TP2.

A total of seven model classes for the prediction of pseudoprogression were considered, including multimodality approaches. Figure 3 shows the best performing ROC curves for all model modalities. The feature weights of the individual models, as well as partial dependence plots are provided in Supplementary Figs. S1 and S2. Table 2 provides an overview of all model metrics including AUC, sensitivity [true positive rate (TPR)], specificity [true negative rate (TNR)], positive/negative predictive value, and positive/negative likelihood ratio.

The first model was only based on the conventional blood markers LDH and S100 and achieved an AUC of 0.71 (sensitivity 0.69, specificity 0.67). Higher values of LDH and S100 indicated a low chance of PP.

As a second step, two models based solely on lesion volume (metastatic size) were generated. First, we created a model correlating PP with the lesion volume at 3 months (TP1). This prediction model achieved an AUC of 0.72 (sensitivity 0.76, specificity 0.60). Smaller lesions were more likely to be PPs. The second model was based on the relative (%) difference (delta volume) between the lesion volume at 3 months (TP1) compared with baseline (TP0) and achieved an AUC of 0.80 (sensitivity 0.81, specificity 0.67). A large increase in lesion volume at TP1 was associated with a reduced likelihood of PP.

Four types of radiomics-based prediction models were generated and were based on either single-timepoint radiomics (TP1) or multi-timepoint delta radiomics (relative difference between TP1 and TP0) with inclusion or exclusion of volume-related features. The best performing single-timepoint radiomics models achieved an AUC of 0.69 (CT, sensitivity 0.71, specificity 0.60) and 0.68 (PET, sensitivity 0.48, specificity 0.80), both performing worse compared with either the blood-based or volume-based models. The delta radiomics models performed better and achieved an AUC of 0.79 (including volume-related features; sensitivity 0.81, specificity 0.67) based on the features mc-volume and fractal dimension and an AUC of 0.78 (excluding volume-related features; sensitivity 0.89, specificity 0.53) based on the feature CT coarseness. Both, an increase in fractal dimension and a decrease in coarseness are indicating that significant changes from a uniform/homogeneous to a nonuniform/heterogeneous texture are more likely to be true progressions.

The combined blood- and volume-based prediction model, based on lesion volume, LDH, and S100 performed equally well compared with the volume-based model and better compared with the radiomics models, achieving an AUC of 0.79 (sensitivity 0.80, specificity 0.67).

A combined blood- and radiomics-based prediction model (including volume-related features) achieved an AUC of 0.78 (sensitivity 0.84, specificity 0.67). The best performing model was, however, a combination of blood- and radiomics-based prediction excluding volume-related features, which achieved an AUC of 0.82 (sensitivity 0.81, specificity 0.73). This prediction model was based on the LDH level at TP1 and the relative change of CT coarseness between TP1 and TP0. Larger values of LDH and a larger decrease in CT coarseness indicated a lower chance of PP.

The 30 lesions with PP were distributed across 20 patients with ≥1 PP lesion (median 1, range 1–5). The 76 lesions with TPD were distributed across a total of 27 patients. An overlap of 11 patients presented with mixed PP and TPD lesions (Fig. 4).

Median follow-up was 22 (range 14–32.5) months. For the whole cohort of 112 patients, median OS, PFS, and iPFS were “not reached,” 6, and 16 months, respectively. Two-year OS, PFS, and iPFS were 69%, 24%, and 42%, respectively. Supplementary Table S2 summarizes the outcome of the landmark analysis for all groups. Additional versions of Fig. 4 and Supplementary Table S2 without the use of the landmark analysis are provided in Supplementary Fig. S3 and Supplementary Table S3.

In the landmark analysis, PP-only patients had a significantly longer median OS of 30 versus 10 months (P = 0.002, FWER = 0.01) compared with TPD-only patients with a 2-year OS of 100% versus 15%, and a better, however not statistically significant, iPFS of “not reached” versus 7 months (P = 0.014, FWER = 0.058). Importantly, PP-only patients did not show a significant difference in OS (P = 0.934), PFS (P = 0.500), and iPFS (P = 0.557) compared with all other patients, who only had responding or stable lesions at TP1. Patients with mixed PP and TPD presented with a worse median OS of 25 versus 30 months compared with PP-only patients (P = 0.127), but had a longer median OS of 25 versus 10 months compared with TPD-only (P = 0.058, FWER = 0.174). These differences were, however, not statistically significant under FWER = 0.05.

As the difference between the TPD-only and mixed PP and TPD cohorts was quite striking, despite the fact that both presented with truly progressive lesions, we compared the overall tumor burden of both cohorts at all three timepoints. The mean baseline (TP0) tumor burden was higher in the mixed PP and TPD group (121 ± 177 cc) compared with the TPD-only group (86 ± 90 cc). This reversed for TP1, where the TPD-only group presented with a higher mean tumor burden of 280 ± 417 cc, compared with 133 ± 175 cc in the mixed PP and TPD group, which was confirmed at TP2 with 383 ± 542 cc (TPD-only) and 153 ± 188 cc (mixed PP and TPD). These differences were, however, not statistically significant with P values of P = 0.639, P = 0.786, and P = 0.415 for TP0, TP1, and TP2, respectively.

The incidence of PP has been described to be 4%–10% in most studies of patients with metastatic melanoma treated with ICI (3, 38–41). In our analysis, 8% of patients presented with PP only. Our study is, however, the first to include a lesion-individual analysis of PP, showing an overall rate of 4.7%. It is especially noteworthy that the 30 PP lesions in our analysis constitute almost one third (28.3%) of the 106 progressive lesions at TP1 and that the 9 PP-only patients constitute almost one third of patients with PD at TP1. This suggests that, although PP might be a rare phenomenon overall, it appears to be quite common among patients with progressive disease.

The improved outcome of PP-only patients with an impressive two-year OS of 100% was comparable with patients who did not have any progressive lesion at TP1 and is in line with an improved OS of patients with PP in other studies (15, 39). TPD-only patients had a significantly shorter OS, iPFS, and a drastically lower 2-year OS of only 14% indicating the need for an early change in treatment strategy. Patients with mixed PP and TPD showed a survival, which was worse compared with PP-only, but better compared with TPD-only patients, although both groups had lesions with true progression and presented with comparable rates of new lesions at TP1 and TP2. This suggests that a lesion-individual response assessment might provide advantages for patient stratification. In addition, we assessed if the overall tumor burden could be responsible for the difference between both groups. Interestingly the baseline tumor burden was actually higher in the mixed PP and TPD group compared with the TPD-only group, suggesting that overall tumor burden may not be the only critical factor.

An explanation might be that mixed PP and TPD patients could have had a slowly developing and/or resolving PP of lesions that were still classified as PD at 6 months. Some studies described continued benefit from immunotherapy, despite confirmed progression according to iRECIST criteria (40–43). It could therefore be possible that some lesions that were defined as confirmed progression in our analysis may respond much later during treatment, as these delayed tumor response dynamics have been described for both patients with melanoma and NSCLC by Nishino and colleagues (43, 44) and Hodi and colleagues (39).

In addition, Kong and colleagues described potentially prolonged residual disease following anti–PD-1 treatment with metabolically inactive lesions on PET/CT imaging (45). This indicates that even iRECIST/irRC have limitations and that it might be necessary to further increase confirmation follow-up time, include additional imaging information (e.g., radiomics), analyze response more deeply on a lesion-individual level, and finally to search for additional biomarkers.

The potential of radiomics to predict treatment response and outcome has been recognized for many tumor types and imaging modalities (46, 47). Tang and colleagues demonstrated that certain radiomic features (low CT intensity/high heterogeneity) predict infiltrating CD3⁺ lymphocytes and PD-L1 expression, associated with outcome in patients with non–small cell lung cancer (NSCLC; ref. 48). As mentioned earlier, Sun and colleagues could predict ICI responders by baseline CT radiomic signatures (29). Trebeschi and colleagues also used baseline CT radiomics for response prediction of patients with metastatic melanoma and NSCLC treated with ICI (49). While their model worked well in patients with NSCLC, it performed poorly in patients with melanoma, which the authors attributed to the large variety of first-line treatments prior to ICI. In our analysis, approximately half of the patients received first-line anti–PD-1 antibodies, while the other half was mainly pretreated with anti–CTLA-4 antibodies, representing a much more homogeneous patient cohort. In addition, our analysis included a 50% larger number of 716 analyzed lesions (112 patients) compared with 483 lesions (80 patients) and an additional third timepoint at 6 months.

A potential explanation why radiomic signatures are able to predict response or differentiate between PP and TPD are the previously described biological differences in lesions with either true progression or PP (22). In the case of TPD, a lesion could consist of mainly tumor cells alone. For PP, the picture could be much more heterogeneous on a cellular level, with a plethora of different involved immune cells, such as dendritic cells, cytotoxic CD8⁺ T cells, macrophages, natural killer cells, and others. This could potentially also lead to differences in radiomic features on CT and PET imaging. Trebeschi and colleagues reported that lesions with a greater morphologic heterogeneity were more likely to respond to immunotherapy (49). Our analysis shows that an increase in fractal dimension and a decrease in coarseness are more likely to be true progressions. Both indicate a change from a homogeneous to a heterogeneous texture. It might therefore be necessary to not only assess the heterogeneity of a lesion on a single timepoint but also the relative change of heterogeneity between different timepoints (delta radiomics) to adequately assess the also dynamic biological processes of immune cell infiltration into the tumor. Together, this could provide a basis for the detection of differences in an in-depth radiomics analysis, although these results will need to be confirmed in future prospective clinical trials. Interestingly, the solely volume-based models showed a slightly higher performance compared with the solely radiomics-based models in our analysis. As response assessment is already based on tumor volume, volume-based prediction models may therefore provide a simple and practical approach.

Surprisingly, biomarkers for ICI response assessment have not evolved substantially over the past decade. PD-L1 expression is one of the only clinically established biomarkers for predicting the response to ICI. However, some studies have also presented contradictory results, as both PD-L1–positive and -negative patients were shown to benefit from ICI (50, 51). Tumor-infiltrating lymphocytes (TIL), as well as neoantigens are other promising biomarkers and a higher number of TILs or neoantigens was associated with better outcomes in a variety of cancers (11, 51–53). Serum biomarkers provide multiple advantages over pathology-based biomarkers, as they can be obtained noninvasively and repetitively throughout the treatment course (54). ctDNA was used to discriminate PP from TPD in 2 patients with lung adenocarcinoma treated with ICI (55). Yoshimura and colleagues described a decrease in serum cytokeratin 19 fragment levels in the setting of PP under ICI (56). Lee and colleagues performed an analysis in 125 patients with metastatic melanoma treated with ICI, differentiating patients with PP from TPD via favorable ctDNA and unfavorable ctDNA profiles (15). 1-year OS was significantly higher in patients with favorable ctDNA (82%) compared to patients with unfavorable ctDNA (39%). Their numbers and percentages of patients with TPD were almost identical to our cohort, and also a third (31%) of their patients with TPD at TP1 had PP. TMB has also been assessed as a predictive biomarker, although two recent studies have shown that TMB was not associated with the efficacy of pembrolizumab in patients with NSCLC (57, 58). Other advanced biomarkers are being investigated including high-dimensional single-cell mass cytometry, which was shown to be able to predict the response to anti–PD-1 immunotherapy in patients with metastatic melanoma (59). In our analysis, the solely blood-based model was the worst performing of all model classes with an AUC of 0.71 but blood in conjunction with either tumor volume or radiomics led to the best performing models, suggesting that multimodality models may represent the best approach.

It has been shown that combined multimodality approaches for outcome prediction could perform better than single modality clinical, radiomics, or genomic data (60). We did not identify a study focusing on the noninvasive differentiation of PP and true progression via radiomics and routine blood markers. The radiomics models of Sun and colleagues and Trebeschi and colleagues were solely based on CT imaging without PET imaging and neither included a multi-timepoint delta radiomics analysis or a combined multimodality model.

The predictive performance of FDG-PET/CT and blood markers for general response assessment in melanoma has only been compared in the setting of chemotherapy, surgery, or radiotherapy. Aukema and colleagues reported a modest 50% positive predictive value of S100 for recurrent disease and recommended FDG-PET/CT for confirmation of recurrences (61). Wieder and colleagues concluded that the diagnostic accuracy and prognostic power of PET/CT is superior to S100 (62). Strobel and colleagues reported that S100 alone was not suitable for response assessment in 37% of patients, because S100 values were normal prior and after treatment, whereas PET/CT was suitable for all patients (63). Together, these results suggest that PET/CT imaging and the routine blood markers LDH and S100 could be combined for improved response assessment, but their use as a combined biomarker in immunotherapy has not been examined systematically.

We could show that the combination of delta radiomics and LDH was the best performing model, performing better than all other model classes. The multimodality model indicated that high levels of LDH and lesions with a large decrease in CT coarseness have a low chance of being a PP.

There are some limitations of our study, including the retrospective character of the analysis and the overall limited number of PP lesions and patients in the individual groups (PP-only, TPD-only, PP and TPD).

No external validation of our model has been performed. The most straightforward way to select and test the optimal model is to split the dataset into three parts: the training set, the validation set, and the test set. The training set is used to fit the models to the data, the validation set is used to select the optimal model, and the test set is used to estimate the expected predictive performance in an independent dataset, that is, generalization performance. This is a recommended approach in large datasets. However, in small datasets the way the dataset is split can significantly affect the performance scores in both validation and testing contributing to a high variance of the performance estimate. A common approach to reduce this variance is cross-validation. In cross-validation, the process of splitting the data to training and validation parts is repeated multiple times, reducing variance by averaging the performance scores (64). However, it has been shown by Stone (65) that cross-validation can lead to optimistically biased performance estimates when it is used for both model selection and predictive performance assessment (64). A solution to this problem was proposed by Varma and Simon (66) who showed that nesting two cross-validation loops, where the inner loop selects the optimal model and the outer loop performs the model assessment, mitigates this issue resulting in an estimation of the true generalization performance with very low bias. In small datasets, nested cross-validation is therefore superior to a training/validation/test split or a single cross-validation as it provides an almost unbiased estimate of true generalization performance with low variance (67).

However, small sample size may have prevented the multivariate models from reaching full capacity. A larger patient cohort could therefore lead to an improved performance of the multivariate models. Another limitation is the heterogenous PET/CT acquisition and reconstruction techniques, which is owing to the retrospective nature of our study. Nevertheless, despite these limitations, our study is by far the largest lesion-level analysis of patients with metastatic melanoma treated with ICI and the only one to include three separate timepoints, blood biomarkers, delta radiomics, and PET/CT imaging.

In conclusion, this study reports the first multimodality radiomics analysis using FDG-PET/CT imaging in the setting of immune checkpoint inhibition. It is also the first to include a multiple timepoint delta radiomics analysis in a multimodality prediction model in conjunction with the blood markers LDH and S100. Noninvasive PET/CT-based radiomics and LDH/S100 are promising biomarkers for early differentiation of PP from true progression at an early timepoint of 3 months, which might help reduce added toxicity or delayed treatment switch in patients with metastatic melanoma treated with ICIs.

M. W. Huellner reports receiving commercial research grants—to him and his institution—from GE Healthcare; reports receiving speakers bureau honoraria from GE Healthcare; and reports receiving other remuneration in the form of grants from Alfred and Annemarie von Sick and the Artificial Intelligence in Oncological Imaging Network. R. Dummer is a paid consultant or advisory board member for Novartis, Merck Sharp & Dohme, Bristol-Myers Squibb, Roche, Amgen, Takeda, Pierre Fabre, Sun Pharma, Sanofi, Catalym, Second Genome, Regeneron, and Alligator. No potential conflicts of interest were disclosed by the other authors.

Conception and design: L. Basler, S.A. Hogan, M. Bogowicz, S. Tanadini-Lang, R. Förster, R. Dummer, M. Guckenberger, M.P. Levesque

Development of methodology: L. Basler, M. Bogowicz, S. Tanadini-Lang, R. Förster, R. Dummer, M. Guckenberger, M.P. Levesque

Acquisition of data (provided animals, acquired and managed patients, provided facilities, etc.): L. Basler, S.A. Hogan, M. Pavic, R. Dummer, M. Guckenberger, M.P. Levesque

Analysis and interpretation of data (e.g., statistical analysis, biostatistics, computational analysis): L. Basler, H.S. Gabryś, S.A. Hogan, S. Tanadini-Lang, R. Dummer, M. Guckenberger, M.P. Levesque

Writing, review, and/or revision of the manuscript: L. Basler, H.S. Gabryś, M. Pavic, M. Bogowicz, D. Vuong, S. Tanadini-Lang, R. Förster, K. Kudura, M.W. Huellner, R. Dummer, M. Guckenberger, M.P. Levesque

Administrative, technical, or material support (i.e., reporting or organizing data, constructing databases): L. Basler, H.S. Gabryś, S.A. Hogan, M.W. Huellner, R. Dummer, M. Guckenberger, M.P. Levesque

Study supervision: R. Dummer, M. Guckenberger, M.P. Levesque

This work was supported by Cancer Research Center, Comprehensive Center Zurich, University Hospital Zurich (CRC_13, to L. Basler); SwissNational Fund (SNF 310030_170159, to H.S. Gabry); and European Training Network MELGEN funded consortium (no. 641458, to S.A. Hogan).

The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.