A Gene Expression Classifier from Whole Blood Distinguishes Benign from Malignant Lung Nodules Detected by Low-Dose CT

With cigarette smoking as the acknowledged root cause, lung cancer remains the primary source of cancer-related deaths worldwide. This is, in part, due to the lack of adequate early detection protocols, and, in part, because early symptoms are so subtle. The demonstration that lung cancer screening by low-dose CT (LDCT) reduces mortality among high-risk current and former smokers (>55 years, >30 pack years; refs 1–3) led to an overall increase in LDCT screening programs (4). Although LDCT does identify significantly smaller nodules than conventional X-rays, this ability comes with the challenge of distinguishing the small percentage of pulmonary nodules that are malignant from the majority of those detected that are benign (5). The National Lung Screening Trial (NLST) detected lung nodules ≥4 mm in diameter in 40% of the patients screened, with 96.4% being false positives over the 3 rounds of screening (6). To reduce this high false positive rate, the recent Lung-RADS classification (7) and new guidelines from the Fleischner group (8) set the detection of nodules ≥6 mm as the positive threshold. However, positive CT scans remain particularly problematic for that class of indeterminate pulmonary nodules (IPN), which range in size from 6 to 20 mm, for which the best course of clinical action is not well specified (6).

Our earlier studies demonstrated that rapidly purified (within 2 hours) peripheral blood mononuclear cells (PBMC) contain gene expression data that can distinguish benign from malignant lung nodules with high accuracy (9). This work established a new paradigm in nodule diagnosis by showing that even an early-stage cancer in the lung affects gene expression in PBMC that is predictive of malignancy. However, this approach was limited by the need to rapidly purify PBMCs from blood samples to maintain sample consistency and RNA integrity. This made it difficult to collect samples in environments where rapid isolation of PBMC was not possible, including most community clinics and physician offices. In addition, the microarrays, which were so useful for diagnostic development, are technically complicated and prone to variabilities associated with reagent batches and enzymatic processes, making them less amenable to clinical applications. The high quality of RNA required for microarray studies is also potentially problematic for studies with patient-derived samples (10). This retrospective/prospective study sought to determine whether accuracies similar to what we achieved in our PBMC studies (9) could be achieved with RNA from whole blood collected in RNA-stabilizing PAXgene tubes. PAXgene RNA is stabilized at the time of collection, immediately fixing the gene expression patterns. The RNA is stable at 15°C–25°C for 5 days and at −20 to −70°C for 8 years. This allows samples to be collected in any clinical setting where blood is drawn without the need for special equipment for storage or for cell purifications (11–13) and allows samples to be transferred to a central facility for testing, as routinely as with other blood tests. In addition, long-term storage with no loss of RNA integrity makes the system well suited for retrospective analyses. We also asked whether a PAXgene signature developed on Illumina microarrays could be transitioned to the NanoString nCounter platform already FDA-approved for the Prosigna Breast Cancer prognosis assay (14) and more recently used to develop a clinical-grade assay that predicts clinical response to PD-1 checkpoint blockade. This PD-1 assay is currently being evaluated in ongoing pembrolizumab clinical trials (15). Because the NanoString assays do not include any enzymatic reactions or amplification steps, the system avoids potential reagent batch effects and PCR biases while decreasing the opportunities for cross contamination by minimizing sample handling. Although we recognized that a gene expression profile from whole blood would be of a greater complexity and could potentially result in a reduction in important diagnostic signals, there was also the prospect that important additional cell types might contribute to the classifier performance.

We now report that the gene expression in whole blood, collected using PAXgene RNA stabilization tubes, can distinguish benign from malignant lung nodules detected by LDCT with high accuracy on independent validation and also report the successful transition of this pulmonary nodule classifier (PNC) from the microarray developmental platform to the NanoString nCounter platform.

The process of biomarker selection and validation across all studies is summarized in Fig. 1. A total of 821 samples from patients with malignant and benign pulmonary nodules were analyzed across three platforms: Illumina microarrays, the NanoString Pan Cancer Immune (PCI) panel, and finally a custom NanoString custom panel. Microarray data from 264 patient samples (Table 1; Supplementary Table S1) from 4 clinical sites was used for microarray model development. Estimations of performance were based on an independent validation set of 51 samples. In addition, 220 samples, including 201 of the 264 microarray samples, were analyzed on the NanoString PCI platform to select additional biomarkers to be included in the custom NanoString panel. Samples from a fifth collection site not included in the biomarker selection process were analyzed only on the custom NanoString platform. The final NanoString PNC (nPNC) was developed on the data generated from the custom NanoString panel using 583 training samples [included 215 samples used originally in the microarray training set, and 368 samples (70%) never used for the biomarker selection] and validated using a set of 158 independent samples never involved in probe selection. The characteristics of the samples used at the different steps of the classifier development are shown in Supplementary Table S2.

Samples were prospectively collected from incidental subjects with a positive LDCT from 5 clinical sites including Helen F. Graham Cancer Center (Newark, DE), The Hospital of the University of Pennsylvania (Philadelphia, PA), Roswell Park Comprehensive Cancer Center (Buffalo. NY), Temple University Hospital (Philadelphia, PA), and subjects from New York University Langone Medical Center (New York, NY). The NYU subjects included patients recruited as a part of an EDRN lung screening program at NYU. The study was Institutional Review Board–approved at each participating site and conducted according to the principles expressed in the Declaration of Helsinki. All participants signed an informed consent before being enrolled. The study population was primarily smokers and ex-smokers, >50 years of age with >20 pack-years of smoking history, and no previous cancer in the past 5 years (except for nonmelanoma skin cancer). Nodules were confirmed as malignant or benign by repeated imaging or by pathologic diagnosis through bronchoscopy, biopsy, and/or lung resection. In addition, >97% of benign nodules had 4 or more years of follow-up with the remainder having 2 or more years at the time of analysis. Samples associated with MNs were collected within 3 months of definitive diagnosis or prior to any invasive procedure including curative surgery. A small number of participants were found to be never smokers after they had been assayed. The effect on classifier performance of including these samples was assessed. In cases where multiple nodules were present, the diameter of the largest nodule was reported.

Each collection site was provided with a standard protocol for sample collection and storage as specified by Preanalytix [https://www.preanalytix.com/products/blood/rna for the PAXgene Blood RNA Tube (IVD)]. Samples were either stored on site and then bulk transferred overnight on dry ice, or they were transferred to Wistar by courier on the day of collection and stored at −70°C until processing. Total RNA was isolated using the PAXgene miRNA Kit (Qiagen), to capture miRNAs as well as mRNAs. Samples were quantitated with NanoDrop 1000 Spectrophotometer (Thermo Fisher Scientific) and assayed for RNA integrity on the Agilent 2100 BioAnalyzer. Average RNA yields were 3 μg/2.5 mL of blood and on average, RNA integrity numbers (RIN) are >8. Only samples with RIN >7.5 were used for the microarray studies. A constant amount (100 ng) of total RNA was amplified (aRNA) using the Illumina-approved RNA Amplification Kit (Epicenter) and hybridized to the Human-HT12 v4 human whole-genome bead arrays. Microarrays were processed in sets of 48 to minimize potential batch effects.

The NanoString hybridization was carried out for a constant 19 (within the recommended 12–25 hours) hours at 65°C. Posthybridization processing in the nCounter Prep Station used the standard settings. The cassette scanning parameter was set at high [555 field of view (FOV)]. Supplementary Fig. S1A shows the total normalized counts detected using 100 ng, 200 ng, and 300 ng of total RNA scanned at the low (280 FOV) and the high (555 FOV) settings. The 555 FOV setting significantly increases the overall signal. All assays were carried out using this setting. The standard sample size was 100 ng, an amount we expected to have in all samples. Supplementary Figure S1B shows the stability of the assay across multiple repeats of the Universal Human RNA control sample. Variations less than 5% were observed for the majority of the gene probes with 50 or more detected counts. Supplementary Figure S1C–S1F show that although most sample RIN numbers were above eight, even samples with RINs ≤3 met all four NanoString quality measures, supporting the platforms utility with degraded RNA samples. We found no significant impact on the overall expression profiles for the degraded RNA.

Microarray raw expression data were exported for analysis using Genome Studio software. The raw data was quantile normalized and log₂-scaled. Genes with average expression values ≥2× the background levels were used to develop the PNC using support vector machines and recursive feature elimination (SVM-RFE) and the 10-fold 10-resample cross-validation (see details below). The top ranked probes (Borda count) that most accurately distinguished malignant from benign nodules were selected as candidates for the NanoString custom panel. The SVM training set was also stratified into subset A, which contained smaller nodules and stage I and II cancers and subset B, which contained malignant and benign nodules that were balanced for lesion size (Supplementary Table S1). The additional analyses of sets A and B considered either nodule class alone (malignant or benign) or sample class plus collection site as the factors in a linear regression model for each observed gene expression. This resulted in six different regression models and two additional sets of genes were selected from these analyses for inclusion in the NanoString model based on the following parameters: (i) 59 genes with a minimal P value across the comparisons using P < 1 × 10⁻⁴ threshold and (ii) 76 genes with a maximum regression coefficient b > log₂(1.2) at P < 0.01. Housekeeping (HK) genes were selected from a candidate pool of well-expressed genes (>5× background) with coefficients of variation (CV) for the absolute and log₂-scaled expression less than 20% and 2.5%, respectively. Twenty candidate HK genes were ultimately selected: the 12 HK candidate genes with the least CV and 8 candidate genes that overlapped with existing NanoString HK probes.

Background correction was performed on NanoString PanCancer Immune Panel samples by subtracting the geometric mean of the counts of negative controls. The sample counts were normalized by scaling all the values by the ratios of geometric mean of sample controls to the overall geometric mean of control gene counts across all samples. This was done for both spike-in positive controls as well as for HK genes. The NanoString custom panel was quantile normalized and the NanoString Code Set batch differences were corrected using the ratios of expression of samples replicated between code sets, as per NanoString's recommendation. Z-scores were calculated from the final values of custom panel counts and used as inputs for SVM-RFE.

Supervised classification using a linear kernel SVMs with RFE (16) was used to analyze a z-score–transformed gene expression data set to develop the microarray classifier based on a training set that can distinguish malignant and benign patient classes. A balanced set of cases and controls was used in classifier development as SVM have been shown to require a balanced input for the development of the most accurate classifiers (17). The independent validation, which tests the validity of the classifier developed in the training on a completely new set of samples, is blinded to the identification of samples as either cases or controls. As described previously (9), we employed a 10-fold cross-validation approach with folds resampled 10 times (100 training–testing split models). For each split of the microarray data, the top 1,000 probes ranked by P value (two-tailed t test on 9-folds) were selected and linear kernel SVM was trained on 9-folds and tested on the remaining folds. Each RFE iteration eliminated 10% of the features with the least absolute model weights, in each round as described by Guyon and colleagues (16). A single feature elimination per SVM iteration was used for NanoString data analysis. The final average scores were calculated as follows. The final score for any sample in a training set is calculated as an average among the scores generated for that sample in all testing folds (10 such folds among all 100 splits). The final score for any independent sample in a validation set is calculated as an average among all 100 split models. Each sample is then assigned to a class using the final average scores and a score threshold determined from the training set (0 for unbiased accuracy, or at a fixed threshold corresponding to 90% sensitivity) and sensitivity, specificity and accuracy were calculated. Probes ranking across all 100 splits were combined according to the following procedure based on the Borda count method. In each ranked list n, each gene i was assigned a score: |{s_{i,n}} = {\frac{1}{{{r_{i,n}}}}$|⁠, where r_i,n is the rank of the gene i in list n. A final score FS for each gene i was calculated by taking the sum of the scores of gene i across all 100 lists: |F{S_i} = \sum _{n = 1}^{100} {s_{i,n}}$|⁠. The resulting final scores for each gene were then used to assign their ranking in the classifier. Final ranking of the probes was produced using all 741 available NanoString samples.

Optimal number of probes for microarray data was determined as the minimum number of probes that maintained an ROC-AUC within 1% of the ROC-AUC achieved by the SVM with top 1,000 gene probes. For the NanoString custom panel, the optimal number of genes was chosen by determining that point where the removal of additional genes/probes resulted in a decline in classification performance. The performance was assessed by determining the ROC-AUC after the removal of each gene using the moving average with a smoothing window size of 5. The probe number at which the ROC-AUC was at maximum was selected as the final optimal classifier.

Microarray data have been deposited to NCBI GEO database (https://www.ncbi.nlm.nih.gov/geo) under accession number GSE108375.

The demographic characteristics for the 315 patients used to develop and validate the microarray lung cancer signature are shown in Table 1. The samples used for model building were primarily early stage non–small cell lung cancer with stage I + II cancers comprising 84% of the training set population, and with 100% of the cancers in the independent validation set being stage I.

Gene expression from 264 samples (Table 1, Illumina Training set) was used to select the microarray gene signature that most accurately distinguished malignant from benign lung nodules using SVM-RFE (9, 16). The accuracy of classification was stable across a wide range of probe numbers (Supplementary Fig. S2A) and a panel of the 1,000 highest ranked probes achieved an ROC-AUC of 0.878. As the performance slowly decreased with elimination of lower ranked probes, we selected the smallest number of probes that maintained an ROC-AUC within 1% of the 0.878 achieved by the top 1,000 SVM gene probes. We identified 311 probes that returned an AUC of 0.866 [95% confidence interval (CI): 0.824–0.910] in the training set (sensitivity 77.9%, specificity 74.4%). The performance was well maintained on independent validation (Table 1, Illumina Validation), achieving an AUC of 0.847 (95% CI: 0.742–0.951) (sensitivity 72.7%, specificity 88.9%), similar to the performance of the training set (Supplementary Fig. S2B and S2C). This demonstrated the accuracy of prediction is similar to that of the 29 gene classifier reported from our purified PBMC study (9) and indicates that the presence of cancer in the lung can be also detected in the PAXgene-collected blood RNA with an equal and, in some cases, better performance. Importantly, the performance is maintained as the number of probes is reduced (Supplementary Fig. S2A and S2B), indicating a robust signature is maintained across different numbers of genes.

Having verified that mRNA expression from PAXgene samples can distinguish malignant from benign pulmonary nodules, we developed a strategy to transition the microarray-based PNC to the NanoString nCounter platform (14). Because it was difficult to know, a priori, how the microarray expression measurements would replicate on the NanoString platform, we designed the custom panel to contain enough redundancy to mitigate platform differences. We included the top ranked 300 biomarkers from the Illumina gene panel identified by the SVM-RFE analysis. We also included an additional set of 59 markers representing the most significantly differentially expressed probes at P < 1 × 10⁻⁴ and 79 probes that exhibited the largest fold change in expression between the malignant and benign groups while maintaining P < 0.01. A set of 20 HK genes with the most consistent expression on the microarrays (see Materials and Methods) was also added.

Because the mRNA samples to be processed on the NanoString platform did not undergo reverse transcription and PCR amplification, we were concerned that some of the microarray probes we selected might be expressed at levels too low for detection without amplification. To establish performance criteria, we analyzed 220 of the mRNA samples with the NanoString PCI panel (catalog no. XT-CSO-HIP1-12; 115 malignant, 105 benign). Although the actual probes were not identical to the Illumina probes, 755 out of 770 genes represented in the PCI panel were also represented on the microarray platform. This study allowed us to correlate the detectable levels of gene expression between the two platforms, providing an estimate of the expression levels that could be robustly detected on both platforms. The results suggested that the probes detected at 2× the background levels on microarrays were robustly detected on the NanoString platform.

The PCI data was also analyzed using SVM-RFE with 10-fold 10-resample cross-validation and although the PCI panel only demonstrated a ROC-AUC = 0.754 compared with the 0.866 achieved with the microarrays (Supplementary Fig. S2D), we selected 106 of the most discriminatory PCI probes for inclusion in our custom panel. An additional 55 probes for genes that were identified as being associated with outcome in our PBMC microarray studies (18, 19) were also added, bringing the number to 619 potential probes for the NanoString custom panel. Supplementary Table S3 summarizes the sources of the final list of the candidate biomarkers selected for the custom NanoString panel. The NanoString probes were then designed to target the same or closely located transcriptome regions as those targeted by the Illumina microarray probes whenever possible. Probes that met the NanoString quality control criteria were successfully designed for 559 of the 619 selected biomarkers (Supplementary File S1).

We first assessed how well the classification accuracies had been retained between the Illumina and NanoString platforms by reassaying 199 of the samples from the microarray training set. For the comparison of the classification accuracies, we only used the 276 microarray biomarkers that were successfully designed as NanoString probes. We observed a Spearman correlation ρ = 0.73 (P < 1 × 10⁻¹²) for the sample classification scores between the two platforms (Supplementary Fig. S3A). The ROC-AUC based on the 276 probes was 0.881 for the microarrays and 0.838 for the NanoString (Supplementary Fig. S3B), indicating that the platform transition was successful.

To carry out an unbiased assessment of the performance of the custom panel, we analyzed a total of 741 patient samples, including samples from a new fifth collection site. The final nPNC training set of 583 samples and the validation of 158 samples had balanced numbers of malignant and benign samples (Table 2) to provide the best conditions for selecting a classifier with good sensitivity as well as specificity (17).

The classification model using all 559 probes demonstrated an ROC-AUC of 0.833 (95% CI: 0.799–0.864) on training set and ROC-AUC of 0.826 (95% CI: 0.760–0.891) on the independent validation set (Fig. 2A). The training set performance remained stable during the recursive feature elimination process (Supplementary Fig. S4A). Incrementally decreasing sets of probes achieved similar ROC-AUCs (Fig. 2B–D; Supplementary Fig. S4B). Sensitivities, specificities, and positive and negative predictive values (PPV and NPV) are also similar in both training and validation sets (Table 3). The rank of each gene in the classifier based on the training set (Supplementary Fig. S4A) and the combined set (Supplementary Fig. S4C) is shown in Supplementary File S1. Although the 41 probe classifier achieved an AUC of 0.834 (95% CI: 0.800–0.865) for training and 0.825 (95% CI: 0.759–0.890) for the independent validation (Fig. 2C), even using as few as six probes maintained ROC-AUC above 0.8, though there is a slight drop in the validation set performance (Fig. 2D).

The optimal 41 gene signature had an unbiased sensitivity and specificity of 68.1% and 82.1%, respectively. It achieved a specificity of 51% at a sensitivity of 90% for both training and validation with NPV and PPV values of 0.99 and 0.0066, respectively, for the independent validation. The classifier detected cancers with 64% sensitivity for stage I and a sensitivity of 70% for the late-stage cancers. Probabilities of malignancy across a range of nPNC classification scores are shown in Fig. 2E. It should be noted that a small number of individuals with malignant or benign pulmonary nodules that had no history of smoking were included in the analysis. Removal of these subjects from the study did not change AUC validation performance for the 41 probe classifier (AUC difference of 0.001) or the full 559 probe classifier (AUC difference of 0.010). Adding age, sex, race, and smoking history as additional factors did not have an impact on the classification producing ROC-AUC of 0.837, as compared with 0.840, when only the gene expression was used.

Focusing on the 41 biomarker panel classifier, we compared the performance of the nPNC on all the samples in the difficult to assess 6–20 mm size range with the performance of three clinical algorithms, the Brock University (St. Catharines, Ontario, Canada) developed in a high-risk population (20, 21) and the Mayo Clinic (Rochester, MN; ref. 22) and Veteran's Affairs (VA; ref. 23) models developed using data from a more incidental nodule population. These algorithms assess the cancer risk of a pulmonary nodule based on a variety of demographic and pathologic parameters including nodule size and location (Fig. 3A). The nPNC outperforms all three clinical models on nodules in the 6–20 mm diameter range. Because nodule size is a well-accepted risk factor included in each of the clinical models, we also demonstrate an increased accuracy of the classification as compared with a classification using only size for the samples in the 6–20 mm range (Fig. 3B).

Because nodule size is an important risk factor and the definition of IPNs is not very well defined (6) with the changing guidelines, we examined the performance when comparing malignant and benign pulmonary nodules that were similar in size ranges. We calculated the performance of the 41 probe classifier across the various nodule size ranges using baseline positive thresholds of 4 mm from the NLST study (24), 6 mm and 8 mm as discussed in the recent reports from the Fleischner Society (8), and the Lung-RADS (7) as well as a baseline threshold of 10 mm. The ROC-AUCs and the specificities when the sensitivity is held at a performance of 90% were calculated for all possible ranges for the selected thresholds for training and independent validation sets (Fig. 4). Overall, training and validation set performance was highly conserved across all size ranges except where only a few validation set samples fall within a particular size range, as is evident in the smaller validation set. The 41 probe nPNC performs particularly well on independent validation with nodules in the difficult-to-diagnose 8–14 mm range, achieving a 64% specificity at 90% sensitivity, although the specificity drops to 48% in the larger combined dataset. Whether 4 mm or 6 mm is used as the threshold for a positive screen, our classifier demonstrates its utility in classifying IPNs by performing well across all the ranges with a ROC-AUC of 0.83 and 0.81, respectively, in the combined dataset and a threshold of 8 mm only reduces the AUC to 0.80. The specificities at 90% sensitivity are similarly stable and are calculated as 0.50, 0.46, and 0.48 for 4, 6, and 8 mm, respectively.

The overall benefits of lung cancer screening programs using LDCT are evident in the reported 20% increase in the patient survival. However, this success comes with the associated problem of to how to evaluate the large numbers of primarily benign IPN being detected and the concern for over-management (25). The recent Lung-RADS assessment has also suggested that the implementation of a positive screening threshold of 6–7 mm rather than the 4 mm used in the NLST study may be more appropriate in the management of lung cancer screening results (7) and that this change would reduce the magnitude of the IPN problem with a minimal effect on patient care (8). Even with new guidelines, the potential for over-management of the estimated 1.6 × 10⁶ lung nodules detected each year in the United States remains a significant challenge particularly for nodules ≥6 mm and less than ≤20 mm, where the risk of malignancy can range from approximately 8%–64% (26). The development of alternative noninvasive approaches to assess these IPNs in a clinically meaningful way is an important goal in pulmonary medicine.

Most noninvasive early detection approaches have depended on the identification of tumor-derived nucleic acids, antibodies, or proteins present in blood, plasma, serum, or sputum (27–29), with the caveat that these analytes are frequently rare in the presence of smaller early-stage cancers that are most amenable to curative surgery and which are now more being readily detected by LDCT. Additional studies that have avoided this issue have combined bronchoscopy with gene expression in normal airway epithelial cells or with gene expression associated with nasal brushings. This approach is based on the concept of “field cancerization” whereby the tumor induces gene expression alterations in the uninvolved respiratory tract that differs with the presence of a malignant or benign lung nodule. These approaches work well for nodules likely to be accessed by bronchoscopy (27, 30, 31), but are less effective with smaller IPNs that also represent a major management concern.

We previously showed that a malignant lesion in the lung can extend its influence beyond the pulmonary cancer field to the peripheral blood, as the gene expression in PBMC-derived RNA efficiently distinguishes malignant from benign lung nodules (9). The existence of this extra-pulmonary effect is supported by early reports from mouse models for lung cancer demonstrating that soluble factors produced by premalignant lesions in the lung influenced the expression of specific activation markers in the bone marrow macrophages and that this effect was enhanced with tumor progression (32–34). Although the PBMC studies provided an important proof of principle for extrapulmonary involvement, the need for the rapid purification of the PBMC samples to stabilize the transcriptional profiles was a hindrance to expanding to the collection sites outside of the academic environments and to the development of a robust clinical platform. We have now demonstrated that RNA from whole blood, easily collected in PAXgene RNA stabilization tubes, can also be mined for gene expression information that distinguishes malignant from benign lung nodules. This minimally invasive, 2.5-mL blood collection system allows samples to be collected not only at major medical centers, but wherever blood is routinely drawn. The RNA stability at room temperature for 5 days means that no special storage system is required to maintain sample integrity, thereby facilitating sample collection and subsequent transfer to a central testing facility even from remote locations. The quality of the RNA makes it amenable to analysis on a wide variety of platforms including a variety of sequencing platforms that require high-quality RNA.

We have tested the utility of the PAXgene collection system using samples collected at four academic pulmonary centers and from a community hospital. Samples were collected, stored, and transferred in bulk, or were collected daily and then transferred by courier to our test site for storage and final processing without any detectable effect on platform performance. We built our diagnostic model from global gene expression assayed on Illumina microarrays with cancers that were primarily stage I (69%) and II (17%) and nodules that ranged in size from the 4 mm threshold measurement of the NLST study to 20 mm, spanning the range of malignancy risk from <1% to 64% (8). Importantly, our PAXgene microarray classifier maintained a ROC-AUC of 0.847 (95% CI: 0.742–0.951) on independent validation almost identical to that of the training set used for classifier development. In many studies, validation set accuracy is somewhat diminished, suggesting the model used for classifier development was not large enough to adequately capture the potential subject diversity (9, 35, 36). Moving forward from the microarray developmental platform, we successfully transitioned the nodule classifier to the NanoString nCounter platform. The nCounter platform requires minimal sample handling, is technically simple, and has the ability to evaluate degraded and nondegraded RNA in the same assay. The FDA approval of the NanoString-based Prosigna Breast Cancer Prognostic Assay based on the PAM50 gene signature (14) and the more recent development of a NanoString-based immune signature that predicts the clinical response to PD1 blockade (37) further supports the clinical utility of this platform.

Although the preliminary gene panel for our NanoString-based classifier included 559 biomarkers, that number could be reduced to 41 probes while maintaining the ROC-AUC and, thus, suggesting the potential for simplifying the test platform. In assessing the contributions of the various probes represented in the 41 probes, 46% of the top ranked probes came from the SVM analysis, 29% from PCI panel, and with the fewest candidates being selected by P value. The myeloid-related genes linked to the survival in our PBMC studies (18) were not represented in the 41 probe classifier, but were well represented in the top 100 ranked probes whereas the NK-related probes were mostly in the lower half of the probe set, perhaps because the NK signal is significantly diluted in the PAXgene samples. As patient outcome data is accumulated, we will further assess the utility of the prognostic biomarkers included in our panel that were selected because of an association with recurrence/survival in our previous PBMC studies (18, 19, 38).

Although robust technical performance is important for any clinical platform, the resultant benefit to the patient is primary. The performance of our NanoString custom panel on the 741 samples analyzed on that platform has significant clinical implications with a potential to impact the use of invasive approaches for assessing some classes of difficult-to-diagnose IPN. Our study does not depend on the presence of circulating tumor cells, tumor proteins, or tumor RNA, whose presence is more consistent with more advanced cancers. In this study, we have primarily addressed that class of IPNs that are 6–20 mm in diameter, are of moderate to high risk (39) and frequently not easily accessible by either bronchoscopy or fine needle biopsies and early-stage cancers that are most amenable to surgical approaches. We have also assessed the performance with smaller nodules in the 4–6 mm range where the risk of malignancy is small but whose presence can remain of some concern. Importantly, our nPNC outperformed clinical algorithms presently used to stratify candidates with IPN for treatment or follow-up, including the Brock University (20, 21), Mayo Clinic (22), and VA (23) clinical models in the 6–20 mm range. Although these algorithms work well when applied to datasets that include mostly smaller benign nodules and larger cancers, performance is somewhat diminished when applied only to malignant and benign in the problematic size range. Although the size range of pulmonary nodules we have analyzed is important, there is still a significant difference in the median size between the malignant and the benign in our study. It will be important to address how well biomarkers and clinical algorithms function when benign and malignant nodules are more closely matched in size and where clinical algorithms are likely to perform poorly. We attempted to test this type of comparison as shown in Fig. 4. Although the overall AUCs, sensitivities, and specificities are well conserved whether we use 4, 6, or 8 mm as a positive threshold, as the comparisons get more granular, some comparisons are significantly more accurate than others and this is particularly evident in the validation study where samples number are smaller. We achieved a specificity of 64% at 90% sensitivity for benign and malignant in the 8–14 mm size range dropping to 40% in the 6–14 mm range.

Nodule size is the primary consideration in how IPNs are treated (40). Although this study has interrogated a large number of patient samples and demonstrated a potential utility, a further assessment with a larger number of samples where malignant and benign nodules are more closely related by size will extend that utility as this is the scenario where size is no longer informative. In moving forward, it will be important to more completely address the issue by the comparison of benign and malignant nodules of similar sizes across the range of nodule sizes that remain problematic. The highly simplified and proven method for the acquisition of large numbers of samples of consistent quality from a variety of locations will facilitate this process. Expanded studies will also allow us to address the biological basis for the differences we are detecting between the patient classes and to assess whether those differences may have therapeutic implications.

A.V. Kossenkov has ownership interests (including stock, patients, etc) in patent applications WST155P1 and WST117P. G. Criner has ownership interests (including stock, patients, etc) in HGE Health Care Solutions and is a consultant/advisory board member for AstraZeneca, Boehringer Ingelheim, AVISA, Lungpacer, GlaxoSmithKline, Philips Respironics, NAMDRC, EOLO Medical Inc, Helios Medical, Novartis, Olympus, Spiration, Holaria Inc., Mereo BioPharma, Third Pole, PneumRx, BTG plc, Pearl Therapeutics, Pearl Therapeutics, and Broncus Medical. A. Vachani reports receiving a commercial research grant from Oncocyte, Inc., MagArray, Johnson & Johnson, and Broncus Medical and is a consultant/advisory board member for Oncocyte, Inc. L.C. Showe is a adjunct professor at the University of Pennsylvania and reports receiving a commercial research grant from SRA from OncoCyte and has ownership interests (including stock, patients, etc) as an inventor on pending unpublished provisional patent application(s) assigned to The Wistar Institute that relates to compositions and methods of using a new lung nodule classifier. No potential conflicts of interest were disclosed by the other authors.

Conception and design: A.V. Kossenkov, H. Pass, A. Vachani, B. Nam, W.N. Rom, M.K. Showe, L.C. Showe

Development of methodology: A.V. Kossenkov, R. Qureshi, C. Chang, T. Kumar, L.C. Showe

Acquisition of data (provided animals, acquired and managed patients, provided facilities, etc.): R.S. Majumdar, C. Chang, T. Kumar, E. Kannisto, G. Criner, J.-C.J. Tsay, H. Pass, S. Yendamuri, A. Vachani, T. Bauer, W.N. Rom, L.C. Showe

Analysis and interpretation of data (e.g., statistical analysis, biostatistics, computational analysis): A.V. Kossenkov, R. Qureshi, N.B. Dawany, J. Wickramasinghe, Q. Liu, W.-H. Horng, H. Pass, S. Yendamuri, M.K. Showe, L.C. Showe

Writing, review, and/or revision of the manuscript: A.V. Kossenkov, R. Qureshi, N.B. Dawany, J. Wickramasinghe, Q. Liu, G. Criner, J.-C.J. Tsay, H. Pass, S. Yendamuri, A. Vachani, T. Bauer, W.N. Rom, M.K. Showe, L.C. Showe

Administrative, technical, or material support (i.e., reporting or organizing data, constructing databases): A.V. Kossenkov, R.S. Majumdar, C. Chang, S. Widura, T. Kumar, H. Pass, L.C. Showe

Study supervision: C. Chang, L.C. Showe

We wish to thank the Wistar Genomics and Bioinformatics facilities for support, Dr. Rachel Locke for editorial assistance, and everyone responsible for recruitment and sample collection. This study was supported by the PA Department of Health grant #4100059200 (Diagnostic Markers for Early-stage Lung Cancer in PAXgene Blood Samples), NCI U01 CA200495-02, and OncoCyte Sponsored Research Agreement to L.C. Showe. NCI U01 CA111295 (to H. Pass), NCI U01 CA086137 (to W.N. Rom), R21 CA156087-02 (to A. Vachani, S. Yendamuri, and L.C. Showe), R50 CA211199-01 (to A.V. Kossenkov), Support for Shared Resources utilized in this study was provided by Cancer Center Support Grant (CCSG) P30 CA010815.

The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.