Data Analysis Strategies in Medical Imaging

Large-scale radiographic imaging of diseased tissue offers an incredibly rich data resource for scientific and medical discovery. Because imaging data are collected during routine clinical practice, large datasets are potentially readily available for medical research. Buoyed by advancements in artificial intelligence (AI), statistical methodology, and image processing capabilities, the number of publications related to big data analysis of radiographic datasets is growing at an exponential pace (1, 2). Indeed, the automated quantification of radiographic characteristics of tissues can be helpful in the detection, characterization, and monitoring of diseases. This process, referred to as “radiomics” (3–6), uses either a set of predefined engineered features (7) that describe radiographic aspects of shape, intensity, and texture or alternatively, features that can be automatically "deep learned" directly from example images (8, 9). Early success of radiomics for assisting clinical decisions related to the diagnosis and risk stratification of different cancers (4, 10–14) has spurred rapid expansion in this field and has opened new avenues of investigating the clinical utility of medical imaging in radiology (15).

The analysis of radiologic data presents many challenges. Although some of these challenges are specific to the imaging field, many are generic and have already been addressed in quantitative fields such as genomics and biostatistics. Microarray data analysis field, for example, although now a mature field, initially struggled with many obstacles related to data normalization (16), batch effects (17, 18), replicability (19), and the use of gene expression profiles for disease subtyping (20) and classification (21). In its current youthful state, data analysis in radiology faces similar challenges (3, 5, 15, 22). However, many researchers in radiology are unaware of some commonly observed and avoidable data analysis pitfalls. It is our contention that the quality of radiology studies could be greatly improved by following a simple set of data science “best practices” specific to the radiology field. These start with basic experimental design principles, including data normalization and standardization protocols, and expand to data analysis and validation strategies with appropriate reporting. In this review, we provide examples of typical data analysis pitfalls observed in radiomics studies and suggest strategies for avoiding them.

Although data analysis strategies may vary considerably between studies in medical imaging, certain common strategies related to study design, analysis, and reporting (see Fig. 1 and Table 1) could enhance the validity and generalizability of the study. Here, we discuss these strategies in detail.

Experimental design should be defined from the outset of a study. An experienced statistician should be consulted from the beginning of the study. By anticipating big picture challenges, the research question should be defined, required resources and data should be identified and curated, and ultimately, high-level decisions related to analysis strategies should be made (see Fig. 1 and Table 1).

To begin with, the overall scope of the research and the potential impact in the field should be assessed by reviewing relevant scientific literature and consulting domain experts. Subsequently, feasibility of the study should be assessed, and a tentative timeline should be established. If possible, effect sizes should be anticipated to assess whether additional data and resources would be required. Finally, the research questions to be investigated and the corresponding resource requirements should be defined.

During this step, imaging and clinical data should be gathered and curated, and access to computational resources and methods should be established. Careful curation and annotation of the imaging and clinical data are important for quality control of the data and analysis. Validated open-source software tools should be preferred over their proprietary commercial counterparts to increase reproducibility and interpretability. Moreover, if multiple analysts are working on the analysis, standardized computational platforms and software versions should be used to increase consistency and reproducibility. Researchers should strive for balance to assure that different phenotypic groups are represented appropriately in the training and validation datasets. Here, balance refers to the balanced proportion of different classes of outcome or target variables. In cases where class imbalance is inevitable, appropriate strategies like augmentation or bootstrapping can be utilized.

Once the scope is defined and the required data and resources are curated, important decisions related to analysis strategies should be made. Different computational approaches should be reviewed to identify suitable methods for the analysis of radiographic data. For example, feature quantification strategies (i.e., engineered features or deep learning), image preprocessing methods, data normalization approaches, dimensionality reduction and feature selection methods, as well as different supervised and unsupervised modeling approaches, should all be explored for the exploratory empirical analysis. Another important aspect in this step is to define and lock (fix) the training and validation cohorts. Within the analysis stage, the training cohort can be used for the exploratory empirical analysis to further select and lock the computational methods. However, the validation data should be kept locked and untouched until all methods are fixed in the analysis phase (see Fig. 1). Locked validation data will prevent information leakage from training to validation and will limit the possibility of overfitting.

A commonly observed pitfall (see Table 2) that can be avoided at the design stage is when the number of samples is realistically too low to attain significance or to train a model. Insufficient training data could diminish the learning capability of a model, whereas insufficient validation data hinder the true evaluation of the underlying hypotheses. In this case, it might be best to gather more data samples, investigate only conservative questions, or postpone the study until a later time. Although seemingly undesirable, calling off a study at the design stage is preferable to investing time and effort in a study that is too premature or underpowered to achieve statistical significance. Such a premature analysis can lead to overfitting as researchers scramble to find a combination of analysis choices that give “publishable” numbers.

In the second phase, the analysis is initiated with the goal of quantifying radiologic characteristics and ultimately, establishing and validating imaging biomarkers (see Fig. 2). When analyzing radiologic cohorts, some preprocessing is required to reduce technical variability across images. Different sources of batch variability should be investigated, including difference in scanning instrumentations, signal drifts, and other calibration-related issues as well as longitudinal effects and changes in imaging protocols. Statistical methods to correct for such batch effects should be applied. These steps ensure true assessment and validation of the underlying hypotheses.

Examples of image preprocessing prior to feature quantification include resampling of dimensions to isometric voxels to homogenize image resolutions. Images with isometric voxel dimensions can be either reconstructed from the raw Digital Imaging and Communications in Medicine (DICOM) data or interpolated from the image data. Moreover, some modalities, such as MRI, require normalization of image intensity values. Sources of variability and image normalization steps are critical correction measures for imaging-related batch effects.

Within radiology, AI methods can perform comprehensive quantifications of tissue characteristics (see Fig. 2). These methods can convert 3D radiologic images into high-dimensional phenotype descriptors. This approach, called “radiomics,” uses engineered features and/or deep learning. Here, we will describe these two approaches independently.

Engineered features are predefined and hard-coded algorithms designed to quantify specific radiographic characteristics of diseased tissues (7). Domain-specific expertise can be used to identify important phenotypic characteristics of diseased tissues, which could then be mathematically defined and programmatically extracted in an automated manner. For example, nodular ground-glass opacity is considered as one of the vital factors for the management of pulmonary nodules (23, 24). This domain-specific knowledge has been used to define different statistical features (mean, median, variance, range, etc.) from the intensity distributions of the nodules (7). It has been shown that the ground-glass nodules have significantly lower median intensity values than partly solid or solid nodules (25). Similarly, it has been demonstrated that different radiomic features based on the shape, texture, and regional heterogeneity of the diseased tissues are associated with several clinical endpoints in oncology (4, 10, 11, 13, 25–32) and in other domains such as cardiology (14).

Additional data normalization steps like feature transformation and standardization are needed for engineered features due to the intrinsic differences of range, scale, and statistical distributions of these features. Untransformed features may have high levels of skewness, which tend to result in artificially low P values in downstream statistical tests for low sample size cohorts. Neglecting feature standardization may lead to individual features being over- or underrepresented in statistical models and eventually introduce bias into the analysis. Methods like standardization and logarithmic transformations (33) can transform features into zero-centered distributions having identical variances and symmetric distributions.

Deep learning enables the extraction of multiple feature levels from data directly without explicit definition (8, 34, 35). It provides a higher level of feature abstraction, thus potentially providing better prediction performance. Although deep learning has recently achieved impressive successes in practice, a strong theoretical backing is yet to be established (36). The lack of rules of thumb makes choosing an appropriate deep learning network architecture challenging. The problem at hand: classification, detection, segmentation, registration, or reconstruction, in addition to the type and size of data, all hint at the appropriate architecture to utilize. Starting from published architectures that have proven successful in their respective tasks is common practice. Convolutional neural networks are the most prevalent deep learning architectures in medical imaging today (37). Transfer learning, or using pretrained networks, is often an attractive option when dealing with scarce data (38). Data normalization is an essential data preprocessing step for deep learning. It ensures increased numerical stability and quicker and stable convergence. This could be achieved through sample-wise, feature-wise, or principal components analysis (PCA) whitening normalization depending on the data type. It is important to note, however, that even with inputs being normalized, their distributions are highly susceptible to change as they propagate through the network where parameters are constantly optimized during training—a problem referred to as "internal covariance shift"—and can be mitigated using batch normalization (39) or layer normalization (40). Normalization is hence made to be an integral part of the network architecture, as opposed to merely being a preprocessing step, and is performed on inputs to each layer based on the distribution of the given batch of cases.

A few important points should be considered before choosing the quantification method. Although deep learning is able to perform quantifications in an automated manner, it generally requires fairly large datasets. Often with quantitative imaging, it could be challenging to gather and curate large cohorts of patients with similar clinical, imaging, and demographic characteristics. Engineered features are less sensitive to the cohort size but need to be defined manually by experts. Moreover, they also require the segmentation of the diseased tissue. Recent efforts utilizing transfer learning have broadened the applicability of deep learning approaches to smaller cohorts (38, 41). Hence, these points should be carefully considered prior to choosing a particular quantification method.

Quantitative analysis of medical image data involves mining large number of imaging features, with the goal of identifying highly predictive/prognostic biomarkers. Here, we discuss these concepts for engineered features and deep learning methods separately.

A large number of engineered features are extracted from the medical image data, which might be highly redundant. The use of appropriate feature selection/reduction strategies can minimize the feature redundancies and mitigate the “curse of dimensionality” (42). Many unsupervised and supervised feature selection methods have been described in the literature (10). Unsupervised methods based on PCA analysis, independent component analysis, or correlation can reduce the feature space without using a target outcome. Alternatively, supervised methods can be used to select features that are relevant to the particular target outcome (43, 44). These supervised methods can be separated into three categories: wrapper, embedded, and filter methods. Computationally expensive wrapper and embedded methods use stricter model structure assumptions and are likely to have low generalizability, whereas model-independent filter methods are relatively efficient and provide better generalizability (43, 44).

Predictive and prognostic models with high accuracy, reliability, and efficiency are vital factors driving the success of quantitative imaging approaches (10). Myriad supervised and unsupervised methods exist in the machine learning literature derived using different statistical assumptions (42, 45, 46). Several investigations have compared these methods for quantitative image analyses (10, 47–49,). These studies have demonstrated that the variability of prediction scores is highly influenced by method choice (10, 42). Therefore, the choice of feature selection and machine learning modeling methods should be cautiously made.

One of the most commonly observed pitfalls during the biomarker identification stage is overfitting. Researchers have a tendency to exhaustively search through different modeling methods and parameter configurations to obtain high and publishable performance, which can result in poor biomarker generalizability. To avoid this, use of standard settings for hyperparameters (50) or use of locked and independent validation data (51) is recommended.

In deep learning, dimensionality reduction and classification are performed alongside feature extraction in an integrated manner (see Fig. 2). However, the quality and output of these cascaded layers depend on different hyperparameters such as the number of layers and feature maps, layer arrangement and architecture, size of the receptor field, etc. Many different network architectures have been described using different sets of hyperparameters (52, 53). Prediction performance could be influenced by the choice of these hyperparameters and architectures. As in the case of engineered features, optimizing the hyperparameters by exhaustively searching the parameter space is a common pitfall in deep learning as well. The deep learning architecture should be selected on the basis of the underlying scope and application of the research, the statistical properties of data in hand, and the effective data size.

Overfitting represents a major challenge in deep learning and can drastically affect a network's ability to generalize to unseen data. Often, deep learning methods are treated as a black box, and not enough attention is given to understanding the actual methods and technical concepts. This is particularly an issue when working with limited training data. Deeper and more complex networks trained on limited data could induce overfitting. The use of shallower networks could avoid overfitting but may result in insufficient learning, also known as underfitting. One solution could be data augmentation, where the training data are expanded by applying label-preserving image transformations (54) like cropping, reflections, and rotations. Dropout or other regularization methods could also be used to reduce overfitting (55, 56). Dropout makes the network less sensitive to individual weights and hence increases the robustness of a network. Other regularization methods allow penalizing large parametric weights to make the network more robust and generalizable. During training, the network performance should be evaluated and monitored using cross-validation. A cross-validation–based early-stopping (57) approach can also be utilized to avoid overfitting. These important factors should be considered during the biomarker identification stage.

After identifying biomarkers, a document should be created listing all the cross-validation and analysis steps taken during the exploratory analysis of training data, the final hypotheses to be validated, the training data used and corresponding inclusion criteria, as well as a list of identified and locked computational approaches. Furthermore, it should also be declared that the validation data were not used during the exploratory analysis or biomarker identification stage. The same document can later be used to incorporate validation results and reported alongside the study findings.

As described in the previous sections, avoiding overfitting and data leakage is essential when working with machine learning and deep learning models. It should be ensured that locked validation cohorts remain blinded during the training and hyperparameter tuning. Only after fixing the models, network architectures, computational methods, and corresponding hyperparameters should the validation be carried out on the locked validation cohorts. These steps ensure the true evaluation of the underlying hypotheses.

Appropriate performance metrics should be used, especially when dealing with highly unbalanced classes. For example, the accuracy of a classifier is potentially sensitive to the event ratio (class distribution) of a population, resulting in deceivingly overoptimistic results. Accurate evaluation can be achieved by reporting multiple performance metrics like AUC, sensitivity, specificity, positive predictive value, negative predictive value, etc. Moreover, robustness of the biomarkers with respect to data perturbation is also a vital aspect of biomarker validation (58). An essential and often overlooked step before choosing candidate biomarkers is performing multiple testing corrections. When testing hundreds of features, it is expected that some associations with clinical outcomes can be found entirely by chance. To avoid this, correction methods like Bonferroni (59) and Benjamini and Hochberg (60) should be applied.

To gauge the true clinical impact, it is also essential to statistically compare the developed biomarkers to standard clinical markers during validation. Furthermore, the additive increase in performance should also be evaluated by combining the developed biomarkers and standard clinical markers in a computational model. Moreover, emphasis should be given to multiple external validation sets, which can ensure the robustness and generalizability of a biomarker. It is our view that studies demonstrating biomarker robustness through reproducibility should be looked upon favorably.

In the final phase, a detailed report should be made about the data, analysis methods, and results. Furthermore, enough emphasis should be given to sharing the analysis protocols, documented code, original as well as processed data, along with detailed descriptions.

Several technical and privacy issues should be considered while sharing biomedical imaging data. Patient confidentiality must be protected throughout the entire process. A system similar to that of the database of Genotypes and Phenotypes (dbGaP) should be adopted, where raw imaging data can be archived and distributed in a manner that enables data sharing without compromising patient confidentiality. The publicly available online medical imaging repository The Cancer Imaging Archive (http://www.cancerimagingarchive.net/) is a great place for sharing and archiving medical image data, as it uses a standards-based approach to de-identification of DICOM images to ensure that images are free of protected health information. For analysis, repositories like GitHub (http://github.com) allow for relative ease in sharing code. Furthermore, computational tools such as Jupyter Notebooks and Sweave can make “executable documents,” which should be included as supplements to scientific articles to promote reproducibility and generalizability. Experiments housed in software environment containers with locked versioning such as Docker (www.docker.com) allow for swift reproducibility and portability.

As building imaging biomarkers from radiologic data involves multiple analytic steps, sharing comprehensive descriptions of statistical methods is essential for reproducibility. In the shared document, we recommend reporting the number of samples and features tested, feature definitions, statistical tests and algorithms used during the analysis, list of model and hyperparameters used, details regarding optimization methods, nominal and corrected P values, and effect size assessments. Once imaging data have gone through the quantification steps outlined above, the result is normally a matrix of features versus samples. As long as sample labels are properly de-identified, this feature matrix can usually be shared without any risk to patient confidentiality. Ideally, all code required to reproduce study findings from the feature matrix should also be shared, with the exception of sections that require the use of protected clinical metadata.

An effective strategy often adopted for data sharing is to group data, analysis reports, and runnable code blocks into a single object such as R data packages with vignettes. For studies that use deep learning approaches, the type of network architecture and hyperparameters should be shared. In addition, sharing entire networks with trained weights can be highly beneficial for transfer-learning efforts. With an increasing number of deep learning libraries, sharing trained networks in universal formats such as ONNX (www.onnx.ai) can facilitate straightforward cross-platform compatibility. It is our experience that sharing methods and code in this manner also has the natural effect of raising the overall quality of data analyses in addition to ensuring reproducibility.

Data science and big data are rapidly becoming major components of health care applications in both the industrial and academic settings. However, performing accurate and meaningful data analyses is challenging and potentially plagued with errors and incorrect conclusions. In this article, we addressed data analysis practices specific to radiographic medical images. We identified commonly observed pitfalls and recommended vital guidelines. Recommendations given here are not exclusive solutions and do not guarantee completely error-free unbiased experimentation. However, they do act as guidelines for the management and handling of research-compromising pitfalls. Mindful implementation of these guidelines will enhance the quality of radiomic analyses as well as allow for better integration of imaging data with other patient-specific data for precision medicine efforts.

No potential conflicts of interest were disclosed.

The authors acknowledge financial support from the NIH (NIH-USA U24CA194354 and NIH-USA U01CA190234 to H.J.W.L. Aerts).