Label-Free Deep Profiling of the Tumor Microenvironment

Tumor cells, cancer-associated stromal cells, and extracellular vesicles (EV) orchestrate their influence on cancer progression with regionally distinct structural, molecular, and metabolic changes (1–6). Direct microscopic analysis of cells and EVs in living tissue is imperative for understanding the spatial and functional heterogeneity or landscape of the tumor microenvironment, which has profound biological and clinical implications. Many advances have been made in the area of microscopy thanks to the rapid development of markers and labels (7–11). However, marker-based analyses are fundamentally limited by the complex tissue distribution of the markers, unknown disturbance of physiologic functions, the requirement of prior knowledge about the targeted molecule or receptor, unavoidable artifacts of nonspecific false-positive binding, and the limited number of fluorescence labels that can be used simultaneously (12, 13).

Label-free nonlinear optical microscopy, which produces high-resolution images with rich functional and structural information based on intrinsic molecular contrast, has demonstrated strong potential to overcome these problems by generating a broader array of volumetric signals from tissue structures and molecular composition (14–20). Here we use two-photon autofluorescence (2PF) and three-photon autofluorescence (3PF) to target the metabolic coenzymes FAD and NAD(P)H, respectively, which are the primary electron acceptor and donor in oxidative phosphorylation (21). Metabolic imaging of the relative concentration of these two coenzymes reflect the activity of the mitochondrial electron transport chain, and can shed light on metabolic changes including cellular metabolic rate and vascular oxygen supply (17, 21, 22). Second harmonic generation (SHG) occurs for noncentrosymmetric molecules, resulting in its high sensitivity and specificity to collagen fibers (23). Third harmonic generation (THG) occurs for interfaces with optical heterogeneity, such as sharp differences in refractive index dispersion, resulting in its sensitivity to a variety of structural interfaces including adipocytes, leukocytes, and microvesicles (14). Using a tailored optical fiber source with unique wavelengths (1,080–1,140 nm), pulse shaping, and repetition rate (10 MHz), the metabolic signatures of the tissue via 2PF of FAD and 3PF of NAD(P)H (15, 24–27), and the structural properties (lipid–water interface, collagen fibers) via THG (optical heterogeneity) and SHG (noncentrosymmetric structures; ref. 14), can be captured with precise temporal and spatial coregistration (25, 28). The previously developed simultaneous label-free autofluorescence multi-harmonic (SLAM) microscopy (see more in Methods) provided a useful platform for simultaneously visualizing and quantifying various cell types, EVs, and the extracellular matrix in living tissue based on their intrinsic, metabolic, and structural optical signatures.

With the rich molecular information provided by label-free nonlinear microscopy, a systematic and comprehensive spatial analysis of the individual cells and EVs promises to provide a unique view of the complex and heterogeneous tumor microenvironment. Computational tools that can capture the spatial diversity and molecular alterations of cancer cells, stroma cells, and EVs in their authentic environment will accelerate the rate of biological discovery as well as the translation of this knowledge into new clinical biomarkers. Here, we developed a computational pipeline that analyzes the major cell types and EVs in the authentic tumor microenvironment from label-free nonlinear microscopy. Identification of the major components of the tumor microenvironment, including tumor cells, stromal components (fibroblasts, endothelial cells, lymphocytes, red blood cells, adipocytes; ref. 29), and EVs, was performed by a multiclass pixel-level deep neural network (DNN)-based segmentation model (30, 31). By comparing cancer tissue with normal tissue, extensive tissue reorganization and formation of a patterned cell–EV neighborhood was observed in the tumor microenvironment. To infuse the metabolic and structural information into the spatial analysis, the morphologic features extracted from the segmentation maps were overlaid with the optical signatures, which is a combination of molecular structural contrast and metabolic contrast. The computational pipeline developed here allows for deep profiling of cells and EVs within the original microenvironmental context in a single-cell/EV, label-free, and multilevel fashion, and will advance investigations and our understanding of the spatial heterogeneity of the tumor microenvironment. In addition, due to the label-free nature and the streamlined analytic capacity, the proposed computational pipeline can potentially serve as a complementary real-time analysis tool for point-of-procedure diagnosis (32), providing new insights into disease and therapy monitoring at the point-of-care through metabolic mapping of the tumor microenvironment (16, 33).

Laboratory-based virtual histology images were collected using a custom-built benchtop multimodal multiphoton microscope capable of performing intravital simultaneous label-free autofluorescence multiharmonic (SLAM) microscopy in normal and tumor-bearing rats (25, 28). The system setup and acquisition parameters largely follow our previous work (25). The excitation beam uses 1,110 ± 30 nm pulses generated from a fiber supercontinuum. For detection channels, THG uses a band-pass filter of 370 ± 10 nm, 3PF 450 ± 30 nm, SHG 555 ± 15 nm, and 2PF 610 ± 30 nm. For this pilot study, three rats were imaged for the control group (normal/healthy) and three rats were imaged for the experimental group (cancer). From these rat models, images were acquired with a field-of-view of 1 mm × 1 mm (4 million pixels) from 7 different imaging sites in the experimental group and from 6 different imaging sites in the control group, at an optical resolution of 700 nm and a pixel size of 500 nm. Animal procedures were conducted in accordance with a protocol approved by the Institutional Animal Care and Use Committee at the University of Illinois at Urbana-Champaign (Champaign, IL). To induce mammary tumors in the female Wistar-Furth rats, NMU (N-Nitroso-N-methylurea; Sigma) diluted in distilled water (12.5 mg per mL) was injected intraperitoneally at a concentration of 55 mg per kg into the left side of the abdomen when the animals (Fisher 344, Harlan) were 7 weeks old. One week later, the same amount of NMU was injected intraperitoneally into the right side of the abdomen. After approximately 12 weeks of age, when mammary tumors became palpable, rats were prepared for in vivo imaging. Surgery to expose the primary tumor and its neighboring mammary tissue was performed under isoflurane anesthesia. During the imaging sessions, each rat was anesthetized with 1% isoflurane mixed with O₂, at a flow rate of 1 L per minute, and every effort was made to minimize discomfort. Physiologic temperature was maintained by a heating blanket. Imaging duration was kept under 3 hours to avoid complications of long-term anesthesia. Rats were euthanized immediately after imaging. Histologic slides were obtained after each imaging session, which were used as our gold standard for correlating and recognizing cells and structures of the tissue.

The input to the algorithm was the raw four-channel virtual slides (multiphoton images) together with the multiclass pixelwise labeling (negative, tumor cells, fibroblasts, endothelial cells, lymphocytes, EV-shedding stroma cells, red blood cells, adipocytes, and EVs). All input images were converted to 8-bit representation and saved directly to the GPU memory to facilitate access for the training process later. During each iteration, a mini-batch of 32 sets of 128 × 128 × 4 tiles was randomly cropped and augmented (flipped and rotated) from randomly chosen virtual slides. The loss function was defined as the cross entropy between the ground truth (manual labels obtained by correlation with histology and domain expertise) and the predicted probability of each class. We followed the modified U-Net (30) architecture by Yang and colleagues (31), where ResNet bottleneck design with identify shortcuts and batch normalization are applied on top of the standard U-Net (Supplementary Fig. S1; ref. 30). An Adam optimizer was used for iteration steps, with a learning rate of 0.0005, first momentum coefficient of 0.9, second momentum coefficient of 0.999, and epsilon of 1 × 10⁻⁸, which were adopted from previous segmentation work (34).

Similar to hematoxylin and eosin (H&E) slides, our images have high resolution (500 nm as pixel size) and large field-of-view (1 mm × 1 mm). Because there are many cells in each mosaicked image, we selected subpatches from the mosaicked image to perform labeling. More specifically, 226 image patches (128 by 128 pixels) were used for training, 16 image patches (128 by 128 pixels) from were used for validation, and 22 image patches (128 by 128 pixels) from were used for testing (results shown in Supplementary Table S1). The main challenge of using a DNN for cell segmentation is the limited training data. As anti-rat antibodies specific to different immune cells are challenging to obtain and apply, we resorted to different approaches for validation of different cell types. Spatial alignment and correlation between label-free multiphoton images and second-round marker-based images can be difficult and was performed with care and labor. For example, in the case of stromal cell labeling in normal breast tissue, as dipping and immersion is required for acridine orange to work on thick tissues, we relied on the geometry of the tissue and collagen fiber structures to maintain the same spatial location, depth, and orientation when the tissue was placed back onto the imaging stage for the follow-up visualization (after acridine orange labeling). A more robust method is needed in the future for a larger-scale study to fully unleash the power of label-free intravital imaging.

The rendering of SLAM images follows our previous work (25). Without image preprocessing, the raw four-channel image/video was loaded onto FIJI (NIH, Bethesda, MD) to apply pseudo-color maps to merge the contrast. The same color maps were used consistently throughout the study, namely magenta (hot) for THG, cyan (hot) for 3PAF, green for SHG, and yellow (hot) for 2PAF. These maps were then loaded into MATLAB to facilitate automatic generation of pseudo colored images. For each set of SLAM image, the segmentation mask was first created as a stack of images with multiclass labels, where each image corresponded to the probability map of a certain class. Then, we assigned different colors to different labels and overlaid the different labels onto one image, which generated the multiclass segmentation mask shown in Fig. 3 (middle). To have a better correlation of the original image and the segmentation mask, the segmentation mask was added to the original image at one-third the intensity, which generated the overlaid image in Fig. 3 (right).

A total of 8 features were chosen among the features extracted from the spatial map and multimodal signatures of the images. Morphologic features included the diameter, symmetry, area, perimeter, roundness, and irregularity of the shape. Optical features included the intensity of 2PF and 3PF, representing the relative level of FAD and NAD(P)H, respectively, and the THG being generated from the lipid layers/structures of each EV (optical heterogeneity at the aqueous–lipid interface). The co-occurrence value describes the probability of the co-occurrence between two cell types (35). A high probability score (or a high degree of positive association between A and B) corresponds to a high probability of co-occurrence of cell type A and cell type B. The probability was estimated by computing the frequency of cell type B as the nearest neighbor in the presence of cell type A (as the center/index cell of niche) divided by the expected frequency of the cell type B. This metric was adapted from Goltsev and colleagues' work on labeled cell analysis of the mouse splenic architecture (35) to quantify the cellular interaction strength.

To describe the discrepancies of the properties between individual cells (Supplementary Table S2), the 95% confidence interval of the average feature value was computed using bootstrap with 10,000 bootstrap samples for each feature. A two-sample Student t test was performed to compare the co-occurrence probability of cell pairs between the cancer samples (n = 7) and normal samples (n = 6; Supplementary Fig. S2A–S2C). As the most frequent cells in cancer tissue (e.g., tumor cells, EV-shedding stromal cells) are absent (or scarce) in the normal tissue, the statistical significance of the high association between two cell types within cancer tissue (e.g., endothelial cells and EV-shedding stromal cells) was tested against other cell pairs (e.g., the normally highly associated pairs—endothelial cells and fibroblast) within the same set of cancer tissue (n = 7) using the paired Student t test (Supplementary Fig. S2D and S2E). To investigate the correlation between the optical signatures of EVs and their distance to different cell types, the Pearson linear correlation coefficient was computed, and the corresponding P value was computed for testing a nonzero Pearson correlation using a Student t distribution (EV number = 10,620).

The data and code to perform the analyses have been deposited in OSF (https://osf.io/tgprd/?view_only=22262b9c2a394af8a41327ff553d3823) and on GitHub (https://github.com/Biophotonics-COMI/Segmentation-SLAM). The authors provide these for joint collaborative investigations with other research groups.

Similar to other cell-centered microscopic analyses, the first step of quantitative analysis for label-free nonlinear microscopy was to identify biological structures of interest and segment the multimodal images into meaningful regions (Fig. 1). Tumor cells, stroma cells, and EVs were of primary interest for their vital roles in carcinogenesis and metastasis. By correlating with histologic images and antibody-labeled images, a wide array of key cell types and structures stand out from the label-free multiphoton images due to their distinct endogenous fluorescence, unique morphology, and consistent spatial contextual patterns in the tumor microenvironment (25–28). Inspired by previous work on artificial labelling of label-free cell/slide imaging (36, 37), here we investigated the possibility of artificial labeling of label-free intravital imaging in a more complex and richer tumor microenvironment. As demonstrated previously, a DNN is a proper candidate for this task due to its ease of integrating multidimensional spatial cues in the absence of explicit physical models. In this case, the DNN directly learns from data about the implicit rules that explore optical signatures, morphologic cues, and contextual relationships for every target of interest.

The main challenge arises from the difficulty of obtaining ground truth for label-free intravital imaging or thick tissue imaging. We have resorted to different validation approaches for different targets of interest within the label-free multiphoton images over a series of studies (25–28). For cells and structures that have distinct morphologic features such as tumor cells, vascular endothelial cells, and red blood cells, we relied on histology for morphologic correlation. Comparisons between representative SLAM microscopy images and corresponding H&E-stained tissue sections are shown in Fig. 2A–C. The high degree of similarity in size, morphology, and spatial distribution provides morphologic evidence for the identity of tumor cells, vascular endothelial cells, and red blood cells in the images. For cells that cannot be reliably recognized from conventional histology, such as leukocytes, we used surface protein markers (Fig. 2D). Without exogenous contrast, lymphocytes are visualized as THG-visible hollow spheres (6–8 μm in size) due to the distribution the intrinsic lipid granules within the cell bodies, which give rise to phase-matching THG signals (38). With the injection of a CD45-based marker (mAb OX1, eBioscience, for leukocytes), lymphocytes are expected to exhibit the color of the tag (cyan channel in this example). The correlation between the CD45 images and THG images provides guidance to identify lymphocytes.

To validate the performance of SLAM microscopy for normal cell detection, we used acridine orange (a nuclear dye) for labeling the fresh normal breast tissue and obtained images before and after the acridine orange staining (Fig. 2E). Despite the unavoidable spatial misalignment (caused by the dipping procedure for acridine orange staining), a largely consistent match can be observed from the content between the label-free SLAM image and the acridine orange-labeled image–cell cytoplasm by label-free SLAM image (THG and 2PF channel) and yellow spindles in the acridine orange–labeled image (stroma, not well-resolved due to saturation of the 2PF/dye channel), which demonstrates the fidelity of SLAM detecting normal stromal cells. To evaluate the efficiency for detecting EVs from mammary cells of varying breast cancer status, our previous work isolated EVs from the conditioned media of different human breast cancer cell lines and quantitatively characterized the reliability of label-free detection of EVs by SLAM microscopy (28). With no exogenous markers, the majority of EVs appear as individual diffraction-limited (their size usually smaller than the optical resolution of the microscope) bright punctate pixels in multiphoton microscopy images, with the intensity of 2PF and 3PF representing the cargo concentrations of FAD and NAD(P)H within the EVs (Fig. 2F), respectively, and the THG being generated from the lipid layers/structures of each EV (optical heterogeneity at the aqueous–lipid interface; ref. 14). Correlations of these label-free signatures with conventional analysis methods for EVs (electron microscopy, PKH26, and nanoparticle tracking analysis) have been demonstrated in a previous study (28), showing high sensitivity (91%) and specificity (99%) of the proposed label-free detection method.

Relying on these validation approaches, we manually annotated the label-free multiphoton images and feed them to the neural network. Training pairs consisting of label-free multicolor images and their corresponding manually labeled pixel-registered segmentation masks were created to train the DNN to segment cells and EVs of interest at the single pixel level. The training datasets focused on major cell types that are present in breast tissue, especially mammary cancer tissue, including tumor cells, fibroblasts, endothelial cells, lymphocytes, red blood cells, adipocytes, and EVs. With these training datasets, a supervised modified U-Net (30) model was trained to perform the task of nonlinear pixel-level classification (see more in Methods and Supplementary Fig. S1). By applying the trained model to label-free nonlinear microscopy images, a probability map for each class was simultaneously produced, assigning a probability of that class for each pixel of previously unseen input images. Figure 3 shows representative examples of automatically generated multiclass segmentation masks of cancer (25), and normal tissue and Supplementary Table S1 reports the segmentation performance for each class in the test dataset. A highly heterogenous distribution of various cell subtypes was observed in the cancer tissue while a relatively homogeneous distribution of fibroblasts and adipocytes was observed to dominate the cell populations in normal tissue (Figs. 3A and B and 4A and B). To capture the spatial and metabolic heterogeneity of cancer tissue, the individual properties, spatial distributions, and neighborhood influence of cells and EVs were quantitatively characterized and compared with normal tissue in the following sections.

With the multiclass segmentation mask generated by the DNN model, individual cells and EVs were readily characterized by their metabolic, structural, and morphologic properties in the living tissue. For this study, three rats were imaged for the control group (normal/healthy) and three rats were imaged for the experimental group (cancer). From these rat models, images were acquired with a field-of-view of 1 mm × 1 mm (4 million pixels) from 7 different imaging sites in the experimental group and from 6 different imaging sites in the control group, at an optical resolution of 700 nm and a pixel size of 500 nm. Metabolic and molecular structural features were extracted by examining the autofluorescence signals from NAD(P)H by 3PF and FAD by 2PF (21, 39), and the harmonic signals from phase-matched molecular structures, including SHG from collagen fibers and THG from micro-sized lipid–water interfaces (23, 40). Morphologic features were extracted by using common metrics for describing cells, including diameter, area, perimeter, roundness, and irregularity (41, 42). To visualize these features associated with different biological entities, two heatmaps were generated (Fig. 4C and D) to display these features for each type of biological entity in cancer tissue and normal tissue, respectively. The average values of these features and the corresponding confidence intervals (without normalization) are reported in Supplementary Table S2. One of the most obvious differences between the cancer and control groups is that the average size of lipid droplets decreased more than two times in the cancer tissue (Supplementary Table S2; refs. 43, 44). This was previously observed in living mice and human breast cancer, and is attributed to the overexpression of collagen VI and the extensive extracellular matrix near tumors (45). As expected, certain cell types demonstrated relatively consistent morphologic features and optical signatures, regardless of cancer status. For example, in both cancer and normal tissues, endothelial cells exhibited the highest level of irregularity with their elongated shape, and fibroblasts exhibited a higher ratio value of FAD/(FAD+NAD(P)H) compared with all other cells and EVs. Interestingly, the NAD(P)H-rich EV-shedding stromal cells were observed to be abundant in cancerous tissues, while completely absent in normal tissues (Fig. 4A–D). Consistent with our previous study (28), EVs from cancer tissue appeared to have lower mean values of FAD/(FAD+NAD(P)H) than EVs from normal tissue. Individual EVs were mapped to the 2-D space of optical signatures featuring the metabolic information by autofluorescence and molecular structure by THG. A larger spread in both dimensions and formation of at least two clusters were observed in the EV scatterplot of cancer tissue, which suggests the higher heterogeneity of EV properties in the form of distinct subpopulations in the tumor microenvironment (Fig. 4E and F). This hypothesis is investigated in later sections. This higher concentration of NAD(P)H has been observed in cancer cells (21, 39) previously as a result of the Warburg effect (increased glycolysis; ref. 46), and also a potential result of the antioxidant defense mechanism in endothelial microvesicles (47). This is an intriguing observation that highlights the possibly unique metabolic functions and signatures of EVs in breast cancer (47–49), which can potentially serve as a biomarker and to better our understanding of the active metabolic role that EVs play in cancer progression.

In addition to single-cell and single-EV analysis, a higher-level spatial analysis was performed from the perspective of cell-to-cell/EV concurrence and proximity to characterize the microenvironmental landscape, which is known to involve the active and dynamic collaboration between different cell types and EVs. Inspired by previous work on deep profiling (35), the probability of the interaction between two cell types, or one cell type and EVs, was quantified by the ratio of the observed and the expected frequency of concurrence between every pair of cell type and/or EV. The heatmaps, which summarize the pairwise cell-to-cell/EV concurrence maps, revealed highly correlated patterns between cell types and EVs. For example, a consistently high degree of association was observed between endothelial cells and NAD(P)H-rich EV-shedding stroma cells (Fig. 5A and B; Supplementary Fig. S2D), which lends further support to the active roles of EVs in angiogenesis (50). In addition, in tumor tissue, smaller and isolated adipocytes are frequently observed at the tumor boundary (Fig. 5C; Supplementary Fig. S2E), which could be a result of lipolysis induced by cancer cells (51). In contrast, the most likely neighbors of endothelial cells in normal tissues are fibroblasts (Fig. 5D and E; Supplementary Fig. S2A), which could be attributed to the reliance on fibroblast-derived matrix proteins in angiogenesis (52). The diagonal line of the heatmap matrix summarizes the likelihood of self-clustering of every cell type and EVs. It was found that in normal tissues, adipocytes (lipids) tend to cluster and pack together with their own type, in the company of fibroblasts (Fig. 5F; Supplementary Fig. S2B and S2C). On the other hand, stroma cells are observed to have a relatively even probability distribution for all the potential neighbors, which is probably due to their prevalent and relatively homogeneous distribution in the stroma, compared with other cells. This frequency analysis of concurrence enabled a high-level view of the cell-to-cell and cell-to-EV interaction landscape with quantitative details, and indicated a significant nonrandom distribution of cells/EVs in the living tumor microenvironment.

The metabolic profiles of EVs and cells are highly heterogenous, depending on their origin as well as their current surroundings (1, 3, 5, 53, 54). The metabolic heterogeneity of EVs was highlighted by the 2-D scatterplot of optical signatures in Fig. 4E and F, which shows subpopulations of cancer EVs that have dramatically different metabolic and structural signatures. Interestingly, a metabolic-focused spatial analysis in the form of EV-to-cell concurrence mapping suggests a profound influence of the cellular neighborhood on the metabolic products and structural properties of the EVs. Each EV was categorized by its closest cell neighbor, for example, if the closest cell neighbor of an EV is an endothelial cell, then this EV counts as an endothelial cell associated EV (defined in this work for simplicity of reference). Thus, the influence of cell neighborhoods on EV properties can be visualized by mapping the optical signatures of individual EVs in combination with the spatial relationship between EVs and different cell types (Fig. 6; Table 1). For instance, EVs in the proximity of endothelial cells (green) and EV-shedding stroma cells (cyan) are strongly correlated with high levels of NAD(P)H, while EVs neighboring tumor cells exhibit the opposite properties, suggesting potentially different metabolic roles of EVs in angiogenesis and tumor growth. In contrast to such distinct distributions, EVs neighboring fibroblasts display much more homogeneous optical properties in both NAD(P)H levels and THG signals, suggesting a weaker influence of fibroblasts on the NAD(P)H levels of neighboring EVs. These results shed light on the high metabolic, structural, and spatial heterogeneity of EVs, and highlight the importance of deep profiling of EVs in the context of various cellular neighborhoods for understanding this heterogeneity. The rise of NAD(P)H levels in the subpopulations of cancer associated EVs has been correlated with early and late stages of human breast cancer (28). The significantly positive correlation between NAD(P)H-rich EVs and vascular endothelial cells observed in this study (Table 1) indicates that the rise of NAD(P)H levels in cancer-associated EVs could be attributed to increased angiogenesis and the antioxidant defense mechanisms surrounding newly growing vessels (47). The specific roles of EVs in cancer progression have been of high interest but remain largely unknown. The proposed methodology provided the capability to map the relationships between cells and EVs in the context of the authentic microenvironment. A large-scale follow-up study in cancer progression is warranted to further illustrate the significance of these observations.

This study developed a computational framework for the systematic spatial analysis of label-free nonlinear microscopy images, with the ultimate goal of understanding the spatial heterogeneity of the tumor microenvironment from the perspective of single-cell/EV properties and cell-to-cell/EV influence within the unperturbed living tissue environment. To enable the subsequent single-cell/EV analysis, automatic segmentation of label-free nonlinear microscopy was achieved by a multiclass (tumors, cancer-associated stroma cells, and EVs) pixelwise DNN-based model. With these generated segmentation masks, the concurrence probability between every pair of cells and/or EVs was computed to provide a high-level view of the highly collaborative tumor microenvironment. By comparing cancer tissue with normal tissue, extensive tissue reorganization and the formation of a patterned cell–EV neighborhood was observed in the tumor microenvironment. The metabolic spatial analysis of individual EVs revealed how the metabolic and structural properties of EVs depend on their local cellular neighborhoods, which provided an important illustration and potential rationale for the essence of in situ imaging to capture and further understand the EV heterogeneity.

Previous work has shown that the metabolic status of tumor cells can be characterized by label-free multiphoton microscopy and has been demonstrated to be indicative of disease progression, as well as the efficacy of treatment regimens (16, 17, 21). The proposed pipeline of automatic segmentation and quantification can enable similar large-scale single-cell single-EV analysis in these efforts. On the other hand, building on the results from previous work of label-free multiphoton microscopy, this work highlights the inclusion of multiple cell types and EVs in the analysis of the tumor microenvironment, which is an authentic reflection of the heterogeneity and complexity of the tumor microenvironment. Part of our observations (e.g., clustering of tumor cells, and adipocytes) corroborate phenomena that have been shown using conventional methods (43, 46, 50, 52), which demonstrates the feasibility of this label-free method. More importantly, because of the direct metabolic and structural contrast, we were able to monitor the metabolic status of individual EVs and cells and their neighborhoods, which could not be easily achieved by marker-based methods. For instance, we observed that EVs in the proximity of vascular endothelial cells are strongly correlated with high levels of NAD(P)H, while EVs neighboring tumor cells exhibit the opposite properties (Fig. 6; Table 1), suggesting potentially different metabolic roles of EVs in angiogenesis and tumor growth. Such single-EV single-cell spatial analysis paints a picture that other methods (lipid biopsy, or marked-based EV imaging) could not, and thus is an attractive complementary method that may advance investigations and our understanding of the metabolic heterogeneity of the tumor microenvironment.

Furthermore, these new features can potentially be used in clinical settings to identify earlier changes in cancer progression as the reorganization of the microenvironment can happen long before the formation of a tumor mass or the actual event of metastasis (6, 55). Because of the label-free and streamlined analytic capacity, the proposed computational pipeline can serve as a real-time analysis tool for point-of-procedure diagnosis. We have demonstrated that cancer classification at the point-of-procedure is possible with the combination of label-free multiphoton microscopy and a convolutional neural network (32). The pipeline proposed in this paper (quantitative single-cell single-EV analysis based on segmentation) will add interpretability and statistical confidence to the classification network, which is largely a black box with limited interpretability. In addition to interpretation, this method promises to give pathologists a full view of the metabolic spatial mapping of the tissue, which is complementary to the existing tissue assessment methods, and can provide new and personalized insights into disease/therapy monitoring (16, 33).

The main bottleneck of the proposed methodology is the availability of training data for segmenting label-free intravital images. Although the validation approaches used in this work (histology, protein surface markers, dyes) are essential for generating pixel-level ground truths and subsequent proof-of-concept analysis for intravital label-free images, we want to emphasize that the user-annotated approach is labor-intensive and might not be a sustainable solution for large-scale biomedical studies. Previous work has elegantly addressed the validation problem by imaging the same sample using both label-free and marker-based methods, which directly provided pixel-coregistrated training data (36, 37). This approach is much more challenging to implement for intravital imaging due to the limited options of in vivo markers for our animal models, the incompatibility between different markers, and potential contamination of the label-free fluorescence channels. These challenges will be investigated fully in the future to establish a more robust framework (e.g., utilizing genetic labeling techniques) for validating label-free multiphoton microscopy. Another opportunity for improvement is that this pipeline only considers two-dimensional images despite the imaging sample being a three-dimensional (3D) thick tissue. As multiphoton microscopy has the advantage of optical sectioning and deeper penetration, it will be an interesting future direction to extend this pipeline to 3D analysis to obtain a more comprehensive view of the 3D tumor microenvironment.

In summary, our study demonstrated a label-free profiling tool that enables simultaneous metabolic, structural, and spatial analysis of the cell–EV architecture within the context of the living tissue/tumor environment. The metabolic, structural, and spatial profiling capacity of the developed computational framework could open possibilities for label-free profiling of other tissue architectures, and enhances our understanding of the spatial heterogeneity involved in a broad array of tissue microenvironments.

S. You reports funding from a McGinnis Medical Innovation Graduate Fellowship (UIUC) during the conduct of the study; in addition, S. You has a patent 10,445,880 issued and licensed to LiveBx, LLC. E.J. Chaney reports grants from NIH during the conduct of the study. H. Tu reports grants from NIH during the conduct of the study and other grants from LiveBx, LLC, outside the submitted work; in addition, H. Tu has two patents, issued and licensed to LiveBx, LLC. Y. Sun reports grants from NIH during the conduct of the study. S.A. Boppart reports grants from NIH during the conduct of the study, and personal fees and equity from LiveBx, LLC, outside the submitted work; in addition, S.A. Boppart has multiple patents pending, issued, and licensed to LiveBx, LLC. No disclosures were reported by the other authors.

S. You: Conceptualization, resources, data curation, software, formal analysis, validation, investigation, visualization, methodology, writing–original draft, writing–review and editing. E.J. Chaney: Resources, investigation. H. Tu: Resources, funding acquisition, investigation. Y. Sun: Resources, data curation, investigation. S. Sinha: Conceptualization, supervision, investigation, writing–review and editing. S.A. Boppart: Conceptualization, resources, supervision, funding acquisition, investigation, project administration, writing–review and editing.

This work was supported by R01 grants CA213149 and EB023232 (to S.A. Boppart, supporting E.J. Chaney, S. Saurabh, and S.A. Boppart) and R01 CA241618 (to H. Tu and S.A. Boppart, supporting S. You, Y. Sun, H. Tu, and S.A. Boppart). S. You acknowledges support from the McGinnis Medical Innovation Graduate Fellowship. The authors thank all members of Biophotonics Imaging Laboratory for the helpful discussion and suggestions. Additional information can be found at http://biophotonics.illinois.edu. The authors also thank the editors and reviewers for their critical review and constructive suggestions for this manuscript.

The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.