Artificial intelligence (AI) trained with a convolutional neural network (CNN) is a recent technological advancement. Previously, several attempts have been made to train AI using medical images for clinical applications. However, whether AI can distinguish microscopic images of mammalian cells has remained debatable. This study assesses the accuracy of image recognition techniques using the CNN to identify microscopic images. We also attempted to distinguish between mouse and human cells and their radioresistant clones. We used phase-contrast microscopic images of radioresistant clones from two cell lines, mouse squamous cell carcinoma NR-S1, and human cervical carcinoma ME-180. We obtained 10,000 images of each of the parental NR-S1 and ME-180 controls as well as radioresistant clones. We trained the CNN called VGG16 using these images and obtained an accuracy of 96%. Features extracted by the trained CNN were plotted using t-distributed stochastic neighbor embedding, and images of each cell line were well clustered. Overall, these findings suggest the utility of image recognition using AI for predicting minute differences among phase-contrast microscopic images of cancer cells and their radioresistant clones.
This study demonstrates rapid and accurate identification of radioresistant tumor cells in culture using artifical intelligence; this should have applications in future preclinical cancer research.
Recently, there has been a remarkable development in image recognition technology based on artificial intelligence (AI) trained with a machine learning method, called deep learning. Also, extensive research on computer-aided diagnosis has been conducted in several fields of medicine using medical images, such as radiologic images (X-ray, CT, and MRI) and pathologic images (cytology and histology; refs. 1–6). The victory at the ImageNet Large Scale Visual Recognition Challenge in 2012 has popularized image recognition using deep learning (7). In 2015, the accuracy of AI exceeded human image recognition performance in the same contest (8).
In deep learning, multilayered learning circuits called neural networks simulate human neurons. Deep learning facilitates the extraction of appropriate features from an image. One of the leading neural networks used in image recognition is the convolutional neural network (CNN; ref. 7). The CNN is organized on the basis of the human visual system and is a robust network against image shift (9). However, deep learning warrants substantial training data to enhance the performance of the CNN, especially with deep multilayered networks. However, it is impractical to prepare such extensive training data in some cases. Thus, while the transfer learning technique is used to reduce the amount of data required for learning, employing a pretrained CNN is used to reduce the number of times learning is required (10). Reportedly, while the lower convolutional layers capture low-level local features such as edges, higher convolutional layers capture more complex features reflecting the entire image (7). In transfer learning, learning efficiency is enhanced by optimizing the parameters of only the higher layers without altering the lower layers.
This study aims to apply image recognition technology with AI in clinical decision-making using microscopic images of clinical specimens. In particular, we intend to establish the technology using microscopic images of cancer cells to predict the effect of chemotherapy and/or radiotherapy. This will enable the development of objective indicators to personalize cancer treatment according to the patient's requirements. Also, we aim to classify controls and radioresistant clones of human and mouse cancer cell lines using phase-contrast microscopic images. Furthermore, we intend to determine the features extracted by the trained CNN and assess the correlation among the features of five classes of cells. This study demonstrates that it is feasible to distinguish between cell lines and their radioresistant clones from the limited amount of visual information available in phase-contrast microscopic images.
Materials and Methods
Figure 1 illustrates the workflow of this study.
Cell lines used in this study included NR-S1 controls, NR-S1 × 60 (radioresistant to X-ray), NR-S1 C30 (radioresistant to carbon ion beam; refs. 11, 12), ME-180 controls, and ME-180 X-ray–resistant cell lines. Cells were cultured in DMEM (Sigma-Aldrich) supplemented with 10% (volume/volume) FBS (HyClone; GE Healthcare) and 1% (volume/volume) penicillin/streptomycin (Sigma-Aldrich), and maintained at 37°C in a 5% CO2 incubator. The NR-S1 cell lines were kindly provided by Dr. Katsutoshi Sato (Icahn School of Medicine at Mount Sinai, New York, NY) in 2014. The ME-180 parental cell line was kindly provided by our colleague Dr. Keisuke Tamari (Graduate School of Medicine, Osaka University, Osaka, Japan) in 2017. The initial passage numbers for cells we used were more than 30. All the relevant experiments were conducted within 10 passages from revival of the initial frozen seeds. All the relevant experiments were conducted within 10 passages from revival of the initial frozen seeds. Mycoplasma testing was performed using the MycoAlert Mycoplasma Detection Kit (Lonza; catalog code: LT07-218). Mycoplasma testing confirmed negative results. The cell authentication of ME-180 was performed with analyzing the short tandem repeat profile by the National Institute of Biomedical Innovation (Osaka, Japan).
Establishment of radioresistant cells
The NR-S1 × 60 and C30 cells were established as described previously (11, 12). Briefly, NR-S1 parental control cells were irradiated with 60 Gy of X-ray at a rate of 10 Gy once every 2 weeks. The NR-S1 C30 cells were established by irradiating NR-S1 parental control cells with 30 Gy of carbon ion beam radiation at a rate of 5 Gy once every 2 weeks. The ME-180 X-ray–resistant cells were established by irradiating ME-180 parental control cells with 60 Gy of γ irradiation at a rate of 2 Gy at every passage. Cells were cultured for a week after the final irradiation and then used for the experiment.
Clonogenic survival assay
Cells were harvested with TrypLE Express (Thermo Fisher Scientific), seeded onto cell culture dishes, and incubated at 37°C under 5% CO2 for 2 hours. Subsequently, cells were irradiated with gamma irradiation using the Gammacell 40 Exactor (MDS Nordion) and incubated at 37°C under 5% CO2 for 7 to 13 days. Cells were then stained with 0.5% crystal violet (w/v) and counted. Colonies containing >50 cells were counted as cells that survived. The number of survived cells were plotted against the dose of gamma irradiation.
Image preparation and preprocessing
Cells were photographed using a phase-contrast microscope (BZ-X700; Keyence). For each cell type, 5,000 images at a resolution of 640 × 480 sq. pixels were captured. Two images of 320 × 320 sq. pixels were cropped from the original images, and the resolution of the images was changed to 160 × 160 sq. pixels to facilitate image processing (Fig. 2). Overall, 50,000 images were captured, with 10,000 images per cell type. The processed images were divided into 8,000 training images and 2,000 test images for each cell type.
Neural network architecture and transfer learning
The CNN comprises multiple convolutional layers to detect the local features of inputs and pooling layers to reduce the computational burden, overfitting, and image shift. A pretrained CNN, namely VGG16, was used in this study; VGG16 has been published by the Visual Geometry Group of Oxford University (Oxford, United Kingdom) and has a high accuracy of image recognition (8, 13). Figure 3 presents the architecture of VGG16, including 13 convolutional layers and 3 fully connected layers. Although the original VGG16 outputs, that is, 1,000 parameters to categorize images in 1,000 classes, are defined in ImageNet, data in this study were categorized in only five classes: NRN, NRX, NRC, MEN, and MEX. Thus, the number of VGG16 outputs was changed from 1,000 to 5.
In this study, only the last 3 convolutional layers and 3 fully connected layers were trained using training images. The training dataset was processed in minibatches. A minibatch was randomly selected from the training dataset at each training step. The learning of all 40,000 images was defined as one learning or one epoch. Also, cross-entropy error was evaluated using the output of the VGG16 model, and the actual class of images was evaluated using “accuracy” as a metric. Then, backpropagation was performed, and model parameters were updated using a momentum stochastic gradient descent algorithm with a learning rate of 0.0001 and a momentum of 0.9. Overall, 20 epochs were trained; one epoch implies completing learning once with all training images. For each epoch, test images were used to assess the accuracy of the trained model. Furthermore, Google's TensorFlow (14) deep learning framework and Keras (15) followed by the data library of TensorFlow were used to train, validate, and test the model.
We declare that we used cell lines only and no animal samples including humans and mice in this study.
Confirmation of the radioresistant phenotype of cell lines
We used mouse squamous cell carcinoma NR-S1 and human cervical carcinoma ME-180 cell lines to establish radioresistant cells. In this study, NR-S1 and its X-ray- and carbon ion beam–resistant cell lines were named “NRN,” “NRX,” and “NRC,” respectively. Similarly, ME-180 and its X-ray–resistant cell lines were named “MEN” and “MEX,” respectively. To confirm the radioresistant phenotype of cell lines, we performed a clonogenic survival assay and estimated the survival fraction of each cell line subjected to 4 and 8 Gy of γ irradiation (Supplementary Fig. S1). Both NRX and MEX exhibited higher survival than the corresponding control cell lines (NRN and MEN).
Phase-contrast microscopic images of cell lines were captured, and square images were cropped from the original rectangular images. From a total of 50,000 images, we obtained 10,000 microscopic images for each cell line at a resolution of 160 × 160 sq. pixels. Representative images are shown in Fig. 2. Cells of NRX were marginally smaller than those of NRN. However, differences between the size of NRC and NRN and between MEN and MEX were ambiguous. Also, we divided the image datasets into 8,000 training images and 2,000 test images; training images were used to train the neural network, and test images were used to validate the accuracy of the neural network and feature engineering.
Training and validating the CNN
In this study, we used the CNN, called VGG16 (Fig. 3), which was pretrained using ImageNet datasets (13). Using the pretrained model, the number of images required for learning was reduced, thus improving the learning speed, a process called transfer learning. We performed transfer learning to optimize only the last 3 convolutional layers and 3 fully connected layers using training data and trained the model for 20 epochs. Figure 4 shows the training course of each epoch. The use of the pretrained model resulted in a dramatic improvement in the accuracy of the model after only one epoch. Using test images, the accuracy of the model reached approximately 96% after one epoch but plateaued thereafter. The accuracy of the model was 99.9%, 98.8%, 99.8%, 98.7%, and 91.1% for NRN, NRX, NRC, MEN, and MEX, respectively, using test images. Although the accuracy of the classification of NR-S1 cell lines was high, it was difficult to classify the ME-180 cell lines, especially MEX, using this model. We performed Receiver Operating Characteristic analysis and calculated the area under the curve (AUC) to predict the trained VGG16 model. The AUC values of NRN, NRX, NRC, MEN, and MEX were 1.00000, 0.99991, 0.99978, 0.99793, and 0.99908, respectively. Overall, the trained VGG16 model showed a very high performance in distinguishing each cell line.
Elucidating the reasons behind predictions
The “Local Interpretable Model-agnostic Explanations” (LIME) method (16) was used to determine the area of emphasis of the trained CNN on the image. Using the LIME method on some test images, we visualized the bases in the images for the classification of the trained CNN. Representative results are shown in Supplementary Fig. S2. We successfully visualized the information emphasized by the trained CNN, suggesting that the CNN registered the shape of the cell or cell population because the boundary of the extracted region was along the edge of the cell.
Next, we extracted internal features from test images using the trained VGG16 model to assess the basis for the categorization of these images. We obtained the output of the last hidden layer of the trained VGG16. The CNN designed in this study changed an input image to 512 feature maps, comprising 5 × 5 square data, using convolutional layers. In addition, it integrated feature maps into 4,096 features by fully connected layers to render features useful for categorization into five classes. Although the 5 × 5 feature maps facilitated the visualization of data extracted from an image by the CNN, these maps were too many and too small to comprehend. Hence, 4,096 multivariate inputs were used for the final layer as features extracted from the image. Moreover, maps were reduced to two dimensions using t-distributed stochastic neighbor embedding (t-SNE) to visualize the features of 4,096 multivariate inputs (17). We created a scatter plot with these features (Fig. 5). Although each point represented a microscopic image of one cell line, each color represented a type of cell line (red, NRN; blue, NRX; black, NRC; orange, MEN; and cyan, MEX). We observed five clusters of points with the same categories. Three clusters of NR-S1 (NRN, NRX, and NRC) were distinct from each other, whereas clusters of ME-180 (MEN and MEX) were distributed relatively close together, implying that the VGG16 model recognized MEN and MEX as similar cell lines.
This study established that it is feasible for AI to accurately differentiate between cancer cell lines and their radioresistant clones, even if simple phase-contrast microscopic images are used as inputs, suggesting clear visual differences among multiple types of cells with different properties. These data suggest the potential to create universal AI using a variety of cells for learning. The distinction among the three types of NR-S1 cells was more accurate than that between the two types of ME-180 cells, which was consistent with our visual intuition. Among all cell types, MEX cells were the least accurate. The trained VGG16 model identified some photographs of MEX as MEN possibly because the MEX cells were very similar to MEN cells, and the VGG16 set the distinction threshold to be strict for MEX. The LIME method was used to illustrate the predictions of any classifier by learning an interpretable model locally around the prediction. We attempted to elucidate the visual basis for the classification of images by the CNN and extracted features using LIME or t-SNE; however, the results were unclear. Thus, it remains unclear whether AI could predict radiosensitivity using microscopic images of cell lines. In this study, only two kinds of cells were available, as the establishment of radioresistant clones of cancer cells requires considerable time and effort. Further investigations are needed to train the CNN with a higher number of cell lines and to verify its ability to predict radiosensitivity using cell lines not used for training.
Although the CNN is an excellent technique in the field of image recognition, several problems await resolution. For example, there exists a black box in the learning processes and extracted features associated with deep learning, such as the CNN. Whether prediction using deep learning is correct and has practical implications warrants further investigation. Also, since no mathematical support exists in modifying hyperparameters to enhance the result of deep learning, there is a need to explore optimal parameters by using random sampling or grid search. Optimization of hyperparameters using a machine learning approach such as Bayesian optimization may resolve this problem (18).
There have been several reports on the efficacy of image recognition using deep learning in the diagnosis of existing lesions and the qualitative diagnosis of tissue type (1–3, 5). However, much less work has been conducted on the prediction of the sensitivity of treatment such as radiotherapy using radiological or pathologic images. In the future, using big data training AI with more information will advance predictive medicine, including the prediction of treatment effect, and contribute to the realization of personalized medicine.
Disclosure of Potential Conflicts of Interest
T. Kudo reports receiving a commercial research grant from Yakult Honsha Co., Ltd., Chugai Pharmaceutical Co., Ltd., and Ono Pharmaceutical Co. Ltd. H. Ishii reports receiving a commercial research grant from Taiho Pharmaceutical Co. Ltd, Unitech Co. Ltd. (Chiba, Japan), IDEA Consultants Inc. (Tokyo, Japan), and Kinshu-kai Medical Corporation (Osaka, Japan). No potential conflicts of interest were disclosed by the other authors.
Conception and design: M. Toratani, M. Konno, M. Mori, K. Ogawa, H. Ishii
Development of methodology: M. Toratani, M. Konno, A. Asai, J. Koseki, K. Tamari, D. Sakai, D. Motooka, H. Ishii
Acquisition of data (provided animals, acquired and managed patients, provided facilities, etc.): M. Toratani, M. Konno, A. Asai, K. Kawamoto, Z. Li, T. Kudo, K. Sato, D. Okuzaki, H. Ishii
Analysis and interpretation of data (e.g., statistical analysis, biostatistics, computational analysis): M. Toratani, M. Konno, A. Asai, J. Koseki, Z. Li, D. Sakai, T. Satoh, K. Sato, K. Ogawa, H. Ishii
Writing, review, and/or revision of the manuscript: M. Toratani, M. Konno, T. Satoh, M. Mori, H. Ishii
Administrative, technical, or material support (i.e., reporting or organizing data, constructing databases): M. Konno, J. Koseki, D. Sakai, Y. Doki, H. Ishii
Study supervision: H. Ishii
We thank the laboratory staff for their helpful discussions. This work received financial support from grants-in-aid for Scientific Research from the Japan Agency for Medical Research and Development and the Ministry of Education, Culture, Sports, Science, and Technology (grant nos. 17H04282 and 17K19698 to H. Ishii), grant no. 16K15615 to M. Konno, and grant no. 15H05791 to M. Mori.