Artificial intelligence (AI) trained with a convolutional neural network (CNN) is a recent technological advancement. Previously, several attempts have been made to train AI using medical images for clinical applications. However, whether AI can distinguish microscopic images of mammalian cells has remained debatable. This study assesses the accuracy of image recognition techniques using the CNN to identify microscopic images. We also attempted to distinguish between mouse and human cells and their radioresistant clones. We used phase-contrast microscopic images of radioresistant clones from two cell lines, mouse squamous cell carcinoma NR-S1, and human cervical carcinoma ME-180. We obtained 10,000 images of each of the parental NR-S1 and ME-180 controls as well as radioresistant clones. We trained the CNN called VGG16 using these images and obtained an accuracy of 96%. Features extracted by the trained CNN were plotted using t-distributed stochastic neighbor embedding, and images of each cell line were well clustered. Overall, these findings suggest the utility of image recognition using AI for predicting minute differences among phase-contrast microscopic images of cancer cells and their radioresistant clones.

Significance:

This study demonstrates rapid and accurate identification of radioresistant tumor cells in culture using artifical intelligence; this should have applications in future preclinical cancer research.

Recently, there has been a remarkable development in image recognition technology based on artificial intelligence (AI) trained with a machine learning method, called deep learning. Also, extensive research on computer-aided diagnosis has been conducted in several fields of medicine using medical images, such as radiologic images (X-ray, CT, and MRI) and pathologic images (cytology and histology; refs. 1–6). The victory at the ImageNet Large Scale Visual Recognition Challenge in 2012 has popularized image recognition using deep learning (7). In 2015, the accuracy of AI exceeded human image recognition performance in the same contest (8).

In deep learning, multilayered learning circuits called neural networks simulate human neurons. Deep learning facilitates the extraction of appropriate features from an image. One of the leading neural networks used in image recognition is the convolutional neural network (CNN; ref. 7). The CNN is organized on the basis of the human visual system and is a robust network against image shift (9). However, deep learning warrants substantial training data to enhance the performance of the CNN, especially with deep multilayered networks. However, it is impractical to prepare such extensive training data in some cases. Thus, while the transfer learning technique is used to reduce the amount of data required for learning, employing a pretrained CNN is used to reduce the number of times learning is required (10). Reportedly, while the lower convolutional layers capture low-level local features such as edges, higher convolutional layers capture more complex features reflecting the entire image (7). In transfer learning, learning efficiency is enhanced by optimizing the parameters of only the higher layers without altering the lower layers.

This study aims to apply image recognition technology with AI in clinical decision-making using microscopic images of clinical specimens. In particular, we intend to establish the technology using microscopic images of cancer cells to predict the effect of chemotherapy and/or radiotherapy. This will enable the development of objective indicators to personalize cancer treatment according to the patient's requirements. Also, we aim to classify controls and radioresistant clones of human and mouse cancer cell lines using phase-contrast microscopic images. Furthermore, we intend to determine the features extracted by the trained CNN and assess the correlation among the features of five classes of cells. This study demonstrates that it is feasible to distinguish between cell lines and their radioresistant clones from the limited amount of visual information available in phase-contrast microscopic images.

Figure 1 illustrates the workflow of this study.

Figure 1.

Schematic representation of the overall experiment. Images of cell samples were obtained using a microscope and trimmed and stored as image data. Image data were separated into training data and test data and subjected to the deep CNN analysis to study data depiction. Training data were used for network optimization, and test data were used for the estimation of performance of trained AI and extracted features.

Figure 1.

Schematic representation of the overall experiment. Images of cell samples were obtained using a microscope and trimmed and stored as image data. Image data were separated into training data and test data and subjected to the deep CNN analysis to study data depiction. Training data were used for network optimization, and test data were used for the estimation of performance of trained AI and extracted features.

Close modal

Cell lines

Cell lines used in this study included NR-S1 controls, NR-S1 × 60 (radioresistant to X-ray), NR-S1 C30 (radioresistant to carbon ion beam; refs. 11, 12), ME-180 controls, and ME-180 X-ray–resistant cell lines. Cells were cultured in DMEM (Sigma-Aldrich) supplemented with 10% (volume/volume) FBS (HyClone; GE Healthcare) and 1% (volume/volume) penicillin/streptomycin (Sigma-Aldrich), and maintained at 37°C in a 5% CO2 incubator. The NR-S1 cell lines were kindly provided by Dr. Katsutoshi Sato (Icahn School of Medicine at Mount Sinai, New York, NY) in 2014. The ME-180 parental cell line was kindly provided by our colleague Dr. Keisuke Tamari (Graduate School of Medicine, Osaka University, Osaka, Japan) in 2017. The initial passage numbers for cells we used were more than 30. All the relevant experiments were conducted within 10 passages from revival of the initial frozen seeds. All the relevant experiments were conducted within 10 passages from revival of the initial frozen seeds. Mycoplasma testing was performed using the MycoAlert Mycoplasma Detection Kit (Lonza; catalog code: LT07-218). Mycoplasma testing confirmed negative results. The cell authentication of ME-180 was performed with analyzing the short tandem repeat profile by the National Institute of Biomedical Innovation (Osaka, Japan).

Establishment of radioresistant cells

The NR-S1 × 60 and C30 cells were established as described previously (11, 12). Briefly, NR-S1 parental control cells were irradiated with 60 Gy of X-ray at a rate of 10 Gy once every 2 weeks. The NR-S1 C30 cells were established by irradiating NR-S1 parental control cells with 30 Gy of carbon ion beam radiation at a rate of 5 Gy once every 2 weeks. The ME-180 X-ray–resistant cells were established by irradiating ME-180 parental control cells with 60 Gy of γ irradiation at a rate of 2 Gy at every passage. Cells were cultured for a week after the final irradiation and then used for the experiment.

Clonogenic survival assay

Cells were harvested with TrypLE Express (Thermo Fisher Scientific), seeded onto cell culture dishes, and incubated at 37°C under 5% CO2 for 2 hours. Subsequently, cells were irradiated with gamma irradiation using the Gammacell 40 Exactor (MDS Nordion) and incubated at 37°C under 5% CO2 for 7 to 13 days. Cells were then stained with 0.5% crystal violet (w/v) and counted. Colonies containing >50 cells were counted as cells that survived. The number of survived cells were plotted against the dose of gamma irradiation.

Image preparation and preprocessing

Cells were photographed using a phase-contrast microscope (BZ-X700; Keyence). For each cell type, 5,000 images at a resolution of 640 × 480 sq. pixels were captured. Two images of 320 × 320 sq. pixels were cropped from the original images, and the resolution of the images was changed to 160 × 160 sq. pixels to facilitate image processing (Fig. 2). Overall, 50,000 images were captured, with 10,000 images per cell type. The processed images were divided into 8,000 training images and 2,000 test images for each cell type.

Figure 2.

Collection of cell images. A, Rectangular images of 640 × 480 sq. pixels were cropped to square images of 320 × 320 sq. pixels and further reduced to 160 × 160 sq. pixels. B, Representative images of each cell line. Similar images of cells were obtained, and data were analyzed from a total of 5,000 images of each cell line. NRN, control parental NR-S1 cells; NRX, X-ray–resistant NR-S1 cells; NRC, carbon ion beam–radioresistant NR-S1 cells; MEN, control parental ME-180 cells; and MEX, X-ray–resistant ME-180 cells.

Figure 2.

Collection of cell images. A, Rectangular images of 640 × 480 sq. pixels were cropped to square images of 320 × 320 sq. pixels and further reduced to 160 × 160 sq. pixels. B, Representative images of each cell line. Similar images of cells were obtained, and data were analyzed from a total of 5,000 images of each cell line. NRN, control parental NR-S1 cells; NRX, X-ray–resistant NR-S1 cells; NRC, carbon ion beam–radioresistant NR-S1 cells; MEN, control parental ME-180 cells; and MEX, X-ray–resistant ME-180 cells.

Close modal

Neural network architecture and transfer learning

The CNN comprises multiple convolutional layers to detect the local features of inputs and pooling layers to reduce the computational burden, overfitting, and image shift. A pretrained CNN, namely VGG16, was used in this study; VGG16 has been published by the Visual Geometry Group of Oxford University (Oxford, United Kingdom) and has a high accuracy of image recognition (8, 13). Figure 3 presents the architecture of VGG16, including 13 convolutional layers and 3 fully connected layers. Although the original VGG16 outputs, that is, 1,000 parameters to categorize images in 1,000 classes, are defined in ImageNet, data in this study were categorized in only five classes: NRN, NRX, NRC, MEN, and MEX. Thus, the number of VGG16 outputs was changed from 1,000 to 5.

Figure 3.

Analysis of the CNN. The CNN, called VGG16, comprised 13 convolutional layers and three fully connected layers, including flattened and dense layers. The max pooling layers were inserted in convolutional layers. The CNN received each image as input data and output the probability of each of the five classes.

Figure 3.

Analysis of the CNN. The CNN, called VGG16, comprised 13 convolutional layers and three fully connected layers, including flattened and dense layers. The max pooling layers were inserted in convolutional layers. The CNN received each image as input data and output the probability of each of the five classes.

Close modal

In this study, only the last 3 convolutional layers and 3 fully connected layers were trained using training images. The training dataset was processed in minibatches. A minibatch was randomly selected from the training dataset at each training step. The learning of all 40,000 images was defined as one learning or one epoch. Also, cross-entropy error was evaluated using the output of the VGG16 model, and the actual class of images was evaluated using “accuracy” as a metric. Then, backpropagation was performed, and model parameters were updated using a momentum stochastic gradient descent algorithm with a learning rate of 0.0001 and a momentum of 0.9. Overall, 20 epochs were trained; one epoch implies completing learning once with all training images. For each epoch, test images were used to assess the accuracy of the trained model. Furthermore, Google's TensorFlow (14) deep learning framework and Keras (15) followed by the data library of TensorFlow were used to train, validate, and test the model.

Research ethics

We declare that we used cell lines only and no animal samples including humans and mice in this study.

Confirmation of the radioresistant phenotype of cell lines

We used mouse squamous cell carcinoma NR-S1 and human cervical carcinoma ME-180 cell lines to establish radioresistant cells. In this study, NR-S1 and its X-ray- and carbon ion beam–resistant cell lines were named “NRN,” “NRX,” and “NRC,” respectively. Similarly, ME-180 and its X-ray–resistant cell lines were named “MEN” and “MEX,” respectively. To confirm the radioresistant phenotype of cell lines, we performed a clonogenic survival assay and estimated the survival fraction of each cell line subjected to 4 and 8 Gy of γ irradiation (Supplementary Fig. S1). Both NRX and MEX exhibited higher survival than the corresponding control cell lines (NRN and MEN).

Image datasets

Phase-contrast microscopic images of cell lines were captured, and square images were cropped from the original rectangular images. From a total of 50,000 images, we obtained 10,000 microscopic images for each cell line at a resolution of 160 × 160 sq. pixels. Representative images are shown in Fig. 2. Cells of NRX were marginally smaller than those of NRN. However, differences between the size of NRC and NRN and between MEN and MEX were ambiguous. Also, we divided the image datasets into 8,000 training images and 2,000 test images; training images were used to train the neural network, and test images were used to validate the accuracy of the neural network and feature engineering.

Training and validating the CNN

In this study, we used the CNN, called VGG16 (Fig. 3), which was pretrained using ImageNet datasets (13). Using the pretrained model, the number of images required for learning was reduced, thus improving the learning speed, a process called transfer learning. We performed transfer learning to optimize only the last 3 convolutional layers and 3 fully connected layers using training data and trained the model for 20 epochs. Figure 4 shows the training course of each epoch. The use of the pretrained model resulted in a dramatic improvement in the accuracy of the model after only one epoch. Using test images, the accuracy of the model reached approximately 96% after one epoch but plateaued thereafter. The accuracy of the model was 99.9%, 98.8%, 99.8%, 98.7%, and 91.1% for NRN, NRX, NRC, MEN, and MEX, respectively, using test images. Although the accuracy of the classification of NR-S1 cell lines was high, it was difficult to classify the ME-180 cell lines, especially MEX, using this model. We performed Receiver Operating Characteristic analysis and calculated the area under the curve (AUC) to predict the trained VGG16 model. The AUC values of NRN, NRX, NRC, MEN, and MEX were 1.00000, 0.99991, 0.99978, 0.99793, and 0.99908, respectively. Overall, the trained VGG16 model showed a very high performance in distinguishing each cell line.

Figure 4.

The accuracy of each epoch. Training and test images of radioresistant and parental cells were analyzed. The accuracy of CNN's prediction of each epoch was plotted as a line graph. The accuracy of training and test images is indicated with broken and solid lines, respectively.

Figure 4.

The accuracy of each epoch. Training and test images of radioresistant and parental cells were analyzed. The accuracy of CNN's prediction of each epoch was plotted as a line graph. The accuracy of training and test images is indicated with broken and solid lines, respectively.

Close modal

Elucidating the reasons behind predictions

The “Local Interpretable Model-agnostic Explanations” (LIME) method (16) was used to determine the area of emphasis of the trained CNN on the image. Using the LIME method on some test images, we visualized the bases in the images for the classification of the trained CNN. Representative results are shown in Supplementary Fig. S2. We successfully visualized the information emphasized by the trained CNN, suggesting that the CNN registered the shape of the cell or cell population because the boundary of the extracted region was along the edge of the cell.

Next, we extracted internal features from test images using the trained VGG16 model to assess the basis for the categorization of these images. We obtained the output of the last hidden layer of the trained VGG16. The CNN designed in this study changed an input image to 512 feature maps, comprising 5 × 5 square data, using convolutional layers. In addition, it integrated feature maps into 4,096 features by fully connected layers to render features useful for categorization into five classes. Although the 5 × 5 feature maps facilitated the visualization of data extracted from an image by the CNN, these maps were too many and too small to comprehend. Hence, 4,096 multivariate inputs were used for the final layer as features extracted from the image. Moreover, maps were reduced to two dimensions using t-distributed stochastic neighbor embedding (t-SNE) to visualize the features of 4,096 multivariate inputs (17). We created a scatter plot with these features (Fig. 5). Although each point represented a microscopic image of one cell line, each color represented a type of cell line (red, NRN; blue, NRX; black, NRC; orange, MEN; and cyan, MEX). We observed five clusters of points with the same categories. Three clusters of NR-S1 (NRN, NRX, and NRC) were distinct from each other, whereas clusters of ME-180 (MEN and MEX) were distributed relatively close together, implying that the VGG16 model recognized MEN and MEX as similar cell lines.

Figure 5.

CNN analysis of radioresistant and parental cell lines. Features were extracted from 2,000 test images of each cell line. Each point represents features obtained from a single image. Data from the mouse squamous cell carcinoma NR-S1 cell line (control; NRN) and its X-ray–resistant (NRX) and carbon ion beam–radioresistant (NRC) cells are shown in red, blue, and black, respectively. Data from the human cervical carcinoma ME-180 cell line (control; MEN) and its X-ray–resistant (MEX) cells are shown in cyan and orange, respectively.

Figure 5.

CNN analysis of radioresistant and parental cell lines. Features were extracted from 2,000 test images of each cell line. Each point represents features obtained from a single image. Data from the mouse squamous cell carcinoma NR-S1 cell line (control; NRN) and its X-ray–resistant (NRX) and carbon ion beam–radioresistant (NRC) cells are shown in red, blue, and black, respectively. Data from the human cervical carcinoma ME-180 cell line (control; MEN) and its X-ray–resistant (MEX) cells are shown in cyan and orange, respectively.

Close modal

This study established that it is feasible for AI to accurately differentiate between cancer cell lines and their radioresistant clones, even if simple phase-contrast microscopic images are used as inputs, suggesting clear visual differences among multiple types of cells with different properties. These data suggest the potential to create universal AI using a variety of cells for learning. The distinction among the three types of NR-S1 cells was more accurate than that between the two types of ME-180 cells, which was consistent with our visual intuition. Among all cell types, MEX cells were the least accurate. The trained VGG16 model identified some photographs of MEX as MEN possibly because the MEX cells were very similar to MEN cells, and the VGG16 set the distinction threshold to be strict for MEX. The LIME method was used to illustrate the predictions of any classifier by learning an interpretable model locally around the prediction. We attempted to elucidate the visual basis for the classification of images by the CNN and extracted features using LIME or t-SNE; however, the results were unclear. Thus, it remains unclear whether AI could predict radiosensitivity using microscopic images of cell lines. In this study, only two kinds of cells were available, as the establishment of radioresistant clones of cancer cells requires considerable time and effort. Further investigations are needed to train the CNN with a higher number of cell lines and to verify its ability to predict radiosensitivity using cell lines not used for training.

Although the CNN is an excellent technique in the field of image recognition, several problems await resolution. For example, there exists a black box in the learning processes and extracted features associated with deep learning, such as the CNN. Whether prediction using deep learning is correct and has practical implications warrants further investigation. Also, since no mathematical support exists in modifying hyperparameters to enhance the result of deep learning, there is a need to explore optimal parameters by using random sampling or grid search. Optimization of hyperparameters using a machine learning approach such as Bayesian optimization may resolve this problem (18).

There have been several reports on the efficacy of image recognition using deep learning in the diagnosis of existing lesions and the qualitative diagnosis of tissue type (1–3, 5). However, much less work has been conducted on the prediction of the sensitivity of treatment such as radiotherapy using radiological or pathologic images. In the future, using big data training AI with more information will advance predictive medicine, including the prediction of treatment effect, and contribute to the realization of personalized medicine.

T. Kudo reports receiving a commercial research grant from Yakult Honsha Co., Ltd., Chugai Pharmaceutical Co., Ltd., and Ono Pharmaceutical Co. Ltd. H. Ishii reports receiving a commercial research grant from Taiho Pharmaceutical Co. Ltd, Unitech Co. Ltd. (Chiba, Japan), IDEA Consultants Inc. (Tokyo, Japan), and Kinshu-kai Medical Corporation (Osaka, Japan). No potential conflicts of interest were disclosed by the other authors.

Conception and design: M. Toratani, M. Konno, M. Mori, K. Ogawa, H. Ishii

Development of methodology: M. Toratani, M. Konno, A. Asai, J. Koseki, K. Tamari, D. Sakai, D. Motooka, H. Ishii

Acquisition of data (provided animals, acquired and managed patients, provided facilities, etc.): M. Toratani, M. Konno, A. Asai, K. Kawamoto, Z. Li, T. Kudo, K. Sato, D. Okuzaki, H. Ishii

Analysis and interpretation of data (e.g., statistical analysis, biostatistics, computational analysis): M. Toratani, M. Konno, A. Asai, J. Koseki, Z. Li, D. Sakai, T. Satoh, K. Sato, K. Ogawa, H. Ishii

Writing, review, and/or revision of the manuscript: M. Toratani, M. Konno, T. Satoh, M. Mori, H. Ishii

Administrative, technical, or material support (i.e., reporting or organizing data, constructing databases): M. Konno, J. Koseki, D. Sakai, Y. Doki, H. Ishii

Study supervision: H. Ishii

We thank the laboratory staff for their helpful discussions. This work received financial support from grants-in-aid for Scientific Research from the Japan Agency for Medical Research and Development and the Ministry of Education, Culture, Sports, Science, and Technology (grant nos. 17H04282 and 17K19698 to H. Ishii), grant no. 16K15615 to M. Konno, and grant no. 15H05791 to M. Mori.

The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.

1.
Esteva
A
,
Kuprel
B
,
Novoa
RA
,
Ko
J
,
Swetter
SM
,
Blau
HM
, et al
Dermatologist-level classification of skin cancer with deep neural networks
.
Nature
2017
;
542
:
115
8
.
2.
Kainz
P
,
Pfeiffer
M
,
Urschler
M
. 
Segmentation and classification of colon glands with deep convolutional neural networks and total variation regularization
.
PeerJ
2017
;
5
:
e3874
.
3.
Aubreville
M
,
Knipfer
C
,
Oetter
N
,
Jaremenko
C
,
Rodner
E
,
Denzler
J
, et al
Automatic classification of cancerous tissue in laserendomicroscopy images of the oral cavity using deep learning
.
Sci Rep
2017
;
7
:
11979
.
4.
Lao
J
,
Chen
Y
,
Li
Z-C
,
Li
Q
,
Zhang
J
,
Liu
J
, et al
A deep learning-based radiomics model for prediction of survival in glioblastoma multiforme
.
Sci Rep
2017
;
7
:
10353
.
5.
Teramoto
A
,
Tsukamoto
T
,
Kiriyama
Y
,
Fujita
H
. 
Automated classification of lung cancer types from cytological images using deep convolutional neural networks
.
Biomed Res Int
2017
;
2017
:
4067832
.
6.
Zhen
X
,
Chen
J
,
Zhong
Z
,
Hrycushko
B
,
Zhou
L
,
Jiang
S
, et al
Deep convolutional neural network with transfer learning for rectum toxicity prediction in cervical cancer radiotherapy: a feasibility study
.
Phys Med Biol
2017
;
62
:
8246
63
.
7.
Krizhevsky
A
,
Sutskever
I
,
Hinton
GE
. 
ImageNet classification with deep convolutional neural networks
.
Proceedings of the 25th International Conference on Neural Information Processing Systems
2012
;
1
:
1097
105
.
8.
Russakovsky
O
,
Deng
J
,
Su
H
,
Krause
J
,
Satheesh
S
,
Ma
S
, et al
ImageNet large scale visual recognition challenge
.
Int J Comput Vis
2015
;
115
:
211
52
.
9.
LeCun
Y
,
Bottou
L
,
Bengio
Y
,
Haffner
P
. 
Gradient-based learning applied to document recognition
.
Proc of the IEEE
1998
;
86
:
2278
324
.
10.
Pan
SJ
,
Yang
Q
. 
A survey on transfer learning
.
IEEE Trans Knowl Data Eng
2010
;
22
:
1345
59
.
11.
Sato
K
,
Imai
T
,
Okayasu
R
,
Shimokawa
T
. 
Heterochromatin domain number correlates with X-Ray and carbon-ion radiation resistance in cancer cells
.
Radiat Res
2014
;
182
:
408
19
.
12.
Sato
K
,
Azuma
R
,
Imai
T
,
Shimokawa
T
. 
Enhancement of mTOR signaling contributes to acquired X-ray and C-ion resistance in mouse squamous carcinoma cell line
.
Cancer Sci
2017
;
108
:
2004
10
.
13.
Simonyan
K
,
Zisserman
A
. 
Very deep convolutional networks for large-scale image recognition
; 
2015
. Available from: https://arxiv.org/abs/1409.1556v6.
14.
Abadi
M
,
Agarwal
A
,
Barham
P
,
Brevdo
E
,
Chen
Z
,
Citro
C
, et al
TensorFlow: large-scale machine learning on heterogeneous distributed systems
; 
2016
. Available from: https://arxiv.org/abs/1603.04467v2.
15.
Chollet
F
. 
Keras
.
GitHub
; 
2015
. https://github.com/fchollet/keras.
16.
Tulio Ribeiro
M
,
Singh
S
,
Guestrin
C
. 
"Why should I trust you?": explaining the predictions of any classifier
; 
2016
. Available from: https://arxiv.org/abs/1602.04938v3.
17.
Van Der Maaten
L
,
Hinton
G
. 
Visualizing data using t-SNE
.
J Mach Learn Res
2008
;
9
:
2579
605
.
18.
Lee
J
,
Bahri
Y
,
Novak
R
,
Schoenholz
SS
,
Pennington
J
,
Sohl-Dickstein
J
, et al
Deep neural networks as Gaussian processes
; 
2017
. Available from: https://arxiv.org/abs/1711.00165v3.

Supplementary data