A Proposal Regarding Reporting of In Vitro Testing Results

Preclinical and translational research, both from academia and from pharmaceutical companies, is under scrutiny. On the one hand, translational research conducted by academic researchers and evaluated by pharmaceutical companies has been found to often lack reproducibility, and claimed conclusions have not been supported by further independent testing (1, 2). At the same time, preclinical testing conducted by pharmaceutical sponsors has been found lacking, as many agents with preclinical packages considered promising have subsequently failed in definitive clinical trials.

There are surely a number of potential explanations for this discouraging perspective on the use of preclinical testing. In this commentary, we address one specific research setting, that of in vitro testing of agents that are advanced in clinical testing and for which clinically relevant drug exposures are established. This is a common setting for academic investigators who use these in vitro results to help support proposals to funding agencies and pharmaceutical companies for clinical trials of these agents. Similar in vitro data are used to support the “repurposing” of nononcology drugs for cancer indications.

The example of sorafenib highlights some of the key problems related to in vitro testing of anticancer agents that have advanced into clinical testing, and so we use it to illustrate uses and misuses of in vitro testing data. A cottage industry has developed around studying the in vitro effects of sorafenib at micromolar level concentrations, with scores of papers describing its effects against a variety of cancer cell lines at these concentrations and with no indication that an end to such publications is near. There are advantages to studying sorafenib in vitro at micromolar level concentrations, as at these concentrations, it consistently kills cancer cell lines and induces a number of biologically interesting effects on signaling pathways associated with cancer. For example, sorafenib at micromolar concentrations reduces phosphorylation levels of eIF4E (3, 4), reduces levels of Mcl-1 (4–6), induces STAT3 activation (7), promotes autophagy (8–10), and inhibits mitogen-activated protein kinase signaling (11). Unfortunately, the effects of sorafenib at these concentrations have little or no relevance to the clinical setting, for the reasons summarized in the paragraphs below.

Sorafenib shows remarkable consistency in killing cancer cell lines at concentrations in the 1 to 10 μmol/L range. This activity is unrelated to histotype and spans a wide range of adult and pediatric cancer cell lines, including cell lines for hepatocellular carcinoma, papillary thyroid cancer with BRAF mutations, breast cancer, lung carcinoma, cholangiocarcinoma, melanoma, neuroblastoma, medulloblastoma, rhabdomyosarcoma, and various types of leukemias (see Supplementary Materials for references). The uniformity of sorafenib cytotoxic effects at micromolar levels is illustrated by the results of the Pediatric Preclinical Testing Program (PPTP), which showed that 21 of 23 cell lines derived from a range of childhood cancers had IC₅₀ values ranging from 3.2 to 10 μmol/L (median 4.3 μmol/L), with only a single cell line not achieving IC₅₀ by 10 μmol/L. This uniformity creates a sense of unease, as it is the type of response associated with nonspecific cytotoxic agents as well as the type of response observed at supraphysiologic concentrations for many kinase inhibitors. This unease increases with the observation that one cell line tested by the PPTP, an acute myelogenous leukemia (AML) cell line with an activating KIT mutation, showed an IC₅₀ of 20 nmol/L. Other cell lines with certain KIT-activating mutations are also inhibited by sorafenib in the 100 nmol/L range (12). These observations support the position that the targeted effects of sorafenib occur in the low nanomolar range, whereas effects in the micromolar range represent nonspecific effects on a range of biologic processes that lead to cell death.

A body of published data documents that the micromolar concentrations of sorafenib required to kill most cancer cell lines exceed by 2 to 3 logs the concentrations at which sorafenib inhibits cell lines with activated kinases that are its more specific molecular targets. For example, sorafenib inhibits FLT3-ITD autophosphorylation with IC₅₀ values in the low nanomolar range (13, 14), and AML cell lines with FLT3-ITD show low nanomolar IC₅₀ values (14, 15). Cell lines with oncogenic RET show IC₅₀ values in the low to mid nanomolar range (16), and cell lines driven by selected KIT-activating mutations show IC₅₀ values less than 100 nmol/L (12, 17). Inhibition of VEGF receptor 2 (VEGFR2) signaling is also observed at submicromolar concentrations (11). Cell lines driven by platelet-derived growth factor receptor (PDGFR) signaling have IC₅₀ values in the 10 to 100 nmol/L range (15, 18), with concentrations of 100 nmol/L providing virtually complete inhibition of phospho-PDGFRA and phospho-PDGFRB expression (15, 18). This activity of sorafenib at low nanomolar concentrations translates into tumor regressions in xenograft models with activating mutations in sorafenib target gene products, as exemplified by the high level of in vivo activity of sorafenib against FLT3-ITD AML xenografts (14). In contrast, xenografts lacking activating genomic alterations in these target genes show slowing of tumor growth (consistent with inhibition of VEGFR2 signaling), but not tumor regressions (17).

Importantly, the in vitro activity of sorafenib at these nanomolar level concentrations has clear clinical significance, which is illustrated by the activity of sorafenib against FLT3-ITD AML (19–22) and its VEGFR2-related antiangiogenic activity, as evidenced by its U.S. Food and Drug Administration approval for renal cell carcinoma (a VEGF pathway–driven tumor for which VEGFR2-targeted agents show class activity; ref. 23). The rarity of PDGFR-driven leukemias and the effectiveness of imatinib as upfront therapy have limited evaluations of sorafenib against this molecularly defined subset of patients (24, 25), although 1 patient showed a brief hematologic response before outgrowth of a resistant mutation (24).

Conversely, clinical experience with sorafenib supports the lack of positive predictive power for in vitro results obtained at micromolar concentrations. Low to nil objective response rates for sorafenib have been observed for multiple tumor types (most of which have cell lines showing in vitro sensitivity at micromolar concentrations), including the following: non–small cell lung cancer, breast cancer, melanoma, prostate cancer, squamous cell carcinoma of the head and neck, biliary tract carcinoma, and uterine carcinoma (see Supplementary Materials for references). Sorafenib response rates for sarcomas were also low, although 5 of 37 patients with angiosarcoma did show responses (26). This tumor type has shown responsiveness to other VEGFR2-targeted agents (27). The paucity of objective responses to sorafenib for most cancers in which the agent has been studied (outside of cancers with the activated kinases described above) despite in vitro cytotoxic activity for sorafenib at micromolar concentrations against cell lines derived from these same cancers supports the lack of clinical relevance for sorafenib in vitro effects observed at micromolar concentrations in 10% serum conditions.

Given the significant clinical activity of sorafenib that can be related to its in vitro activity at nanomolar concentrations against cell lines dependent upon its target kinases and given the otherwise limited clinical activity, what has maintained enthusiasm within the research community for studying sorafenib against cell lines that show no effect until micromolar concentrations are reached? One factor is the clinically achievable sorafenib drug levels, which are indeed in the 10 μmol/L range (28). However, sorafenib is highly protein bound (99.7%; ref. 29), meaning that sorafenib concentrations that are effective in 10% FBS are ineffective in plasma conditions. For example, FLT3-ITD–driven cell lines are inhibited by 50% at 3 nmol/L sorafenib in 10% FBS but require more than 100-fold higher sorafenib concentrations (approximately 500 nmol/L) in plasma for a comparable level of inhibition (13). Because clinically achievable sorafenib levels are only in the 10 μmol/L range, sorafenib effects observed in vitro in 10% FBS at 1 to 10 μmol/L concentrations lack plausibility for successful clinical translation.

Evaluation of sorafenib at micromolar concentrations is extending into “next generation” in vitro testing. The Cancer Cell Line Encyclopedia project evaluated sorafenib at concentrations ranging from 2.5 nmol/L to 8.0 μmol/L, and 98% of the 504 tested cell lines showed IC₅₀ values more than 1 μmol/L (30). For the reasons outlined above, these data at higher sorafenib concentrations lack clinical relevance. Analyses that incorporate these data to identify insights into sorafenib mechanism of action or mechanism of synergy against human cancers will produce misleading findings regarding potential clinical applications of sorafenib.

Disconnects between in vitro observations and clinical reality extend beyond the realm of kinase inhibitors. For example, vorinostat inhibits the growth of most cancer cell lines, but with IC₅₀ values generally exceeding 1 μmol/L (31–33). Concentrations of vorinostat from 2.5 to 10.0 μmol/L are commonly used to show the in vitro effects of vorinostat on cellular processes related to its enhancement of protein acetylation. However, maximum achievable drug concentrations in humans are in the 1 to 2 μmol/L range at the standard vorinostat dose of 400 mg (34–37). Because of the short half-life of vorinostat (1.5 to 2.0 hours), concentrations less than 1 μmol/L are maintained for at most a few hours of each treatment day with concentrations less than 0.1 μmol/L for most of each treatment day (36, 38). Thus, the clinical relevance of the large body of literature on vorinostat's in vitro effects with prolonged exposure to concentrations exceeding 1 to 2 μmol/L is highly questionable, and indeed, 3- or 6-hour exposure to 1 to 10 μmol/L vorinostat in vitro fails to elicit significant growth inhibition, and exposure for 24 or more hours is required for significant effects (39). This skepticism about the clinical relevance of in vitro results using micromolar concentrations of vorinostat is supported by the disconnect between the broad in vitro cytotoxic activity of vorinostat and its narrow range of therapeutic activity in the clinic (primarily cutaneous T-cell lymphoma; ref. 37). In fairness to vorinostat, other histone deacetylase (HDAC) inhibitors show similar relationships between concentrations that are effective in vitro and those studied in the clinic. Panobinostat, which was characterized as a “broadly active compound” in a report describing the Cancer Cell Line Encyclopedia, showed an IC₅₀ of approximately 60 nmol/L against the approximately 500 cell lines against which it was tested (30). However, at the commonly used dose and schedule of this agent (20 mg administered orally 2–3 times per week), C_max values are only in the 40 to 70 nmol/L range (40–42), and concentrations above 15 nmol/L are maintained for less than 12 hours of every 48-hour dosing interval (40). Of note, in vitro exposure to panobinostat for 12 hours or less produces limited in vitro effects (39). The lack of clinical relevance for these nanomolar range IC₅₀ values for panobinostat is illustrated by its low single-agent activity for patients with multiple myeloma (42), despite myeloma cell lines showing a median IC₅₀ value of approximately 15 nmol/L (30).

In vitro testing results are also being used to support “repurposing” of nononcology drugs for anticancer indications. Since the observation that the antidiabetic agent metformin could reduce the overall incidence of cancer in diabetics and reduce the risk of pancreatic and other cancers (43–45), a plethora of reports have appeared showing the biochemical effects of metformin, and effects on inhibition of proliferation, survival, invasion, motility, DNA repair, and synergy with cytotoxic agents in vitro. A review of recent publications through PubMed reveals that the vast majority of these studies used metformin at 1 to 20 mmol/L (usually > 5 mmol/L; see Supplementary Materials for references). The concern about the value of such in vitro studies is not because metformin binds to plasma proteins (it has very low or negligible binding), but that accumulation in cells is mediated by organic cation transporters [(OCT) SLC22A1, SLC22A2, or SLC22A3; refs. 46, 47]. These transporters are highly restricted to expression in liver and kidney, and are dramatically reduced during malignant transformation, even in hepatocellular carcinoma (48). Population pharmacokinetics modeling in humans shows that plasma levels of metformin are low, with median C_max of approximately 10 μmol/L [95% confidence interval (CI): 2.8–21.6 μmol/L] following an oral dose of 663 mg (49). Some biochemical effects of metformin have been shown at 50 to 100 μmol/L, under conditions of physiologic glucose, in vitro; however, these effects are relatively small (50), and even these exposures to metformin exceed those attainable in patients. Although it is clear that metformin can retard progression of some cancers in rodents, it is unlikely that the mechanism of action is a consequence of direct effects on tumor cell metabolism. The antitumor effects are more likely via indirect effects on gluconeogenesis or secretion of insulin-like growth factors from liver (51). Thus, biochemical changes or physiologic changes (proliferation/survival) observed in vitro at metformin concentrations in the millimolar range seem irrelevant for understanding the in vivo activity of metformin.

A basic challenge in interpreting in vitro testing results is the difficulty in assessing therapeutic window. The obvious point is that 96- and 386-well plates do not develop mucositis, neutropenia, or diarrhea, making it virtually impossible to assess whether the concentration of a novel agent showing in vitro activity is clinically relevant. Uniform cytotoxic activity, as shown for sorafenib at micromolar concentrations, can imply that the agent at these concentrations is extraordinarily promising or alternatively that it is a nondiscriminant cytotoxic agent at these concentrations with no greater therapeutic relevance than Clorox. Attempts at using nontransformed cell lines as a surrogate for the host have not been effective at assessing therapeutic window. The one consistent in vitro predictive factor for clinical success for a candidate oncologic agent is wide variation in sensitivity across cell lines, with most cell lines requiring high concentrations and with a select few cell lines with distinctive molecular characteristics being sensitive at much lower concentrations. Examples of the successful application of this principle include small-molecule kinase inhibitors that show specificity for BRAF V600E-mutated cell lines, ALK-translocated cell lines, or EGFR-mutated cell lines, to name a few.

There are costs to publishing results claiming anticancer activity based on clinically unachievable exposures, the most important being the potential for misallocation of clinical research resources and patients to clinical trials with little or no chance of success. In reference to sorafenib again, more than 500 clinical trials for sorafenib are listed in ClinicalTrials.gov, and nearly 200 clinical trials are listed as currently active across a wide range of cancer types. Although many of these sorafenib trials are for either renal cell carcinoma or hepatocellular carcinoma (cancers for which sorafenib is licensed by regulatory agencies) or for cancers for which an antiangiogenic effect is targeted, other trials are for cancers for which the justification is based in part on sorafenib's micromolar level in vitro activity. Similarly, approximately 50 cancer-related metformin clinical trials are listed. Would these trials have been initiated if there had been greater scrutiny of the clinical relevance of the drug concentrations used in the in vitro testing experiments that contributed to the rationale for these clinical trials?

What can be done to address the issues described above? One step that journals can take is to reject without review manuscripts that make activity claims for sorafenib at concentrations exceeding 1 μmol/L, metformin at concentrations exceeding 50 μmol/L, and vorinostat at concentrations exceeding 1 μmol/L. More general steps that journal editors and reviewers can take include the following:

• - Requiring that manuscripts describing the in vitro activity of agents that have advanced into clinical evaluation specifically address and provide convincing rationale with appropriate reference to the published literature for the potential clinical relevance of the concentrations explored.

• - Rejecting manuscripts that make activity claims for targeted agents at concentrations that substantially exceed (e.g., ≥ 10-fold) the concentrations at which the agents are effective against cell lines expressing the agents' molecular targets.

• - Rejecting manuscripts that make activity claims for agents based on “clinically achievable drug levels,” when the concentrations used in vitro either exceed those observed in humans or else are present for only a short period of each treatment day.

• - Requiring that manuscripts describing the in vitro testing of agents with high protein binding adjust for low serum versus plasma protein concentrations whenever comparisons to clinically achievable concentrations are made.

We recognize that exceptions to the above “rules” will be justifiable in some cases (e.g., for agents that have active metabolites that achieve sufficiently high levels in patients), but authors proposing exceptions should provide a clear rationale to justify their position. Importantly, we believe that these proposals, if consistently applied, can help direct investigators away from lines of research that lack clinical relevance and toward lines of research that are more likely to lead to positive clinical trials and to improved treatments for patients with cancer.

No potential conflicts of interest were disclosed.

Conception and design: M.A. Smith, P. Houghton

Development of methodology: M.A. Smith

Writing, review, and/or revision of the manuscript: M.A. Smith, P.Houghton