Predicting Cancer Prognosis Using Interactive Online Tools: A Systematic Review and Implications for Cancer Care Providers

Patient–provider communication in oncology often involves conveying a large amount of highly complex information (1). Cancer prognosis is one of the leading topics of interest for patients with cancer, their caregivers and providers, and is relevant at all stages of the cancer continuum (2–5). Prognostic information can support decision-making and communication around therapeutic and palliative treatment decisions, management of comorbid conditions, palliative care, and decisions regarding prioritization of management of other chronic conditions and other life events (2, 6, 7). Information about prognosis can also address psychosocial needs of patients with cancer and their caregivers, including uncertainty and empowering the patient and family to participate in the decision-making process and help patients understand how behavior changes may impact prognosis (2, 6, 8, 9).

Many compelling challenges arise when discussing prognosis with patients with cancer. Among these are uncertainties regarding the integrity of the data upon which the prognosis is based, individual variation in prognosis and response, which resources or tools to use to estimate prognosis, the timing of when to discuss prognosis given a patient's informational, cognitive, and social support needs, how best to frame and present prognostic probabilities so that the patient interprets probabilities correctly, given health literacy and numeracy challenges, and the time required to convey prognostic information to patients. Clear and understandable communication by a provider also requires communication skills and the willingness of the health care provider to engage in this complex information exchange with patients, many of whom are facing the most profound crisis of their lives (2, 10–12). For providers, the communication of prognosis about cancer is complex and emotional and can involve the use of qualitative statements (e.g., good chances or not so great chances of survival), comparison with peers (more or less likely to die from cancer than peers), and actual numbers (e.g., percentages, fractions, or relative or absolute risk probabilities over a given period of time).

Providers can use a number of resources to make cancer prognostic estimations, including their own experience and intuition, clinical data from scientific publications (especially clinical trials), guidelines such as those published by the National Comprehensive Cancer Network as well as textbooks, look-up tables, and nomograms (13, 14). Literature suggests that nomograms that use prognostic algorithms integrating several predictors improve prognostic accuracy (15, 16). More recently, there has been increasing interest in the availability and use of web-based tools by both health care providers and patients for obtaining information about health promotion and disease management (17–20). This is especially evident in the emergence of web-based prognostic tools for cancer. Cancer prognostic tools are designed to support decision-making including treatment decisions through providing cancer outcome predications that take diverse patient characteristics into account. These web-based tools, the majority of which are also available to patients, have not been systematically described or reviewed, nor their use in real world settings often described. Furthermore, clear criteria for the development of an evaluation of decision aids only exist for patient-facing tools. No guidelines have been developed to assess the quality of provider-facing decision aids (21). The goals of this study were to review cancer prognostic tools available online, describe their content and format, and consider the implications for use of these tools for different purposes by cancer care providers.

Interactive cancer prognostic tools were identified using three approaches. First, the Internet search engine Google was used with a combination of search terms describing cancer (i.e., cancer, leukemia, lymphoma, carcinoma, malignancy, hematologic malignancy, and melanoma), prognosis (i.e., prognosis, survival, predictive, and prediction), and tool (i.e., tool, calculator). For each search, we reviewed the first 10 pages of results for relevant tools (100 per search term for a total of 5,600 search results). Second, we searched Medline for peer-reviewed publications from 1996 through July 2011 using the same search terms as in the web search. Titles and abstracts and when necessary, full text articles were reviewed to identify additional prognostic tools. Third, we sought input from cancer specialists regarding any existing or emerging prognostic tools that they might be aware of or use in their own practice.

Tools identified through one of the three search approaches were then assessed for eligibility. To be eligible, tools had to have an English version, a focus on cancer, have an interactive component (i.e., data or information entered by the clinician or patient is manipulated via an algorithm that draws on a dataset to produce prognostic estimates), and provide output measures. The following output measures were part of the inclusion criteria: (i) cancer- or noncancer-specific mortality/overall survival; (ii) disease-free survival (DFS) or progression-free survival (PFS); (iii) clinical response to treatment; or (iv) cancer therapy-induced side effects. Tools focusing solely on prevention and the risk of developing cancer were not included in this review.

Some of the tools were not publicly available. To access these tools we contacted the developers of each tool via email and/or telephone. We made a minimum of three attempts to gain access via the developers. Tools for which we were unable to obtain access were excluded from this analysis.

We developed and refined an abstraction protocol based on existing literature on patient-centered communication, decision aids, usability testing and input from providers, and cancer communication and eHealth researchers (6, 22). The abstraction protocol was then pilot tested by three abstractors (B.A. Rabin, T. Sanders, and B. Gaglio) and further refined. Domains for the abstraction included (i) general characteristics and purpose of the calculator i.e., goal, cancer sites (e.g., breast, prostate, etc.), cancer stage, and patient population best served by the tool, intended users (including patient user), and disclaimer for users; (ii) data (i.e., algorithm, validation, data source, and publications); (iii) input factors (i.e., patient demographic and genetic characteristics, health status/comorbidities, modifiable risks, impact of therapy, and adjuvant treatment); (iv) output measures (i.e., prognostic measures, side effects, quality of life, recurrence, and spread); and (v) website specifications (i.e., access, funding source, and developer). Additional domains of presentation of results, usability, and patient centeredness were also abstracted and will be summarized in a separate report. The complete abstraction protocol is available upon request from the first author.

All three abstractors coded the same six calculators (emerging from two tools) separately. After review of the initial attempt, the same calculators were reviewed again by the abstractors. These two rounds resulted in acceptable consistency. Then, each abstractor reviewed and abstracted information on a set of tools individually. Issues that arose during the abstraction were resolved by the three abstractors using a consensus approach.

Thirty-six interactive cancer prognostic tools were identified through the three search approaches. We were unable to obtain access to six tools. Eight tools did not meet our inclusion criteria largely due to lack of interactive component (n = 5) or lack of focus on desired outcome measures (n = 3). Thus, we were able to review and abstract information on 22 tools (See Fig. 1). There was a high degree of variability with regard to the number of embedded calculators in each tool (a total of 107 calculators; minimum: 1, maximum: 22) and the tools addressed a total of 89 different cancers (minimum: 1, maximum: 84). We classified unique cancer sites under 13 main categories. A list of abstracted tools, their brief name developed for the purpose of this article (and used throughout the article to refer to the tools), the number of included calculators and main cancer categories included are summarized in Table 1. A full list of calculators, the name of the developer/developing institution and their web address is provided in Table 2.

Some of the identified tools were cancer-specific tools (prostate, n = 6 and breast, n = 2), although about half of the tools (n = 12% or 55%) addressed multiple cancer sites. Overall, tools focused on prostate (n = 11), colorectal (n = 10), breast (n = 8), and melanoma (n = 6), although at least one tool was identified for most malignancies. Other genitourinary cancers were also commonly evaluated (n = 6). Characteristics of each tool are summarized in Table 3 and are discussed in the next section. Cancer site–specific descriptions of the tools are provided in Supplementary Tables S1.1–S1.13. Some of the tools included multiple calculators for the same cancer site focusing on different outcomes. To facilitate the review of the tools, we combined information from these calculators under the tool's cancer-specific summary in the Supplementary Tables.

There was wide variation in factors that were accounted for in the prognosis estimates. We classified these factors as cancer characteristics, demographic characteristics, comorbidities, therapy impact, genetic characteristics, and modifiable risk factors (see Table 3). The cancer characteristics category referred to any clinical and pathologic results that are associated with the patient's cancer diagnosis (e.g., tumor size, nodal status, grade, histology, etc.). All abstracted tools included information about at least one cancer characteristic. Demographic characteristics (e.g., age, gender, marital status, etc.) were accounted for in most (n = 18 or 82%) tools. In contrast, only nine tools included noncancer comorbidities or health status in their estimates. There was variability in how these factors were captured; for example, using the health status of the patient (good, bad, etc.; Adjuvant) versus a complex comorbidity calculator that takes into account 16 itemized conditions (CSQS). The differential impact of therapy on prognosis was accounted for in at least one calculator of 15 tools. However, there was also variation in how therapy was captured. Some tools' inputs included detailed information about specific chemo- or radiotherapy agents and dose, whereas, others were more simple (e.g., chemotherapy: yes–no).

Only five tools took genetic/genomic characteristics into account. This included information about breast cancer receptor status [estrogen receptor (ER) in adjuvant, ER/progesterone receptor/HER-2 in Finn Prog] and antigen Ki67 for mantel cell lymphoma (QxMD). Furthermore, one tool (MSK) assessed for an ovarian cancer-specific indicator (hereditary breast and ovarian cancer, HBOC syndrome). Modifiable risk factors were also very sparsely factored into the calculations of these tools (only n = 2% or 9%) and solely focused on smoking status (lung cancer calculator for MAASTRO and bladder cancer calculator for Nomograms).

The most common predicted outcome was cancer-specific survival/mortality (n = 17) followed by DFS/PFS (n = 16). A few tools also included noncancer-specific survival (n = 4), clinical response to treatment (n = 2), and therapy-induced side effects (n = 2) as outcomes. No tool used quality of life or quality adjusted life years as one of its outcomes.

More than half of the tools (n = 12) reported on multiple outcomes. Outcomes were reported for one or multiple timeframes, most commonly 1 and 5 years. Format for presenting prognosis varied, with half of the tools presenting only numerical presentation (i.e., ANN, Biochemical, Capra, EBRT, GBM, Han, Lerner, Nomograms, Numeracy, QxMD, and Roswell). The remainder used a combination of numerical and graphical display.

Among the stated purposes abstracted from the respective websites, the tools were primarily designed to support decision-making about treatment. However, the tools were also intended to be used the following treatment completion. Most of the abstracted cancer prognostic tools were designed with cancer specialists or physicians as exclusive users. Only four tools (i.e., BioChemical, GBM, MSK, and Nomograms) mentioned patients as potential users, but recommended patients to discuss results with their cancer specialists. None of the tools stated intent was to support care transitions or were specifically design for the primary care context.

Tools were classified on the basis of their primary sources of data as population-based (i.e., underlying data is population-based dataset, often a national registry) or clinic-based (i.e., underlying data is based on a patient population from one or multiple health care settings and/or clinical trials). A number of tools (n = 4% or 18%) used mixed data sources, most commonly a population-based dataset such as the Surveillance, Epidemiology, and End Results (SEER) database complemented by data from institutional patient databases or clinical trials.

All except for one tool (QxMD) have reported on validation of at least one of their calculators and these validation efforts were published either as peer-reviewed manuscripts or as technical reports. Only one tool (Adjuvant) reported any information on actual use of the tool in real world settings. Funding was received from diverse sources including federal, institutional, and industry.

All except for one prognostic tool (QxMD) was available in a web-based format. QxMD works as an application designed for handheld devices such as the iPhone, Blackberry, or Android. Adjuvant is also available for handheld device and on CD-ROM and EBRT had a downloadable excel version. All abstracted tools were free of charge for users and most were open access (n = 20% or 91%). Two tools (adjuvant and QxMD) required login information before accessing the tool; (CSQS) and (MSK pancreatic cancer calculator) required access to be granted by the developer.

In this systematic review we identified and summarized information on 22 interactive cancer prognostic tools. Our review showed that advanced and multiple prognostic tools for cancer prognosis are currently freely available online for a large number of cancer sites, diverse types of patients with cancer, and clinical scenarios. We found great variability in terms of number of tools for different cancer sites ranging from 11 tools for prostate cancer to two tools for several cancer sites. Overall, we noted a major variation across existing tools in terms of format and content. This lack of standardization is not surprising given that this review focuses on the first generation of cancer prognostic tools and to date most of these tools were developed in isolation by separate research and practice groups in the general absence of guidelines or recommended practices. Use of these tools in clinical practice may support communication and understanding about cancer prognosis, but further testing and dissemination of these tools in clinical settings is needed. We found limited reports on actual use and application in real world clinical settings for only one tool. Validation of the underlying algorithm is only the first of many issues to be considered in the use of prognostic tools. If the purpose of clinical shared decision-making is a goal of an instrument, the scientific validity alone does little good if patients, family members, or different groups of providers cannot adequately understand and use the information presented (23). This is especially critical given the emerging information on low health literacy and numeracy (12, 24).

Our review found that cancer prognostic tools differed markedly in the type and number of input variables they took into account. The differences in input variables across cancer site–specific calculators are partially due to the variability in the available scientific information on certain input variables across cancer sites. Most tools included basic tumor and patient characteristics, but lacked information about the effect of genomics as well as patient comorbidities. As genomic medicine continues to evolve, incorporation of these factors into prognostic tools as well as environmental exposures will be critical. Furthermore, understanding how cancer relates to overall prognosis in the context of other conditions (e.g., cardiovascular disease) is important as the population of cancer patients' ages. Very few tools accounted for modifiable risk factors including smoking, overweight/obesity, alcohol consumption, physical activity, or other lifestyle into measures of prognosis. As data continue to add support for the potential effect of these factors on prognosis, refinement of tools will be needed. Until prognostic systems include all of these risk factors, the promise of precision medicine will only be partially fulfilled.

While most tools focused on similar outcomes, there was some variation with respect to the time frames reported as well as presentation of estimates. Furthermore, the development of the tools was based on variable datasets, ranging from single site clinic population to population-based datasets such as SEER. The lack of standardization makes comparison of results from different tools difficult (25). This might require that users experiment with different tools to determine which tool(s) fit best with their clinical practice or specific patient with cancer.

Only two tools included potential adverse impacts of therapy as an outcome variable and there was no tool that used quality of life as one of its outcomes. As suggested by Browman and colleagues: “experienced oncologists understand that for significant portion of patients with cancer, maintaining and improving health-related quality of life is an important objective when considering treatment options” (26). As quality of life and side effect-related outcomes are most meaningful and important to patients with cancer when making treatment decisions, prognostic tools should strive to include them as one key outcome variable. For this to happen, data on both these variables and the earlier mentioned modifiable risk factors should be consistently collected and made available.

The purpose of the cancer prognostic tools was not always clearly defined and ranged from providing baseline prognostic estimates to supporting treatment decisions after diagnosis or adjuvant treatment decisions after primary (e.g., surgery) treatment. It is important to clearly differentiate tools that provide information on prognosis without manipulation of treatment and those that can account for prognosis differences based on therapy of choice. Similarly, only two tools accounted for potential impact of behavior change. Such component could be an important motivator as well as a key aspect for informed decision-making. Discussion of cancer prognosis may be a classic "teachable moment or setting" (27, 28). Furthermore, it is important that tools directly specify their optimal users and timing of use. Specifically, the prediction tools developed by the Memorial Sloan-Kettering Cancer Center (MSK tool; New York, NY) included an introductory page for each cancer site that includes a clear listing of the mission of each calculator, who should use them, the dataset it is building on, the type of outcomes the calculator will provide, the data elements needed to use the calculator (e.g., age, cancer stage, etc.) and any limitations and special notes to the user. A similar functionality was developed for the revised version of the CSQS tool (developed by the National Cancer Institute), but not found on most other tools.

Moreover, it is important that the tools specify whether developers intend tools for use by patients. As indicated by our review, almost all of the cancer prognostic tools we found were accessible to the general public although only four tools mentioned patients as potential users. Cancer prognosis is among the leading topics of information that is sought by patients with cancer (5, 29). Anecdotal evidence and simple review of online patient discussion groups, indicates that patients with cancer already access these tools and discuss and compare prognostic results from the tools with each other (30). While studies have shown that patients may benefit from cancer prognostic tools (31) studies have not assessed the impact of cancer prognostic tools when directly accessed by patients or how they may optimally be used with both patient/family and provider focused interfaces to facilitate better informed decisions.

Finally, we found that only two of the reviewed cancer prognostic tools had applications for a handheld device. Recent research suggests that there is an increasing number of health care providers (especially physicians) who own smart phones and access medical information via their handheld devices (32, 33). There is also increasing desire to access medical applications such as decision aids remotely and at the place of service (34). Making existing prognostic tools accessible through multiple devices would increase their access and use. Furthermore, methods to integrate the use of these tools into patient care, including cancer care plans are still unclear (35).

Limitations of this report include potential underestimation of the number of prognostic tools eligible. From our initial review, we found six tools, which were excluded because of inability to gain access to the site. On the basis of our search criteria, these sites were eligible for inclusion in this review. We included tools identified through May 2011. There could have been more tools published since then, although in scanning the literature in March 2013, no additional tools were identified. Furthermore, our review did not compare the algorithms driving each tool as this was beyond the scope of our article. However, despite the potential limitations, our review used comprehensive search criteria, had clear enumeration of criteria and recommendations for reporting, and compared a large number of tools. To our knowledge, this is the first such review and compilation undertaken to date.

In summary, our study systematically identified and reviewed 22 cancer prognostic tools on several important criteria. Cancer prognostic tools hold great promise in facilitating patient-centered communication and decision-making and helping patients prepare for life post treatment. Providers can also anticipate that increasing number of patients will use, or may already be using, these tools especially in early stages of the cancer journey. The process of identifying available tools was time consuming as there was no one location where all existing cancer prognostic tools were easily accessible and compared. Thus, finding appropriate tools and evaluating them for use may be challenging if not impossible for a busy practicing clinician. The intent of this review is to assist providers and health care teams/systems in becoming more aware of available interactive cancer prognostic tools so that informed decisions about their use in clinical practice can be made. This review makes a significant contribution to the literature and has considerable potential to enhance clinical practice. Technology has afforded unprecedented opportunities to assist cancer care specialists to improve care delivery and to support treatment decisions of the patients. To maximize these opportunities, we advocate clear delineation between tools and pages on websites that are intended for providers and those that are intended for patients, with careful attention to state-of-the-art and science of health communication to design appropriate content and format for each audience.

Future dissemination, testing, and refinement of existing tools and development of new tools based on scientific and risk communication evidence, as well as evolving technologies will be needed. More specifically the following next steps and recommendations emerge from our work:

1. Clear criteria should be developed to guide the development of new and evaluation of existing provider-facing tools in terms of content and format. This could include best practices guidelines on the standardization of prognostic tools and adequate information about intended use and limitations; eventually lead to accreditation of existing tools.

2. Several prominent cancer sites have only small number of tools available indicating need for additional developmental work focusing on cancers that may not be well represented to date.

3. There should be a special focus on adaptation of existing and new tools to newly emerging platforms such as iPads, smart phones, and mobile devices.

4. Furthermore, integration of tools into electronic medical records would further increase their use and smooth integration into practice.

5. Tools should be compared in terms of the consistency and reliability of algorithms and their applicability and usefulness in real world settings.

6. Tools should include factors and outcomes that have increasing relevance to providers and patients including impact of modifiable risk factors and genomic, epigenetic, environmental and patient behavior, and preference factors and measures of quality of life.

No potential conflicts of interest were disclosed.

The opinions or assertions contained herein are the private ones of the authors and are not considered as official or reflecting the views of the NIH.

Conception and design: B.A. Rabin, T. Sanders, L. Nekhlyudov, S. Bull, R.E. Glasgow

Development of methodology: B.A. Rabin, B. Gaglio, T. Sanders, L. Nekhlyudov, R.E. Glasgow

Acquisition of data (provided animals, acquired and managed patients, provided facilities, etc.): B.A. Rabin, B. Gaglio, T. Sanders

Analysis and interpretation of data (e.g., statistical analysis, biostatistics, computational analysis): B.A. Rabin, B. Gaglio, T. Sanders, L. Nekhlyudov, S. Bull

Writing, review, and/or revision of the manuscript: B.A. Rabin, B. Gaglio, T. Sanders, L. Nekhlyudov, J.W. Dearing, S. Bull, R.E. Glasgow, A. Marcus

Administrative, technical, or material support (i.e., reporting or organizing data, constructing databases): B.A. Rabin, T. Sanders

Study supervision: B.A. Rabin

The authors thank Ms. Sara Hoerlein and Michelle Henton for their support with the preparation of the manuscript.

This work was supported by the National Cancer Institute (P20 CA137219). The authors of this manuscript (all except R.E. Glasgow) have received financial support from the National Cancer Institute to evaluate the feasibility and small-scale implementation of one of the cancer prognostic tools (Cancer Survival Query System) included in this review.