Choosing Wisely: Applying Value-Based Economic Principles to Population Science Research Investment

Research requires a substantial investment of time, effort, and resources. Researchers planning and conducting studies and funding agencies supporting the research both face the challenge of an imbalance between available funds and the number of research applications (1, 2). Establishing a framework to guide considerations of when, where, and how much to invest can be useful in making resource allocation decisions, whether that resource is the researcher's personal time and effort invested in research planning or a sponsor's available funding. We propose a conceptual framework using economics, which uses optimization principles, that can be applied to guiding investments of time, effort, and money.

The research trajectory from inception to translation and implementation is frequently a long and circuitous route. Research in medical and population health sciences may have varying primary goals, but the ultimate goal is to develop technologies and interventions that improve individual and population health. Scientific peer review is the bedrock foundation for selecting high-quality research for funding considerations by major research sponsors. Scientific review addresses the core principles of importance, rigor, and feasibility. More than enough meritorious applications are submitted to use available resources. Thus, the second level, programmatic review, which also often involves scientific peer reviewers as outlined by several major grant funding organizations [American Cancer Society, Congressional Directed Medical Research Programs Review (CDMRP), and NIH] becomes critically important in determining resource distribution (3–5). At this level, the evaluation process selects among meritorious applications identified through the initial scientific review considering other elements such as mission priority, budget, and potential duplication of past or ongoing research activities both within and across other sponsor organizations. The process at this level of review tends to be less proscribed, making it potentially less transparent and more subject to biases. Once a research concept is developed and research begins, the challenge is to conduct evaluation throughout its trajectory to inform decisions of whether to continue on the original path, to fund routes that follow but diverge from the path, or to abandon the path if it is apparent that its goals cannot be achieved. As resources are finite, rationales for the process of allocating resources should be transparent.

This commentary has two purposes: to alert researchers to considerations relevant to planning projects and programmatic-level review, and to suggest a potential framework for discussion that may reduce potential subjectivity and biases during programmatic review. The framework draws on basic economic principles and applies them to the research setting. Examples use cancer control population science research, as the nature of observational and interventional research involve large population studies (which include large sample size, recruitment, and often long-duration follow-up costs) that demand a high level of resource utilization; the same principles can be applied throughout medical and population health research.

Six economic principles are outlined and defined in Table 1: opportunity costs, diminishing marginal productivity, economic optimization, sunk costs, expected return on investment, and option value. The table also summarizes how the principles could be considered for both research planning and funding decisions.

Opportunity costs are the value of the next best alternative use of resources. Including these in the valuation process forces consideration of what research will not be done by the individual researcher or what research will not be funded if resources are used for a project under consideration. Consideration of opportunity costs occurs after scientific merit assessment, where tradeoffs among all meritorious applications are considered, and issues beyond ranking are considered in funding decisions.

Diminishing marginal productivity considers how the “bang for the buck” may change as more money is invested. For example, how does the value of extending a cohort study change as the study is lengthened? There are some pieces of information that can only be obtained over a specific timeframe, such as cancer outcomes after exposures. However, what happens when the number of individuals in a cohort study diminishes substantially through failed retention or death? The value of assessing new exposures, for example, diminishes. This concept not only applies within a study but also is reflected by whether the next study in an area will be as valuable as previous studies. Replication of study results has high value but the question becomes how much replication is sufficient? When is it time to move to the next step of translation or new investigations (6, 7)? In his 1999 commentary, Dr. Kuller noted that replication is a “pillar of epidemiology;” however, he also notes that too much repetition leads to stagnation and failure to evolve to translation and implementation (6).

A closely related concept is economic optimization (8): given additional investment, what is the additional value obtained? The principle suggests that investment in a research area (by an individual researcher and by a funding organization) should continue until the value of the next investment is just equal to the cost of making that investment. As an example, once a sample size is attained to adequately detect or rule out a clinically significant association, expansion of the sample may incur significant costs, but only produce small gains in the precision of an estimate or new knowledge.

Sunk costs and return on investment focus on time. Sunk costs are past expenditures that cannot be “unspent,” e.g., prior investments. The lamentation, “Why should what we've spent go to waste?” is inconsistent with the logic of maximizing the value of additional investments. While past investments shape the value of the future investments and the amount of resources required to make the next investment, resource allocation decisions for ongoing studies should be forward looking, based on the gain from allocating additional resources and the findings from those additional resources. A study, for example a clinical trial, may have inadequate participant accrual to meet future expectations to adequately address research questions. It may be preferable to find a new use of resources rather than continue to invest in research no longer expected to yield conclusive outcomes. The combination of sunk and opportunity costs can guide discussion away from “how much has already been invested” to focus on “what will more investment add and prevent us from investing in?”

Expected return on investment looks forward, with the challenge of anticipating what value might be created and the time required. These expected returns may be diminished if, for example, the project fails to achieve the needed sample size in a timely manner or a newer technology develops that makes the approach under study outmoded. The return on investment concept also forces an assessment of when the resources will be used and when the results will be obtained and translated into new medical or population health interventions.

Option value reminds us that there may be aspects of data or biological samples that will have value in the future when new questions emerge or when other data are collected and can be combined with existing data. This value is often speculative. However, it should at least be considered and juxtaposed against the cost of continuing to maintain the data or samples and the relative cost and benefits of keeping the data or samples to have the option of using them in the future and compared with other uses of the resources.

For long-term projects, continued funding should focus on future expected returns of additional investment. However, criteria commonly used to judge ongoing research project investments are retrospective, e.g., number of publications, citations, patents, or relative citation ratio (9). Similar to personal financial investment, these do not guarantee high levels of future productivity on a specific project or in other areas. None of the metrics of past productivity necessarily indicate the potential to move toward the goal of knowledge innovation, translation, implementation, and scaling needed to improve population health. One study used the CD index, a measure to characterize whether papers represent breakthroughs in a scientific field, i.e., are disruptive (D) or novel, or consolidate (C) prior work. This analysis of published scientific papers and patents from 1950 to 2010 noted a marked decline in innovative, disruptive research across all scientific fields, including dramatic declines in life and social sciences (10). A challenge is to find the balance between emphasis on innovation and the importance of replication and stepwise evolution of research towards translation and implementation. Adding new metrics and considering the path to translation at the initial concept stage can assist in the evolution of research (7).

Data sharing is not applicable to this article as no data were created or analyzed in this study.

Innovation is considered at both scientific peer review and programmatic review. Tensions exist among the values of innovation, reproducibility, and generalizability. Funding mechanisms such as the “High Risk, High Reward Research Program” (11) emphasize novel and transformative research; however, research that settles areas of uncertainty, uses existing resources to inspire new questions and apply existing approaches or technology in novel ways is valuable. With limited resources, the opportunity costs may become too great when research perseverates and does not advance to translate the results to the clinical and population setting (7, 12).

Cohort studies and prevention trials often require following large numbers of individuals over extended periods during which technology may advance and new exposures may arise. Without adaptation of approaches to address technological innovation or novel exposures, the question for continued investment is, “At what point does continued investment become too costly for the future expected returns considering lost opportunities to fund new research?” Inflection points that may be considered include response rates, age of participants that may be past the critical window of opportunity to study new exposures, and/or attrition due to losses to follow-up or mortality. Does ongoing investment provide future value, e.g., the ability to study new exposures, or has the window of opportunity with the existing population passed? Does expected future return justify continued funding? What are the opportunity costs of continued funding?

Conducting population health research is expensive, and study subject accrual and retention rates are critical milestones. Failure to achieve enrollment and retain participants compromises the ability to address the primary research question successfully, reduces both individual and societal benefits, and alter the participants’ benefit to risk ratio (13, 14). Thus, early termination or adaptation of designs may be necessary to minimize ongoing risks and reset the ratio. Having an independent Data and Safety Monitoring Board, or a similar independent entity, focused on data monitoring, can ensure that the value to participants is preserved. This has the added benefit of having independent oversight to help investigators and funders overcome bias that may be introduced by focusing on sunk costs rather than opportunity costs and future returns.

The path from research project inception to translation and implementation is frequently long and circuitous. Once research is initiated, ongoing evaluation should inform decisions of whether to continue on the original path, fund routes similar to but divergent from the original path, or abandon the path entirely. Funding to support research operates in the reality of the growing demand for limited resources; difficult allocation choices must be made. Economic value-driven questions can be tailored to the project under review and priorities of the funder; these serve as a guide in decision-making for the allocation of research time, effort, and funding. The proposed framework provides a transparent, structured, pragmatic process to consider and discuss tradeoffs and facilitate transparent decisions concerning initiation of new projects and continuation of ongoing projects in the context of limited resources. Consistent application of the framework to funding decisions may help to improve transparency, minimize bias by making the evaluation process more explicit, and, thus, optimize the use and distribution of limited resources.

This commentary was stimulated by our collective experience and observations as researchers developing projects, peer reviewers, and participants in programmatic funding reviews and decisions across multiple sponsor organizations. We hope that through this commentary, researchers become more aware of the programmatic review stage and will have the opportunity to enter into a dialogue concerning how the research community could be engaged to arrive at a consensus on how the questions suggested (for which there are not any absolutely correct answers) could be put into practice in a research planning and decision-making process. It is notable that most major sponsors include investigators in both levels of the review process (e.g., NIH councils and CDMRP programmatic reviews include members from the research community), and a broad dialogue should take place. We are not proposing a new review system but are suggesting guidance for discussion that can help increase transparency and level the playing field for consideration of meritorious applications.

K.D. Frick reports personal fees from NCI during the conduct of the study. No disclosures were reported by the other author.