Crowdsourcing to Curb Bad Science

Weary of flawed biomedical research tools that generate questionable data, an international panel of scientists has created the Chemical Probes Portal—a community-driven website (www.chemicalprobes.org) to “crowdsource medicinal chemistry and pharmacology expertise,” according to their recent commentary (Nat Chem Biol 2015;11:536–41). The portal will address a range of common research queries, such as whether probes for specific target proteins are available, and how these probes should be used.

“We're talking about chemical compounds used to investigate biological processes in cells and to determine the druggability of molecular targets,” says panelist Paul Workman, PhD, chief executive and president of The Institute of Cancer Research in London, UK. That reagents of poor quality and selectivity are still widely used “is a problem I feel very strongly about,” he adds. “Many have off-target effects or are indiscriminately active. As such, scientists risk drawing the wrong conclusions in their research, confusing themselves and others.”

“People can't keep up, or don't, with the literature showing that these compounds are problematic, and the suppliers keep selling them,” writes panel member Derek Lowe, PhD, in a July 22 blog post (available at http://blogs.sciencemag.org/pipeline/archives/2015). He points out that the polyphenol rottlerin has been used as a PKC-δ inhibitor “in uncounted studies in the literature” even though its selectivity has long been questioned.

Another example highlighted in the panel's commentary is dorsomorphin's continued use as a probe of both TGFβ receptors and AMP-activated kinase: “Which activity is responsible for the observed biological effects? Perhaps neither. At least 10 other kinases are more potently inhibited by dorsomorphin.”

Workman believes that “the self-correction inherent in crowdsourcing” will ensure that the new portal contains accurate, up-to-date comparative information. “No system is perfect, but I expect ours to be much more effective than the current one where search engines introduce historical bias through accumulated citation frequency,” he says.

The portal's success ultimately hinges on active community input, effective leadership, and sustained funding—currently, it's operating on seed money from the Wellcome Trust. “We've kicked things off and will act as supporters and mentors,” Workman says of the scientist panel, “but making this a thriving, independent mechanism for boosting chemical probe quality will require professional staff and volunteers, along with all the help we can get from social media and high-profile advocates.”

Quality control through independent curation of the portal's content is also essential—“submitted opinions should be shown as ‘unreviewed' until they're approved by specialists familiar with proper usage of the probes in question,” Workman explains.

“There'll definitely be some challenges, getting this [portal] together,” says Kip Guy, PhD, chair of chemical biology and therapeutics at St. Jude Children's Research Hospital in Memphis, TN. “Just thinking about the probes I know well, I'd imagine it's a solid couple days' work to assemble reasonable, well-referenced, and accessible descriptions of each, which will then need to be regularly updated. However, Wikipedia's success makes me think this is possible.”

Workman hopes this community-led resource significantly improves the quality of studies using chemical probes. “Gold-standard research is what we should expect, and what patients with diseases like cancer deserve,” he says.

“The excuses for using worthless chemical probes have never been good,” Lowe adds, “and with any luck, there eventually won't be any excuses left.”

For more news on cancer research, visit Cancer Discovery online at http://CDnews.aacrjournals.org.