Perspectives for Consideration in the Development of Microbial Cell Reference Materials

As evidenced by recent publication trends in microbiome research, researchers are moving the field of metagenomics toward large epidemiologic studies and commercial development at a rapid pace. If these analyses are to ever evolve into reliable assays (e.g., for etiologic studies or clinical diagnostics), the measurement process must be regularly assessed to ensure measurement quality. A key aspect of this validation is the routine analysis of reference materials as both positive and negative controls. A reference material is any stable, broadly available, and well-characterized specimen that is used to assess the quantitative and/or qualitative validity of a measurement process. Many sorts of reference materials are available, and we focus here on the use of microbial whole-cell reference materials (WCRM) for characterizing metagenomic analyses.

To make the most accurate comparisons, a reference standard should closely resemble the physical, chemical, and biological characteristics of the sample of interest. However, it is rarely, if ever, possible to identify WCRMs which are identical to the sample source and that have been characterized to the extent necessary. Therefore, compromises must be made, and a universal WCRM is not feasible. Furthermore, the specific questions being addressed in an investigation will dictate the most appropriate WCRM (balancing, e.g., taxonomic similarity to test samples, depth of characterization, availability, and reproducibility). Many diverse WCRMs have been utilized by different research efforts and commercial interests (Table 1).

Microbe-based reference materials can be roughly divided into three key categories (Fig. 1). Each of these is associated with benefits and drawbacks that need to be assessed in the context of experimental design for a given project.

Samples from existing microbial environments may be suitable as WCRMs if they can be (i) obtained in large enough quantities, (ii) homogenized to remove variability between aliquots, and (iii) stored for long periods of time with either negligible or immeasurable deterioration. The most obvious benefit of this kind of WCRM is that its composition will closely match the sample of interest that will be collected from that environment.

There are, however, several drawbacks to using environmental samples as WCRMs. One of the major limitations of using such a sample is that it is challenging to define (i.e., measure) the exact composition and abundance of species. This is particularly difficult for ecosystems that are complex and contain many low abundance taxa. Also, the ecosystem source for the WCRM may change over time; thus the WCRM cannot easily be reproduced if more material is needed later. For example, although an individual's fecal microbial ecosystem may remain generally stable over long periods, variations (e.g., in abundance profiles) may be observed, reflecting changes in host diet, immune status, physical fitness, and environment, etc. over time (1–3).

The abundance of a given WCRM sourced directly from the environment is a major limitation to its usefulness. To overcome this limitation, several groups have developed laboratory systems that mimic many parameters that shape ecosystems, while at the same time allowing these parameters to be measurable and controlled (reviewed in ref. 4). For example, bioreactors support and maintain the environmental pH and oxygen tension matched to the source environment and provides a constant supply of nutrients and removal of waste material. They can be seeded with environmental inocula and allowed to attain an equilibrium (steady state) under user-defined conditions. The steady state ecosystem may be maintained for several weeks or more, depending on the environment being modeled, and the system can also be scalable. Because these kinds of model systems attempt to recapitulate the natural environment, an added benefit is that any reference material obtained from them will be supported within a matrix that will at least partially reflect that of the natural source. Another advantage is that a scalable quantity of material can be produced at one time. Taken together, these factors partially overcome the limitations of reference material availability described for the first category, above.

When seeded with complex microbial ecosystems from a given natural source, however, in vitro models remain subject to batch-to-batch variations that need to be completely characterized each time. This problem may be mitigated by storing multiple identical inocula to restart the bioreactor on multiple occasions. However, seed stocks are limited and may suffer from deterioration over time even under optimal storage conditions. As with the environmental sample WCRMs discussed above, complete characterization of these in vitro expansions remains a significant technical challenge.

Many representative organisms from diverse ecosystems have been isolated and cultured as pure isolates. Defined mixtures of these pure strains can be used to generate WCRMs in two ways: (i) a subset of a natural community may be combined to inoculate a defined ecosystem in an in vitro model, as described above; and (ii) pure isolates may be grown separately and subsequently mixed in defined ratios.

Defined subset in vitro communities are cultured in systems that potentially restore some metabolic functionality and matrix components of the ecosystem. Although inherently artificial, this modality offers a balance of a defined set of input strains while allowing for those strains to adopt a seminatural state (e.g., compositional profile and metabolic activity) to mimic some aspects of the in vivo situation.

In contrast to this approach, pure microbial isolates have several key benefits as components of microbial standards. Using adequately banked strains and well-characterized propagation procedures, it is possible to obtain an almost limitless supply. In addition, pure isolates can be subjected to thorough phenotypic and genotypic characterization, which adds value to any resulting microbial standard. Furthermore, aspects of experimental design can be accommodated, such as the need to balance for example, Gram-positive and Gram-negative bacterial species in a useful way. Microbial isolates may also be grouped into a standard at specified abundances to mimic natural ecosystem, that is, a “mock community”; however, the resulting ecosystem will be inherently artificial and will lack properties, such as metabolite profiles, that can only be gained through culture of an ecosystem as a unit.

Although many banked microorganisms are in principal available for fit-for-purpose assembly of mock communities, they often lack the level of characterization required for use as an appropriate reference material. Many consist of incomplete genomes and/or are poorly annotated. For example, the copy number of the 16S rRNA gene in databased bacterial sequences can be unreliable. Whole-genome sequencing and complete genome assembly of candidate strains could resolve these issues but requires a large upfront expense. Furthermore, a repository for characterization data would aid the utility and uptake of appropriate WCRMs.

Synthetic mock community WCRMs provide an opportunity to design mixtures that are particularly well suited to test known microbial measurement challenges. For example, bacterial WCRMs should include both Gram-positive and Gram-negative organisms. A range of relative abundances should be present with at least a 100-fold difference between high and low abundance organisms. The genetic diversity of the WCRM should also be considered. Some naturally occurring microbiomes are comprised of >1,000 phylogenetically diverse species, while other microbial communities are much simpler. Ideal WCRMs will encompass the expected diversity and phylogenetic breadth of actual samples. The additional inclusion of groups of closely related organisms, including strains within the same species, would be useful for assessing the sensitivity of measurement pipelines to genotypes exhibiting high genetic similarity. Finally, many microbial samples include matrices that complicate analysis (e.g., PCR inhibitors are common in fecal samples). WCRMs can be designed to include matrix elements that are known to be problematic. Alternatively, assembled WCRMs spiked into relevant matrices may be used.

WCRMs can be used at the beginning of a project to define, evaluate, and assess the protocols to be used before going on to work with perhaps expensive or precious samples. During a project, WCRMs evaluated regularly alongside unknown samples will increase confidence that the measurement process is consistently performing as intended. WCRMs can be evaluated when introducing different protocols and platforms to an existing workflow (e.g., use of a new instrument or a different sequencing center), and this provides an opportunity to increase the value and portability of resulting datasets. An important caveat here is that WCRMs are likely to differ, sometimes substantially, from the actual samples of interest. It remains possible that a measurement process that appears well characterized and predictable based on WCRM analysis will be impacted by particular attributes of actual samples in unanticipated ways. Thus, while WCRMs are very useful for identifying measurement failures, they cannot guarantee valid measurement results. Some common questions and considerations about the measurement process that WCRMs can help address are provided in Table 2. It is recommended that when microbial measurement results are published, the data from routine analysis of well-characterized WCRMs are also made available (i.e., as data critical in supporting the research conclusions). Some studies of mock communities with known composition have been used to demonstrate biases in observed relative abundances that can occur at any stage in the process from DNA extraction, to PCR amplification, to bioinformatics (9, 10). Posted work by McLaren and colleagues describes how to estimate taxon-specific biases in mock samples (11). While such research is promising, it remains unclear whether WCRM data could be used to adjust the relative abundances measured in complex experimental samples; perhaps more formal methods for using the WCRM data will become available.

Some common characteristics to be considered for metagenomic WCRMs are taxonomic composition, known and accessible sample source, growth conditions (if cultivable), readily available reference sequence, genome size, GC content, percent repeats in the genome, known copy number of target sequences (e.g., 16S or 18S rRNA genes), and functional information.

For example, one potential use for mock communities is to measure the differential in nucleic acid extraction efficiencies among community constituents. To make such assessments, we must first quantify the constituents of the mock community and be able to add each in a known quantity; by processing a known quantity one should be able to calculate an expected result. There are various means to counting cells such as plating and counting colony forming units, flow cytometry, and spectrophotometry but none of these offer the accuracy required to make downstream measurement on the molecular level. Once the mock community has been subjected to the extraction process the resulting DNA must be measured, and quite often this is found to be at a concentration of <1ng/μL. Spectrophotometry (NanoDrop) and fluorescent dyes (PicoGreen) lack the accuracy to measure concentrations typical of metagenomic samples, which leaves qPCR as the choice of measurement. In order for qPCR to be used to measure the individual constituents in a mixed population one must design primers to a sequence that is both unique to a given component strain and occurs at a known copy; this often proves to be problematic. Furthermore, a target that is common to a constituent and occurs in equal copies should be chosen to serve a calibrator for making copy number calculations, which can also be challenging. Without an accurate means of measuring both input (cells) and output (DNA) it becomes very difficult to assess the efficiency of the extraction process.

As all samples of interest are compared with a common reference standard, a poorly suited standard may result in a systemic bias. For example, assembling mock communities from single isolates to assimilate metagenomic samples is a logical approach but has proven difficult to achieve.

Epidemiologic microbiome studies benefit greatly from the routine inclusion of comparable standard samples in sequencing experiments to allow laboratories to assess their own performance over time and facilitate data pooling or meta-analyses that are anticipated. As no complex WCRMs for positive controls for microbiome analysis are currently readily available, researchers at the U.S. NCI have recently developed limited quantities of three types of standard reference material for future etiologic studies (Table 1). Samples have been divided into multiple, identical one-use aliquots for distribution to the cancer epidemiologic studies. Cancer epidemiologists interested in obtaining these samples should get in touch with Rashmi Sinha (sinhar@nih.gov) at NCI (Bethesda, MD). Because of limited availability of samples, requests will be reviewed by a committee and a material transfer agreement will need to be established.

The first type of WCRM is a community of 45 purified bacterial strains, grown to known biomass quantities, and mixed together in known ratios which will serve as the quantitative “gold standard” artificial community. These bacterial strains were similar to fecal-derived species but considered rare within most donors, and are provided in known ratios as a mock community in a saline solution. The second of these reference materials is fecal material. Researchers collected >200 g fecal samples each from five individual donors (healthy adult, adult on low carbohydrate diet, adult with high body mass index, adult with inflammatory bowel disease, and healthy infant) to optimize the ability to observe microbial differences. These five samples were spiked with the artificial community mentioned above. The third is material from a chemostat grown to steady-state equilibrium under conditions mimicking those found within the mouth, and seeded with samples of saliva, tongue scraping, and plaque and oral bacterial communities grown as biofilms. These WCRM samples will be extracted and the 16S ribosomal RNA amplicons as well as the metagenomes will be sequenced in multiple laboratories to provide reference data.

The primary use of the WCRM samples will be to see whether a particular batch in a given laboratory meets quality control specifications. If not, data from that batch might need to be discarded. So long as cases and controls are balanced within batches, however, some batch aberrations might be acceptable. Each WCRM needs to be characterized (preferably in more than one expert laboratory) to determine relative abundances of taxa at several taxonomic levels and corresponding measures of alpha diversity, such as Shannon index. For each parameter to be used for quality control (relative abundances and alpha diversity), at least two expert laboratories should make the measurement repeatedly on different days to estimate within laboratory and between laboratory components of variance. These can be used to determine tolerance limits for quality control testing; values outside such limits indicate that the measurement is bad.

Then, to determine whether an experimental batch is acceptable in a new study, the following quantities might be tested to see whether they are within tolerance limits to what was expected:

1. alpha diversity

2. relative abundances of phyla and their ranks (other taxonomic levels could be examined also)

3. ability to detect some rare genera or species that had been intentionally spiked in

4. if contaminants not in the WCRM are appreciable, issue a warning. Perhaps eliminate that batch if the relative abundance of the contaminant is large

5. negative controls should also be tested for contaminants and the batch eliminated if large quantities are detected.

In a similar manner, data from WCRM could be used to eliminate bad batches for pooling data among different study centers or for meta-analyses. In each study center, the investigator could discard bad batches and only use the good batches to compute summary statistics such as the logistic regression slope of case–control status on alpha diversity. These summary statistics could be submitted for meta-analysis, although the WCRM quality control would still be based on assessment of a given batch. For measures of beta diversity, investigators could pool data only from batches that passed WCRM quality control criteria.

Another aspect that is critical is the need to incorporate negative controls into a study design. Studies should routinely include collection and extraction buffers alone for sequence analysis as they may contain low-level contamination that, despite being minimal, may lead to incorrect detection of nonbiologically relevant bacteria. This is especially important in the context of microbiome studies that propose to identify rare taxa in their biological samples. Recently, the Microbiome Quality Control Study found that even simple buffer blank controls had significant contamination (6). Contamination in buffer blanks is normally a result of trace amounts of DNA present in the DNA isolation, barcoding, and sequencing kits (12), or as a result of cross-contamination of samples that are processed together (13). In general, researchers should be aware that reagents found within most molecular biology kits can introduce contaminants that can mask true biological signal from low biomass samples, and that this can be a particularly thorny problem when trying to assess the presence of low abundance taxa.

A variety of WCRMs have been developed to provide “ground truth” benchmark data for characterizing metagenomic measurements. The authors are left with the following thoughts:

• Measurement quality is a critical part of any experiment and is particularly important for analysis of complex samples as is commonly carried out in microbiome studies.

• The use of WCRMs represents a logical approach for characterizing and validating experimental procedures and measurement processes.

• WCRMs can increase confidence in experimental design and resulting measurements but cannot guarantee accurate or reliable results.

• To be useful, a given WCRM needs to be consistent (i.e., unchanging in time or between laboratories), well-characterized, widely available (ideally in unlimited supply), and inexpensive.

  • A one-size-fits-all WCRM is thus an unattainable goal

• WCRMs need to be chosen that best fit a given experiment or question, taking into account, for example, diversity (both phylogenetic and in terms of microbial characteristics such as GC ratios or cell wall thickness), abundance, and source environment.

  • A research question may require the use of multiple microbial reference materials throughout the entirety of a research investigation.

  • A modular design (addition of subgroups of samples) may address some challenges in the application of WCRMs to different projects.

• Large prospective cohort studies being planned now should include standard WCRMs to facilitate future data pooling and meta-analyses by excluding batches where the quality control standards were not within the limit of tolerance.

E. Allen-Vercoe reports receiving a commercial research grant from and has ownership interest (including patents) in NuBiyota. No potential conflicts of interest were disclosed by the other authors.

Commercial material suppliers are identified in this article to foster understanding; such identification does not imply recommendation or endorsement by the National Institute of Standards and Technology, nor does it imply that the materials identified are necessarily the best available for the purpose.

The authors would like to acknowledge and thank participants at the 2016 Standards for Microbiome Measurements Workshop at NIST where the ideas contained herein were nucleated. The authors would also like to acknowledge Dr. Rob Knight (University of California, San Diego, CA) and Drs. Deirdre Devine, Phil Marsh, and Thuy Do (University of Leeds, United Kingdom) for their help in developing the new WCRMs for prospective cohort studies. Funding for developing WCRM for prospective studies was supported by the Intramural Research Program of the NCI at the NIH. Funding for the 2016 Standards for Microbiome Measurements Workshop was provided by NIST, NIAID, and multiple corporate sponsors (including Zymo Research and ATCC). The workshop proceedings are publicly available online (14).

The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.