Pathology Image Informatics Platform (PIIP) is an NCI/NIH sponsored project intended for managing, annotating, sharing, and quantitatively analyzing digital pathology imaging data. It expands on an existing, freely available pathology image viewer, Sedeen. The goal of this project is to develop and embed some commonly used image analysis applications into the Sedeen viewer to create a freely available resource for the digital pathology and cancer research communities. Thus far, new plugins have been developed and incorporated into the platform for out of focus detection, region of interest transformation, and IHC slide analysis. Our biomarker quantification and nuclear segmentation algorithms, written in MATLAB, have also been integrated into the viewer. This article describes the viewing software and the mechanism to extend functionality by plugins, brief descriptions of which are provided as examples, to guide users who want to use this platform. PIIP project materials, including a video describing its usage and applications, and links for the Sedeen Viewer, plug-ins, and user manuals are freely available through the project web page: http://pathiip.org. Cancer Res; 77(21); e83–86. ©2017 AACR.
With the advent of whole slide digital scanners, histopathology slides can be digitized into very high-resolution digital images, realizing a new “big data” stream that can potentially rival “-omics data” in size and complexity (1–3). The first FDA approval for marketing a whole slide imaging system for primary diagnosis was granted in April 2017, and this will generate increased interest in using digital scanners by pathologists in hospital settings. Features extracted from whole slide images (WSI) using sophisticated image analysis methods have already been shown to improve diagnosis (4) and have prognostic power in a number of applications (5). There is also growing interest in combining morphological information extracted from WSIs with genomic data and in vivo medical images to discover new biomarkers (6, 7).
There is a need for user-friendly software tools that can support the display, annotation, and analysis of different sources of image data. For researchers, the ability to use analysis algorithms that have been contributed by other researchers is also a key requirement.
Although other image analysis platforms exist, most have not been designed to handle digital pathology WSIs. For example, Fiji (fiji.sc) is an image processing package aimed at providing scientific image analysis tools for life sciences researchers and it is built on ImageJ (imagej.nih.gov), which is an extensible, open source, Java-based program developed by the NIH. Fiji is popular for cell image analysis but it was not designed to view and process large tiled WSIs.
Sedeen is a pathology image viewer (http://pathcore.com/downloads/), which was initially developed with funding from the Ontario Cancer Research Institute. It was designed specifically to address the unmet need for a platform that could be used by the academic community to share both established and novel digital pathology visualization and analysis tools with pathologists and other imaging researchers (8). It is built in C++, which ensures good computational performance while also making it simple to integrate with common image analysis libraries such as ITK, VTK, and OpenCV.
The pathology image informatics platform (PIIP) is a multi-institutional project supported by the NCI/ITCR, which is leveraging the Sedeen viewer framework (see Fig. 1). It is a collaboration between five different participating academic institutions (University of Pennsylvania, University of Michigan, Ohio State University, Case Western Reserve University, University at Buffalo in the US and Sunnybrook Research Institute in Canada) and Pathcore, a for-profit company, which has been providing free access to Sedeen to the academic community since 2010. See Supplementary Video S1 for a brief overview of the PIIP project and a demonstration of the Sedeen viewer.
Sedeen allows users to view images from multiple modalities (radiology and pathology in particular) and to perform visual overlays and registrations for biomarker validation (e.g., IHC and hematoxylin and eosin) and multimodality comparisons (hematoxylin and eosin and MRI). Among other capabilities, Sedeen provides rich annotation capabilities, supports a wide range of WSIs, and is extensible through the Pathcore Software Development Kit (SDK), allowing researchers to share and validate novel cancer informatics tools. At present, each slide scanner vendor has a proprietary WSI format. Typically, an image is saved as a series of tiles at multiple resolution levels; this allows subregions of the image to be retrieved without having to load the whole image into memory at once. An essential feature of the Sedeen viewer is the ability to import multiple file formats, including Aperio SVS, Leica SCN, Hamamatsu NDPI, Zeiss MRXS, VisionTek SVSlide, other TIFF-based formats, and JPEG-2000.
Although the viewer is currently distributed as a Windows application, it has been designed to be portable to other platforms, for example, MacOS and Linux, if required. Dr. Emily Patterson (Ohio State University, Columbus, OH), a human factors expert, has conducted a heuristic review of all the toolbars and menus on the Sedeen interface and will also provide input on the design of plugin interfaces (9). This will ensure that the software, including its plugins, is intuitive to users, which is important in collaborative research.
Software Development Kit
The Pathcore SDK is a tool for integrating quantitative image analysis algorithms with the Sedeen viewer. The SDK is written in C++ and uses an open-source tool, CMake (cmake.org), to manage the build process in a way that is independent of both the operating system and the compiler used. The SDK application program interface (API) is fully documented and generated using Doxygen, which is a standard tool for extracting documentation from source file comments. It is distributed with a set of tutorials and exemplar algorithms, which can be selected from a pull-down menu at run time. Sedeen uses an on-the-fly image decoding strategy (directly from the disk), which is transparent to the clients and is very efficient. This allows client code to access arbitrary regions in an image without first loading the entire image into memory. The SDK simplifies access to WSI data by providing a set of format-independent data extraction routines, the ability to create analysis pipelines, and the ability to easily visualize analysis results. Algorithm designers can therefore develop algorithms much more easily by leveraging the capabilities of Sedeen and the image analysis architecture provided by the Pathcore SDK.
The Pathcore SDK also provides numerous widgets that allow user parameters to be collected from the analysis manager GUI and transferred to the analysis algorithms. The output from the algorithm can then be rendered within the Sedeen viewer in the form of an overlay, new annotations, or as text.
With funding from the ITCR program, groups at Case Western University, The Ohio State University, and Sunnybrook Research Institute are implementing a variety of quantitative image analysis tools as plugins. Examples are described below and all of these plugins will be distributed with the next release of Sedeen. A github repository (https://github.com/sedeen-piip-plugins) contains the source code and user documentation for the plugins developed by the PIIP. Pathcore is providing support and is also working to expand the capabilities of the SDK to facilitate academic research, for example by developing methods for accessing MATLAB and Python code directly from the Sedeen Viewer.
The co-principal investigator (A. Madabhushi) and his team have developed a powerful tool for high-throughput biomarker quantification named Hierarchical Normalized Cuts (HNCuts; ref. 10). The method combines a frequency-weighted version of the mean shift (MS) algorithm with the normalized cuts scheme to segment all the image pixels into representative classes. Unlike supervised segmentation strategies, our scheme only requires specification of a small representative sample of the target class to rapidly identify similar objects in the image.
The PIIP provides algorithms needed to compute user-provided inputs (images, parameters, and annotations) and reports results through the Sedeen Viewer's GUI. In this case, users access the viewer's annotation tools to define the sample color swatch, which can be accessed by the HNCuts plugin. Once processing is carried out, the segmentation and quantification results are rendered within the Sedeen Viewer as a color/contour overlay. This plugin also demonstrates how algorithms implemented in Matlab can be integrated into a plugin.
Out of focus detection
As whole slide imaging becomes more prevalent and many centers, particularly in Europe and Canada, adopt digital pathology, the demand for scanning whole slides will increase. Although many modern scanners are equipped with software to automatically detect areas that are out of focus, and adjust the scanning parameters accordingly, large numbers of digital slides contain out of focus areas. According to one study conducted at a Dutch academic pathology laboratory, 5% of the cases had problems with scanning artifacts such as out of focus areas (11). In a large-scale digitization effort, this percentage is expected to increase considerably, as the large volume of cases make it impractical to visually identify problematic cases and then rescan them if necessary.
The goal of this plugin is to detect out of focus areas and report them to the user. In its implementation, a plugin was created using Pathcore SDK and integrated with the Sedeen Viewer. This plugin also demonstrates the use of the OpenCV API, a popular computer vision library, with the SDK. The plugin identifies regions that are out of focus and produces an overlay showing where these regions are located.
Transforming and exporting annotations
One of the key differences between the Sedeen viewer and other platforms is that it provides tools for users to manually coregister images. This allows users to make direct comparisons between images acquired using different modalities or stained using different antigens. The ExportTransformedROI plugin extends the capabilities of Sedeen by allowing an annotation defined on a source image to be exported to a target image. If the source image and the target image have been aligned, then the transformations needed to achieve the alignment are also applied to the annotations.
Use case example: analysis of multiple IHC slides
Serial sections cut from a tissue block and stained using different IHC markers can be used to compare the relative expression of different antigen receptors in tumors. In the Supplementary File, the workflow needed to spatially align two IHC images and then carry out a quantitative comparison of tissue staining over corresponding regions of interest is described. This demonstrates how Sedeen's built-in functionality can be combined with user-contributed plugins to achieve a specific analysis task.
The Sedeen Viewer has all of the advantages of a viewer built specifically for digital pathology, with a user interface that is intuitive for pathologists and biologists, while also providing a mechanism to expand its functionality through Pathcore SDK. The viewer is easy to download and install on Windows platforms and, in time, it will be made available for other platforms. We have thus far successfully demonstrated that image analytic algorithms from three different institutions (Ohio State University, Case Western Reserve University, and Sunnybrook Research Institute) can be integrated within the PIIP. Our future plans involve enabling the seamless integration of analytic tools developed by other users into the platform. We envision a PIIP that will be dynamically populated with validated image analytic “apps” developed by the community, and hence provide immense benefit to end users, including clinical and research centric pathologists and computational image analysis scientists. Pathcore has been providing free access to Sedeen since 2010 and is extending the capabilities of both the viewer and the Pathcore SDK in response to feedback from the PIIP project team. They are currently working on an improved mechanism to support the compilation and distribution of contributed plugins from the wider user community.
Other novel aspects of this project are also underway. Working with an expert on disease ontologies, Dr. Barry Smith (University of Buffalo, Buffalo NY), we have proposed a quantitative histopathology image ontology (QHIO), which will facilitate interoperability between histopathology datasets and methods used in pathological imaging and analysis. This ontology fosters data compatibility and provides a mechanism for linking algorithms with compatible datatypes. In ref. 12, we demonstrate how the QHIO can be applied to the problem of hot spot detection in breast IHC WSIs. Human-computer interaction and user experience is an integral part of the PIIP. We are working with a human-computer interaction expert, Dr. Emily Patterson, to define user interface design requirements. On the basis of her suggestions, several design decisions have already been modified to make the interface friendlier to users with different experience levels.
Disclosure of Potential Conflicts of Interest
A.L. Martel is a chief science officer and has ownership interest (including patents) in Pathcore.com. D. Hosseinzadeh has ownership interest (including patents) in Pathcore. A. Madabhushi reports receiving a commercial research grant from Philips and Inspirata Inc., has ownership interest (including patents) in Inspirata Inc. and Elucid Bioimaging, and is a consultant/advisory board member for Inspirata Inc. and Astrazeneca. M. Gurcan is a consultant/advisory board member for Inspirata, Inc. No potential conflicts of interest were disclosed by the other authors.
The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Cancer Institute or the NIH.
Conception and design: A.L. Martel, D. Hosseinzadeh, E.S. Patterson, A. Madabhushi, M.N. Gurcan
Development of methodology: A.L. Martel, C. Senaras, Y. Zhou, A. Yazdanpanah, R. Shojaii, A. Madabhushi, M.N. Gurcan
Analysis and interpretation of data (e.g., statistical analysis, biostatistics, computational analysis): A. Madabhushi, M.N. Gurcan
Writing, review, and/or revision of the manuscript: A.L. Martel, D. Hosseinzadeh, C. Senaras, Y. Zhou, E.S. Patterson, A. Madabhushi, M.N. Gurcan
Administrative, technical, or material support (i.e., reporting or organizing data, constructing databases): A.L. Martel, A. Madabhushi, M.N. Gurcan
Study supervision: A.L. Martel, A. Madabhushi
We are grateful to Craig Madho and Deyu Wang at Pathcore for their help in the coordination of the project and facilitating the writing process.
The project described was supported in part by award number U24CA199374 (principal investigators: M.N. Gurcan, A. Madabhushi, A.L. Martel) from the National Cancer Institute. This project is also supported by Pathcore, a digital pathology software company. Pathcore has been supporting the research community since 2011 via the Sedeen Viewer and PIIP.