PurposeThe rapid development of highly multiplexed microscopy has enabled the study of cells embedded within their native tissue. The rich spatial data provided by these techniques have yielded exciting insights into the spatial features of human disease. However, computational methods for analyzing these high-content images are still emerging; there is a need for more robust and generalizable tools for evaluating the cellular constituents and stroma captured by high-plex imaging. To address this need, we have adapted spectral angle mapping—an algorithm developed for hyperspectral image analysis—to compress the channel dimension of high-plex immunofluorescence (IF) images.ApproachHere, we present pseudo-spectral angle mapping (pSAM), a robust and flexible method for determining the most likely class of each pixel in a high-plex image. The class maps calculated through pSAM yield pixel classifications which can be combined with instance segmentation algorithms to classify individual cells.ResultsIn a dataset of colon biopsies imaged with a 13-plex staining panel, 16 pSAM class maps were computed to generate pixel classifications. Instance segmentations of cells with Cellpose2.0 (F1-score of 0.83±0.13) were combined with these class maps to provide cell class predictions for 13 cell classes. In addition, in a separate unseen dataset of kidney biopsies imaged with a 44-plex staining panel, pSAM plus Cellpose2.0 (F1-score of 0.86±0.11) detected a diverse set of 38 classes of structural and immune cells.ConclusionsIn summary, pSAM is a powerful and generalizable tool for evaluating high-plex IF image data and classifying cells in these high-dimensional images.
Breast cancer (BC) remains the deadliest cancer for women worldwide. Neoadjuvant immunotherapies have demonstrated improved responses for some patients. Unfortunately, no robust method exists for predicting which patients will respond to immunotherapy. Imaging of diagnostic BC biopsies has revealed that the spatial distribution of tumor infiltrating lymphocytes (TILs) and other immune cells within and around the tumor can help stratify BC patients into responders and non-responders. However, clinical microscopy cannot differentiate between subtypes of TILs; numerous markers are needed to capture the heterogeneity of cancer cells and immune cells in the TME. Highly multiplexed fluorescence microscopy, or high-plex IF, has emerged as a workhorse in data collection for spatial proteomics. We present a pilot study of the TME of BC patients treated with neoadjuvant immunotherapy. Specifically in this abstract, we discuss computer vision methods for analyzing the cellular constituents probed in these complex and rich images. We discuss image stitching and channel registration for high-plex modalities, deep learning algorithms for cell detection and segmentation, and pseudo-spectral angle mapping (pSAM) for cell classification. We present strategies for accurate quantification of these images, facilitating investigations into immune activity in breast tumors with high phenotypic accuracy.
Single-cell sequencing and proteomics have been critical for the study of human disease. However, highly multiplexed microscopy has revolutionized spatial biology by measuring cell expression from ~50 proteins while maintaining spatial locations of cells. This presents unique computational challenges; acquiring manual annotations across so many image channels is challenging, therefore supervised learning methods for classification are undesirable. To overcome this limitation we have developed a decision-tree classifier for the multiclass annotation of renal cells that is analogous to well-established flow cytometry-based cell analyses. We demonstrate this method of cell annotation in a dataset of 54 kidney biopsies from patients with three different pathologies: 25 with lupus nephritis, 23 with renal allograft rejection, and six with non-autoimmune conditions. Biopsies were iteratively stained and imaged using the PhenoCycler protocol to acquire high-resolution, full-section images with a 43-marker panel. Nucleus segmentation was performed using Cellpose2.0 and whole cell segmentation was approximated by dilating the nucleus masks. In our decision tree, cells are sequentially sorted into marker-negative and marker-positive populations using their mean fluorescence intensity (MFI). A multi-Otsu threshold, in conjunction with manual spot checking, is used for determining the optimal MFI threshold for each branching of the decision tree. Marker order is based upon well-established, hierarchical expression of immunological cell markers created in consultation with expert immunologists. We have further developed another algorithm to probe microtubule organizing center polarization, an important immunologic behavior. Ultimately, we were able to assign biologically-defined cell classes to 1.59 million of 2.19 million cells captured in tissue.
Lupus nephritis (LN) is a severe manifestation of systemic lupus erythematosus, with up to 30% of LN patients progressing to end-stage kidney disease within ten years of diagnosis. Spatial relationships between specific types of immune cells and kidney structures hold valuable information clinically and biologically. Thus, we develop a modular computational pipeline to analyze the spatially resolved molecular features from high-plex immunofluorescence imaging data. Here, we present three modules of the pipeline, with the goal of achieving multiclass segmentation of renal cells and structures.
The goal of this work is to reduce the complexity of cell and cell neighborhood annotations in studies for spatial immunity. Specifically, we use a method inspired by spectral angle mapping to collapse multichannel images into class-level representations. We will demonstrate that these class maps assist in characterizing immune cell infiltration in renal pathologies.
Highly multiplexed fluorescence microscopy is an emerging technology that allows for spatial analysis of increasingly more classes of cells within human tissue—state-of-the-art methods are now probing up to 60 different protein markers within an image. This level of phenotypic resolution is ideal for uncovering the spatial underpinnings of immune cell interactions. However, defining cell types from this high-plex data is non-trivial. We present a method that borrows from hyperspectral image analysis to improve the accuracy and efficiency of immune cell classification in highly multiplexed fluorescence microscopy images. Treating the protein marker image channels as the spectral dimension of the images, we define reference “pseudospectra” representative of the ideal marker expression for all cell types of interest probed by the marker panel. Cosine similarity is computed for each reference pseudo-spectra to create class maps for each cell type in question. Features are extracted from these class maps—rather than the fluorescence images. We compare these methods to a decision-tree based classification method for classifying immune cells and unsupervised K-means clustering of mean pixel intensities across all image channels. We demonstrate that pSAM performs comparably, and potentially outperforms methods with similar levels of supervision.
SignificanceManual annotations are necessary for training supervised learning algorithms for object detection and instance segmentation. These manual annotations are difficult to acquire, noisy, and inconsistent across readers.AimThe goal of this work is to describe and demonstrate multireader generalizations of the Jaccard and Sørensen indices for object detection and instance segmentation.ApproachThe multireader Jaccard and Sørensen indices are described in terms of “calls,” “objects,” and number of readers. These generalizations reduce to the equations defined by confusion matrix variables in the two-reader case. In a test set of 50 cell microscopy images, we use these generalizations to assess reader variability and compare the performance of an object detection network (Yolov5) and an instance segmentation algorithm (Cellpose2.0) with a group of five human readers using the Mann–Whitney U-test with Bonferroni correction for multiplicity.ResultsThe multireader generalizations were statistically different from the mean of pairwise comparisons of readers (p < 0.0001). Further, these multireader generalizations informed when a reader was performing differently than the group. Finally, these generalizations show that Yolov5 and Cellpose2.0 performed similarly to the pool of human readers. The lower bound of the one-sided 90% confidence interval for the difference in the multireader Jaccard index between the pool of human readers and the pool of human readers plus an algorithm were −0.019 and −0.016 for Yolov5 and Cellpose2.0, respectively.ConclusionsMultireader generalizations of the Jaccard and Sørensen indices provide metrics for characterizing the agreement of an arbitrary number of readers on object detection and instance segmentation tasks.
Deep convolutional neural networks (CNNs) have demonstrated high accuracy in a wide range of computer vision applications, including medical and biological imaging. Many CNNs are fully supervised learning algorithms, and their performance is directly associated with the quality of the training data labels, which are human-defined. In this work, we investigate the fidelity of human-defined truth for cell detection, segmentation, and classification tasks in multiplex microscopy images. We compare manual annotations from human readers on three tasks. Readers were asked to (1) segment all cells in single-channel fluorescence images of a pannuclear stain (DAPI), (2) segment cells in two-channel fluorescence images (CD20/DAPI), only identifying cells with both nuclear signal (DAPI) and signal from a cell surface marker (CD20), and (3) segment two separate cell classes in three-channel fluorescence images (CD3/CD4/DAPI). In this third task, readers were asked to identify cells that had nuclear signal and were CD3+/CD4- and CD3+/CD4+. By comparing these manual segmentations within and between readers, we demonstrate that human readers show the least variability in single-channel DAPI segmentation (p<<0.05, F test for equal variance). We also compared the agreement of human readers with one another to the agreement of an object-detection network, Yolov5, on cell detection in DAPI images. All pairwise comparisons of human readers with other human readers yielded an average F1-score of 0.83±0.14, and comparisons of Yolov5 with human readers yielded an average F1-score of 0.84±0.12 (p=0.26, Welch’s T test). We therefore demonstrate that out of the provided tasks, DAPI detection provides the highest fidelity ground truth, and were unable to show a difference between Yolov5 and human readers in this task.
Systemic lupus erythematosus (SLE) is a complex, systemic autoimmune disease with many clinical presentations including lupus nephritis (LuN), or chronic inflammation of the kidneys. Current therapies for SLE are only modestly effective, highlighting the need to better understand networks of immune cells in SLE and LuN. In this work, we assess the performance of two convolutional neural network (CNN) architectures –Mask R-CNN and U-Net— in the task of instance segmentation of 5 immune-cell classes in 31 LuN biopsies. Each biopsy was stained for myeloid dendritic cells (mDCs), plasmacytoid dendritic cells (pDCs), B cells, and two populations of T cells, then imaged on a Leica SP8 fluorescence confocal microscope. Two instances of Mask R-CNN were trained on manually segmented images—one on lymphocytes (T cells and B cells), and one on DCs (pDCs and mDCs)—resulting in an average network sensitivities of 0.88 ± 0.04 and 0.82 ± 0.03, respectively. Five U-Nets, one for each of the five individual cell classes, were trained resulting in an average sensitivity of 0.85 ± 0.09 across all cell classes. Mask R-CNN yielded fewer false positives for all cell classes, with an average precision of 0.76 ± 0.03 compared to the U-Net object-level average precision of 0.43 ± 0.12. Overall, Mask R-CNN was more robust than the U-Net for segmenting cells in immunofluorescence images of kidney biopsies from lupus nephritis patients.
Lupus nephritis (LuN) is an inflammatory kidney disease characterized by the infiltration of immune cells into the kidney, including T-cells, B-cells, and dendritic cells. Here, we combine high-dimensional immunofluorescence microscopy with computer vision to identify and segment multiple populations of cells. A U-Net was trained to segment CD4+ T-cells in high-resolution LuN biopsy images and subsequently used to make CD4+ T-cell predictions on a test-set from a lower-resolution, high-dimensional LuN dataset. This produced higher precision, but lower recall and intersection over union for cells in the low-resolution dataset. Further application of U-Nets to immune cell segmentation will be discussed.
Several disease states, including cancer and autoimmunity, are characterized by the infiltration of large populations of immune cells into organ tissue. The degree and composition of these invading cells have been correlated with patient outcomes, suggesting that the intercellular interactions occurring in inflamed tissue play a role in pathology. Immunofluorescence staining paired with confocal microscopy produce detailed visualizations of these interactions. Applying computer vision and machine learning methods to the resulting images allows for robust quantification of immune infiltrates. We are developing an analytical pipeline to assess the immune environments of two distinct disease states: lupus nephritis and triple-negative breast cancer (TNBC). Biopsies of inflamed kidney tissue (lupus) and tumors (TNBC) were stained and imaged for panels of 20 markers using a strip-reprobe technique. This set of markers interrogates populations of T-cells, B-cells, and antigen presenting cells. To detect T cells, we first trained a U-Net to segment CD3+CD4+ T-cells in images of lupus biopsies and achieved an object-level precision of 0.855 and recall of 0.607 on an independent test set. We then evaluated the generalizability of this network to CD3+CD8+ T cells in lupus nephritis and CD3+CD4+ T cells in TNBC, and the extent to which fine-tuning the network improved performance for these cell types. We found that recall increased moderately with finetuning, while precision did not. Further work will focus on developing robust methods of segmenting a larger variety of T cell markers in both tissue contexts with high fidelity.
Lupus nephritis (LuN) is an autoimmune disease characterized by chronic kidney inflammation, which can lead to loss of kidney function, known as end-stage renal disease. The cellular mechanisms causing this progression are not well-defined. Radiomic texture analysis was used to identify image features of biopsies from ESRD+ and ESRD- LuN patients. Each biopsy was stained with 6 markers to identify 5 cell classes in fluorescence confocal microscopy images. Image features associated with the CD20 stain (B lymphocytes), image summary metrics of mean and standard deviation, and 4 GLCM features were identified as most effective in classifying ESRD+/- biopsy images.
Lupus nephritis (LuN) is a manifestation of systemic lupus erythematosus defined by chronic infiltration of immune cells into the kidneys—particularly lymphocytes and dendritic cells (DCs). Ultimately, our goal is to characterize the cellular communities associated with progression to kidney failure. To accomplish this, we have generated a dataset of fluorescence confocal microscopy images of kidney biopsies from 31 LuN patients that have been stained for two T-lymphocyte populations, B-lymphocytes and two DC populations. We are using convolutional neural networks (CNNs) with a Mask R-CNN architecture to perform instance segmentation on these five classes. This multi-class instance segmentation task is hindered by an inherent class imbalance between lymphocytes and DCs, with DCs being much less prevalent. Here we discuss methods for managing class imbalance to achieve comparable instance segmentation of both DCs and lymphocytes in LuN biopsies. A network trained to identify all 5 classes yielded higher sensitivity to DCs when the training set was filtered to contain images with all 5 cell classes present. Average DC sensitivity on an independent test set improved from 0.54 to 0.63 with filtered training data. DC segmentation improved further when the network was trained specifically for DC classes. Average DC sensitivity reached 0.91 when trained separately from lymphocytes, with average Jaccard index of DCs improving from 0.69±0.2 to 0.76±0.2. Accurate segmentation of all cell types relevant to LuN pathogenesis enabled in-depth spatial analysis of the immune environments that result in renal failure in LuN patients.
Significance: Lupus nephritis (LuN) is a chronic inflammatory kidney disease. The cellular mechanisms by which LuN progresses to kidney failure are poorly characterized. Automated instance segmentation of immune cells in immunofluorescence images of LuN can probe these cellular interactions.
Aim: Our specific goal is to quantify how sample fixation and staining panel design impact automated instance segmentation and characterization of immune cells.
Approach: Convolutional neural networks (CNNs) were trained to segment immune cells in fluorescence confocal images of LuN biopsies. Three datasets were used to probe the effects of fixation methods on cell features and the effects of one-marker versus two-marker per cell staining panels on CNN performance.
Results: Networks trained for multi-class instance segmentation on fresh-frozen and formalin-fixed, paraffin-embedded (FFPE) samples stained with a two-marker panel had sensitivities of 0.87 and 0.91 and specificities of 0.82 and 0.88, respectively. Training on samples with a one-marker panel reduced sensitivity (0.72). Cell size and intercellular distances were significantly smaller in FFPE samples compared to fresh frozen (Kolmogorov–Smirnov, p ≪ 0.0001).
Conclusions: Fixation method significantly reduces cell size and intercellular distances in LuN biopsies. The use of two markers to identify cell subsets showed improved CNN sensitivity relative to using a single marker.
We demonstrate an instance segmentation method with Mask R-CNN using a ResNet-101 plus Feature Pyramid Network convolutional backbone to segment and classify T cells and antigen presenting cells (APCs) in multi-channel fluorescence confocal images. This network improves on our previous cell distance mapping (CDM) pipeline, which used a custom 10- layer convolutional neural network for cell segmentation. We have validated Mask R-CNN on two independent datasets of fluorescence confocal images: 1) mouse lymph node tissue, and 2) human lupus nephritis (LuN) biopsies. For dataset 1, mice were injected with fluorescent dendritic cells and two populations of fluorescent T cells. Mask R-CNN improved sensitivity averaged across all cell types from 0.88 to 0.94. Specificity improved from 0.92 to 0.95 across all cell types, and intersection over union score (IOU) improved significantly from 0.82 to 0.86 (p < 0.0001). Human LuN biopsies in dataset 2 were stained with two T cell markers and two APC markers, with separate staining panels to identify different populations of APCs. Mask R-CNN again improved segmentation and classification averaged across all cell types, increasing overall sensitivity from 0.72 to 0.76, specificity from 0.86 to 0.93, and significantly increasing IOU from 0.71 to 0.81 (p < 0.0001). Improved IOU scores are particularly important in CDM to be able to quantify cell shape for identification of functional interactions of immune cells. Mask R-CNN is therefore a superior method for instance segmentation of immune cells in microscopy images for image analysis of cellular function in pathological immune states.
Computer vision and deep learning are integral tools in the improvement of high-throughput analysis of cellular images. Specifically, optimization of algorithms for object detection and instance segmentation tasks are important in cellular image analysis to segment and classify multi-object, multi-class images. In this work, we employ an instance segmentation pipeline with Mask RCNN, using a ResNet-101 and Feature Pyramid Network convolutional backbone to segment and classify T cells and antigen presenting cells (APCs) in multi-channel fluorescence confocal images of lupus nephritis biopsies. This task was first performed on a dataset of fresh frozen biopsies stained for T cells (CD3 and CD4) and two APC populations: 1) myeloid dendritic cells (BDCA1 and CD11c), and 2) plasmacytoid dendritic cells (BDCA2 and CD123). The network achieved an average sensitivity of 0.82, specificity of 0.91, and Jaccard index of 0.79 across all cell types. However, relative to fresh frozen tissue, samples prepared through formalin fixation and paraffin embedding (FFPE) provide larger potential datasets for investigating immune activity. Training this same network architecture on an FFPE database of lupus nephritis tissue stained with the same antibody panel, the network achieved an average sensitivity of 0.82, specificity of 0.92, and Jaccard index of 0.77 across all cell types. In addition to working with FFPE tissue, it would also be beneficial to identify APCs with a single stain and image more cell types with a single staining panel. We have trained this network on a single-stained APC panel FFPE dataset to achieve an average sensitivity of 0.79, specificity of 0.86, and Jaccard index of 0.63 across all cell types. These three trained networks were used to assess differences in cell shape features between fixation and staining protocols.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.