PurposeThe rapid development of highly multiplexed microscopy has enabled the study of cells embedded within their native tissue. The rich spatial data provided by these techniques have yielded exciting insights into the spatial features of human disease. However, computational methods for analyzing these high-content images are still emerging; there is a need for more robust and generalizable tools for evaluating the cellular constituents and stroma captured by high-plex imaging. To address this need, we have adapted spectral angle mapping—an algorithm developed for hyperspectral image analysis—to compress the channel dimension of high-plex immunofluorescence (IF) images.ApproachHere, we present pseudo-spectral angle mapping (pSAM), a robust and flexible method for determining the most likely class of each pixel in a high-plex image. The class maps calculated through pSAM yield pixel classifications which can be combined with instance segmentation algorithms to classify individual cells.ResultsIn a dataset of colon biopsies imaged with a 13-plex staining panel, 16 pSAM class maps were computed to generate pixel classifications. Instance segmentations of cells with Cellpose2.0 (F1-score of 0.83±0.13) were combined with these class maps to provide cell class predictions for 13 cell classes. In addition, in a separate unseen dataset of kidney biopsies imaged with a 44-plex staining panel, pSAM plus Cellpose2.0 (F1-score of 0.86±0.11) detected a diverse set of 38 classes of structural and immune cells.ConclusionsIn summary, pSAM is a powerful and generalizable tool for evaluating high-plex IF image data and classifying cells in these high-dimensional images.
Breast cancer (BC) remains the deadliest cancer for women worldwide. Neoadjuvant immunotherapies have demonstrated improved responses for some patients. Unfortunately, no robust method exists for predicting which patients will respond to immunotherapy. Imaging of diagnostic BC biopsies has revealed that the spatial distribution of tumor infiltrating lymphocytes (TILs) and other immune cells within and around the tumor can help stratify BC patients into responders and non-responders. However, clinical microscopy cannot differentiate between subtypes of TILs; numerous markers are needed to capture the heterogeneity of cancer cells and immune cells in the TME. Highly multiplexed fluorescence microscopy, or high-plex IF, has emerged as a workhorse in data collection for spatial proteomics. We present a pilot study of the TME of BC patients treated with neoadjuvant immunotherapy. Specifically in this abstract, we discuss computer vision methods for analyzing the cellular constituents probed in these complex and rich images. We discuss image stitching and channel registration for high-plex modalities, deep learning algorithms for cell detection and segmentation, and pseudo-spectral angle mapping (pSAM) for cell classification. We present strategies for accurate quantification of these images, facilitating investigations into immune activity in breast tumors with high phenotypic accuracy.
The goal of this work is to reduce the complexity of cell and cell neighborhood annotations in studies for spatial immunity. Specifically, we use a method inspired by spectral angle mapping to collapse multichannel images into class-level representations. We will demonstrate that these class maps assist in characterizing immune cell infiltration in renal pathologies.
Single-cell sequencing and proteomics have been critical for the study of human disease. However, highly multiplexed microscopy has revolutionized spatial biology by measuring cell expression from ~50 proteins while maintaining spatial locations of cells. This presents unique computational challenges; acquiring manual annotations across so many image channels is challenging, therefore supervised learning methods for classification are undesirable. To overcome this limitation we have developed a decision-tree classifier for the multiclass annotation of renal cells that is analogous to well-established flow cytometry-based cell analyses. We demonstrate this method of cell annotation in a dataset of 54 kidney biopsies from patients with three different pathologies: 25 with lupus nephritis, 23 with renal allograft rejection, and six with non-autoimmune conditions. Biopsies were iteratively stained and imaged using the PhenoCycler protocol to acquire high-resolution, full-section images with a 43-marker panel. Nucleus segmentation was performed using Cellpose2.0 and whole cell segmentation was approximated by dilating the nucleus masks. In our decision tree, cells are sequentially sorted into marker-negative and marker-positive populations using their mean fluorescence intensity (MFI). A multi-Otsu threshold, in conjunction with manual spot checking, is used for determining the optimal MFI threshold for each branching of the decision tree. Marker order is based upon well-established, hierarchical expression of immunological cell markers created in consultation with expert immunologists. We have further developed another algorithm to probe microtubule organizing center polarization, an important immunologic behavior. Ultimately, we were able to assign biologically-defined cell classes to 1.59 million of 2.19 million cells captured in tissue.
Lupus nephritis (LN) is a severe manifestation of systemic lupus erythematosus, with up to 30% of LN patients progressing to end-stage kidney disease within ten years of diagnosis. Spatial relationships between specific types of immune cells and kidney structures hold valuable information clinically and biologically. Thus, we develop a modular computational pipeline to analyze the spatially resolved molecular features from high-plex immunofluorescence imaging data. Here, we present three modules of the pipeline, with the goal of achieving multiclass segmentation of renal cells and structures.
Highly multiplexed fluorescence microscopy is an emerging technology that allows for spatial analysis of increasingly more classes of cells within human tissue—state-of-the-art methods are now probing up to 60 different protein markers within an image. This level of phenotypic resolution is ideal for uncovering the spatial underpinnings of immune cell interactions. However, defining cell types from this high-plex data is non-trivial. We present a method that borrows from hyperspectral image analysis to improve the accuracy and efficiency of immune cell classification in highly multiplexed fluorescence microscopy images. Treating the protein marker image channels as the spectral dimension of the images, we define reference “pseudospectra” representative of the ideal marker expression for all cell types of interest probed by the marker panel. Cosine similarity is computed for each reference pseudo-spectra to create class maps for each cell type in question. Features are extracted from these class maps—rather than the fluorescence images. We compare these methods to a decision-tree based classification method for classifying immune cells and unsupervised K-means clustering of mean pixel intensities across all image channels. We demonstrate that pSAM performs comparably, and potentially outperforms methods with similar levels of supervision.
SignificanceManual annotations are necessary for training supervised learning algorithms for object detection and instance segmentation. These manual annotations are difficult to acquire, noisy, and inconsistent across readers.AimThe goal of this work is to describe and demonstrate multireader generalizations of the Jaccard and Sørensen indices for object detection and instance segmentation.ApproachThe multireader Jaccard and Sørensen indices are described in terms of “calls,” “objects,” and number of readers. These generalizations reduce to the equations defined by confusion matrix variables in the two-reader case. In a test set of 50 cell microscopy images, we use these generalizations to assess reader variability and compare the performance of an object detection network (Yolov5) and an instance segmentation algorithm (Cellpose2.0) with a group of five human readers using the Mann–Whitney U-test with Bonferroni correction for multiplicity.ResultsThe multireader generalizations were statistically different from the mean of pairwise comparisons of readers (p < 0.0001). Further, these multireader generalizations informed when a reader was performing differently than the group. Finally, these generalizations show that Yolov5 and Cellpose2.0 performed similarly to the pool of human readers. The lower bound of the one-sided 90% confidence interval for the difference in the multireader Jaccard index between the pool of human readers and the pool of human readers plus an algorithm were −0.019 and −0.016 for Yolov5 and Cellpose2.0, respectively.ConclusionsMultireader generalizations of the Jaccard and Sørensen indices provide metrics for characterizing the agreement of an arbitrary number of readers on object detection and instance segmentation tasks.
Deep convolutional neural networks (CNNs) have demonstrated high accuracy in a wide range of computer vision applications, including medical and biological imaging. Many CNNs are fully supervised learning algorithms, and their performance is directly associated with the quality of the training data labels, which are human-defined. In this work, we investigate the fidelity of human-defined truth for cell detection, segmentation, and classification tasks in multiplex microscopy images. We compare manual annotations from human readers on three tasks. Readers were asked to (1) segment all cells in single-channel fluorescence images of a pannuclear stain (DAPI), (2) segment cells in two-channel fluorescence images (CD20/DAPI), only identifying cells with both nuclear signal (DAPI) and signal from a cell surface marker (CD20), and (3) segment two separate cell classes in three-channel fluorescence images (CD3/CD4/DAPI). In this third task, readers were asked to identify cells that had nuclear signal and were CD3+/CD4- and CD3+/CD4+. By comparing these manual segmentations within and between readers, we demonstrate that human readers show the least variability in single-channel DAPI segmentation (p<<0.05, F test for equal variance). We also compared the agreement of human readers with one another to the agreement of an object-detection network, Yolov5, on cell detection in DAPI images. All pairwise comparisons of human readers with other human readers yielded an average F1-score of 0.83±0.14, and comparisons of Yolov5 with human readers yielded an average F1-score of 0.84±0.12 (p=0.26, Welch’s T test). We therefore demonstrate that out of the provided tasks, DAPI detection provides the highest fidelity ground truth, and were unable to show a difference between Yolov5 and human readers in this task.
Triple-negative breast cancer (TNBC) is defined by a lack of biomarkers in the tumor. This inherent lack of targets results in a lack of effective therapeutic tools. However, immunotherapies have shown promise in treating TNBC. Here, we present computer vision methods for automatic detection of immune cells and larger immune structures in TNBC. We demonstrate accurate cell detection and segmentation in highly-multiplexed, whole-slide images of TNBC biopsies. Additionally, we show preliminary spatial analyses that identify and characterize tertiary lymphoid structures within the tumor. Ultimately, we hope to implement these methods to predict responders and non-responders to immunotherapy regimens for TNBC.
Triple-negative breast cancer (TNBC) is an aggressive subtype of breast cancer defined by the lack of hormone receptor overexpression. TNBC patients are at a higher risk of recurrence than patients with other breast cancers. As this disease disproportionally affects young women of color, there is an urgent need to address this health inequity by improving disease detection, prognostication, and therapy guidance. Currently, TNBC patients are expected to have better prognosis if a biopsy analysis shows more tumor-infiltrating immune cells. However, immunotherapies such as PD-L1 checkpoint blockade have shown variable efficacy. Studies on the immune constituents of a tumor with high phenotypic resolution have largely been reliant on tissue-destructive methods. Immunofluorescence microscopy, which conserves the spatial distribution of cells, is traditionally limited in collecting high numbers of colocalized antibody markers, which limits the phenotypic specificity of a spatial analysis of tumor immunity. To better understand the immune landscape of TNBC, we have collected highly multiplexed immunofluorescence microscopy images of 19 TNBC samples with 20 cellular markers using an iterative staining and imaging method. Here, we show the generalizability of pre-trained convolutional neural networks to T cell segmentation in TNBC, and further improve these algorithms through fine-tuning. We demonstrate a significant improvement in sensitivity after fine-tuning with domain-specific data (p < 0.05, Mann-Whitney U-test with Bonferroni correction). Additionally, we demonstrate that augmenting the fine-tuning dataset with training images from a different pathology can significantly improves cell detection performance.
Systemic lupus erythematosus (SLE) is a complex, systemic autoimmune disease with many clinical presentations including lupus nephritis (LuN), or chronic inflammation of the kidneys. Current therapies for SLE are only modestly effective, highlighting the need to better understand networks of immune cells in SLE and LuN. In this work, we assess the performance of two convolutional neural network (CNN) architectures –Mask R-CNN and U-Net— in the task of instance segmentation of 5 immune-cell classes in 31 LuN biopsies. Each biopsy was stained for myeloid dendritic cells (mDCs), plasmacytoid dendritic cells (pDCs), B cells, and two populations of T cells, then imaged on a Leica SP8 fluorescence confocal microscope. Two instances of Mask R-CNN were trained on manually segmented images—one on lymphocytes (T cells and B cells), and one on DCs (pDCs and mDCs)—resulting in an average network sensitivities of 0.88 ± 0.04 and 0.82 ± 0.03, respectively. Five U-Nets, one for each of the five individual cell classes, were trained resulting in an average sensitivity of 0.85 ± 0.09 across all cell classes. Mask R-CNN yielded fewer false positives for all cell classes, with an average precision of 0.76 ± 0.03 compared to the U-Net object-level average precision of 0.43 ± 0.12. Overall, Mask R-CNN was more robust than the U-Net for segmenting cells in immunofluorescence images of kidney biopsies from lupus nephritis patients.
Lupus nephritis (LuN) is an inflammatory kidney disease characterized by the infiltration of immune cells into the kidney, including T-cells, B-cells, and dendritic cells. Here, we combine high-dimensional immunofluorescence microscopy with computer vision to identify and segment multiple populations of cells. A U-Net was trained to segment CD4+ T-cells in high-resolution LuN biopsy images and subsequently used to make CD4+ T-cell predictions on a test-set from a lower-resolution, high-dimensional LuN dataset. This produced higher precision, but lower recall and intersection over union for cells in the low-resolution dataset. Further application of U-Nets to immune cell segmentation will be discussed.
Lupus nephritis (LuN) is a manifestation of systemic lupus erythematosus defined by chronic infiltration of immune cells into the kidneys—particularly lymphocytes and dendritic cells (DCs). Ultimately, our goal is to characterize the cellular communities associated with progression to kidney failure. To accomplish this, we have generated a dataset of fluorescence confocal microscopy images of kidney biopsies from 31 LuN patients that have been stained for two T-lymphocyte populations, B-lymphocytes and two DC populations. We are using convolutional neural networks (CNNs) with a Mask R-CNN architecture to perform instance segmentation on these five classes. This multi-class instance segmentation task is hindered by an inherent class imbalance between lymphocytes and DCs, with DCs being much less prevalent. Here we discuss methods for managing class imbalance to achieve comparable instance segmentation of both DCs and lymphocytes in LuN biopsies. A network trained to identify all 5 classes yielded higher sensitivity to DCs when the training set was filtered to contain images with all 5 cell classes present. Average DC sensitivity on an independent test set improved from 0.54 to 0.63 with filtered training data. DC segmentation improved further when the network was trained specifically for DC classes. Average DC sensitivity reached 0.91 when trained separately from lymphocytes, with average Jaccard index of DCs improving from 0.69±0.2 to 0.76±0.2. Accurate segmentation of all cell types relevant to LuN pathogenesis enabled in-depth spatial analysis of the immune environments that result in renal failure in LuN patients.
Several disease states, including cancer and autoimmunity, are characterized by the infiltration of large populations of immune cells into organ tissue. The degree and composition of these invading cells have been correlated with patient outcomes, suggesting that the intercellular interactions occurring in inflamed tissue play a role in pathology. Immunofluorescence staining paired with confocal microscopy produce detailed visualizations of these interactions. Applying computer vision and machine learning methods to the resulting images allows for robust quantification of immune infiltrates. We are developing an analytical pipeline to assess the immune environments of two distinct disease states: lupus nephritis and triple-negative breast cancer (TNBC). Biopsies of inflamed kidney tissue (lupus) and tumors (TNBC) were stained and imaged for panels of 20 markers using a strip-reprobe technique. This set of markers interrogates populations of T-cells, B-cells, and antigen presenting cells. To detect T cells, we first trained a U-Net to segment CD3+CD4+ T-cells in images of lupus biopsies and achieved an object-level precision of 0.855 and recall of 0.607 on an independent test set. We then evaluated the generalizability of this network to CD3+CD8+ T cells in lupus nephritis and CD3+CD4+ T cells in TNBC, and the extent to which fine-tuning the network improved performance for these cell types. We found that recall increased moderately with finetuning, while precision did not. Further work will focus on developing robust methods of segmenting a larger variety of T cell markers in both tissue contexts with high fidelity.
Lupus nephritis (LuN) is an autoimmune disease characterized by chronic kidney inflammation, which can lead to loss of kidney function, known as end-stage renal disease. The cellular mechanisms causing this progression are not well-defined. Radiomic texture analysis was used to identify image features of biopsies from ESRD+ and ESRD- LuN patients. Each biopsy was stained with 6 markers to identify 5 cell classes in fluorescence confocal microscopy images. Image features associated with the CD20 stain (B lymphocytes), image summary metrics of mean and standard deviation, and 4 GLCM features were identified as most effective in classifying ESRD+/- biopsy images.
Significance: Lupus nephritis (LuN) is a chronic inflammatory kidney disease. The cellular mechanisms by which LuN progresses to kidney failure are poorly characterized. Automated instance segmentation of immune cells in immunofluorescence images of LuN can probe these cellular interactions.
Aim: Our specific goal is to quantify how sample fixation and staining panel design impact automated instance segmentation and characterization of immune cells.
Approach: Convolutional neural networks (CNNs) were trained to segment immune cells in fluorescence confocal images of LuN biopsies. Three datasets were used to probe the effects of fixation methods on cell features and the effects of one-marker versus two-marker per cell staining panels on CNN performance.
Results: Networks trained for multi-class instance segmentation on fresh-frozen and formalin-fixed, paraffin-embedded (FFPE) samples stained with a two-marker panel had sensitivities of 0.87 and 0.91 and specificities of 0.82 and 0.88, respectively. Training on samples with a one-marker panel reduced sensitivity (0.72). Cell size and intercellular distances were significantly smaller in FFPE samples compared to fresh frozen (Kolmogorov–Smirnov, p ≪ 0.0001).
Conclusions: Fixation method significantly reduces cell size and intercellular distances in LuN biopsies. The use of two markers to identify cell subsets showed improved CNN sensitivity relative to using a single marker.
Triple-negative breast cancer (TNBC) is an aggressive subtype of breast cancer characterized by a high level of inflammation. In this work, we present a computational method for assessing immune activity across many populations of immune cells. For this analysis, we have a dataset of 18 TNBC biopsies, each stained with a ‘strip-and-reprobe’ approach consisting of multiple rounds of staining and imaging the tissue. Each sample is stained with 8 separate antibody panels, resulting in ~20 different stains per sample, plus tissue autofluorescence images. Biopsy sections are imaged across a ~4mm x 5mm area on a Caliber ID confocal microscope with a pixel size of 221 nm, resulting in image composites of ~27,000x27,000x20 pixels per sample. Using radiomic texture analysis, we have analyzed a subset of these images to determine which texture features are predictive of inflammation. From 3 of these samples, we have selected inflamed and uninflamed regions of interest (ROIs) for three lymphocyte markers: 1) CD4, 2) CD3, and 3) CD8. We have computed 20 texture features, specifically gray level co-occurrence matrix features, for each of these 42 ROIs and their corresponding tissue autofluorescence images. From this analysis, we have found multiple features that successfully differentiate inflamed tissue from uninflamed tissue. These features will be calculated across the full dataset to analyze the overall TNBC immune architecture of these samples as diffuse or compartmentalized, and which cell populations tend to coaggregate.
We demonstrate an instance segmentation method with Mask R-CNN using a ResNet-101 plus Feature Pyramid Network convolutional backbone to segment and classify T cells and antigen presenting cells (APCs) in multi-channel fluorescence confocal images. This network improves on our previous cell distance mapping (CDM) pipeline, which used a custom 10- layer convolutional neural network for cell segmentation. We have validated Mask R-CNN on two independent datasets of fluorescence confocal images: 1) mouse lymph node tissue, and 2) human lupus nephritis (LuN) biopsies. For dataset 1, mice were injected with fluorescent dendritic cells and two populations of fluorescent T cells. Mask R-CNN improved sensitivity averaged across all cell types from 0.88 to 0.94. Specificity improved from 0.92 to 0.95 across all cell types, and intersection over union score (IOU) improved significantly from 0.82 to 0.86 (p < 0.0001). Human LuN biopsies in dataset 2 were stained with two T cell markers and two APC markers, with separate staining panels to identify different populations of APCs. Mask R-CNN again improved segmentation and classification averaged across all cell types, increasing overall sensitivity from 0.72 to 0.76, specificity from 0.86 to 0.93, and significantly increasing IOU from 0.71 to 0.81 (p < 0.0001). Improved IOU scores are particularly important in CDM to be able to quantify cell shape for identification of functional interactions of immune cells. Mask R-CNN is therefore a superior method for instance segmentation of immune cells in microscopy images for image analysis of cellular function in pathological immune states.
Computer vision and deep learning are integral tools in the improvement of high-throughput analysis of cellular images. Specifically, optimization of algorithms for object detection and instance segmentation tasks are important in cellular image analysis to segment and classify multi-object, multi-class images. In this work, we employ an instance segmentation pipeline with Mask RCNN, using a ResNet-101 and Feature Pyramid Network convolutional backbone to segment and classify T cells and antigen presenting cells (APCs) in multi-channel fluorescence confocal images of lupus nephritis biopsies. This task was first performed on a dataset of fresh frozen biopsies stained for T cells (CD3 and CD4) and two APC populations: 1) myeloid dendritic cells (BDCA1 and CD11c), and 2) plasmacytoid dendritic cells (BDCA2 and CD123). The network achieved an average sensitivity of 0.82, specificity of 0.91, and Jaccard index of 0.79 across all cell types. However, relative to fresh frozen tissue, samples prepared through formalin fixation and paraffin embedding (FFPE) provide larger potential datasets for investigating immune activity. Training this same network architecture on an FFPE database of lupus nephritis tissue stained with the same antibody panel, the network achieved an average sensitivity of 0.82, specificity of 0.92, and Jaccard index of 0.77 across all cell types. In addition to working with FFPE tissue, it would also be beneficial to identify APCs with a single stain and image more cell types with a single staining panel. We have trained this network on a single-stained APC panel FFPE dataset to achieve an average sensitivity of 0.79, specificity of 0.86, and Jaccard index of 0.63 across all cell types. These three trained networks were used to assess differences in cell shape features between fixation and staining protocols.
The feasibility of localizing, segmenting, and classifying individual cells in multi-channel confocal microscopy images is highly dependent on image quality. In certain applications, with good image quality, the segmentation can be trivial and accomplished via thresholding, watershed, or a collection of other well-established and studied heuristics; however, at the limit of poor image quality and complex image features, these techniques fail. It is at this limit that deep convolutional neural network (DCNN) approaches excel. Our research studies the interaction of individual immune cells and their shape changes relative to inflammatory immune reactions1 using multi-channel immunofluorescence imaging of renal biopsies from patients with inflammatory kidney disease. We present here a deep learning methodology for application to nuclear and cell membrane immunofluorescent stains to automatically segment and classify multiple T-cell and dendritic cell types. With both T-cells and dendritic cells segmented, we are able to study how T-cells with different surface antigens change shape with proximity to dendritic cells. Shape changes are seen when T-cells move close to dendritic cells and interact. We use a sliding window, max-filtering DCNN to segment and classify 3 cell types from 6 image stains channels within a single DCNN. This DCNN maintains images at original resolution throughout the network using dilated convolutions and max-filtering in place of max pooling layers. In addition, we use 3D convolution kernels with two spatial dimensions and one channel dimension. This allows us to output a multi-class binary classification based on multichannel data at the original image resolution. We trained and validated the network across 24 patients with 8,572 segmented cells. Our results demonstrate a mean Dice-Sorensen score of 0.78 +/- 0.18, a mean classification sensitivity of 0.76, and a mean classification specificity of 0.75 across all 3 segmented cell types.
Analysis of interactions between B and T cells in tubulointerstitial inflammation is important for understanding human
lupus nephritis. We developed a computer technique to perform this analysis, and compared it with manual analysis.
Multi-channel immunoflourescent-microscopy images were acquired from 207 regions of interest in 40 renal tissue
sections of 19 patients diagnosed with lupus nephritis. Fresh-frozen renal tissue sections were stained with combinations
of immunoflourescent antibodies to membrane proteins and counter-stained with a cell nuclear marker. Manual
delineation of the antibodies was considered as the reference standard. We first segmented cell nuclei and cell
membrane markers, and then determined corresponding cell types based on the distances between cell nuclei and
specific cell-membrane marker combinations. Subsequently, the distribution of the shortest distance from T cell nuclei
to B cell nuclei was obtained and used as a surrogate indicator of cell-cell interactions. The computer and manual
analyses results were concordant. The average absolute difference was 1.1±1.2% between the computer and manual
analysis results in the number of cell-cell distances of 3 μm or less as a percentage of the total number of cell-cell
distances. Our computerized analysis of cell-cell distances could be used as a surrogate for quantifying cell-cell
interactions as either an automated and quantitative analysis or for independent confirmation of manual analysis.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.