Deep convolutional neural networks (CNNs) have demonstrated high accuracy in a wide range of computer vision applications, including medical and biological imaging. Many CNNs are fully supervised learning algorithms, and their performance is directly associated with the quality of the training data labels, which are human-defined. In this work, we investigate the fidelity of human-defined truth for cell detection, segmentation, and classification tasks in multiplex microscopy images. We compare manual annotations from human readers on three tasks. Readers were asked to (1) segment all cells in single-channel fluorescence images of a pannuclear stain (DAPI), (2) segment cells in two-channel fluorescence images (CD20/DAPI), only identifying cells with both nuclear signal (DAPI) and signal from a cell surface marker (CD20), and (3) segment two separate cell classes in three-channel fluorescence images (CD3/CD4/DAPI). In this third task, readers were asked to identify cells that had nuclear signal and were CD3+/CD4- and CD3+/CD4+. By comparing these manual segmentations within and between readers, we demonstrate that human readers show the least variability in single-channel DAPI segmentation (p<<0.05, F test for equal variance). We also compared the agreement of human readers with one another to the agreement of an object-detection network, Yolov5, on cell detection in DAPI images. All pairwise comparisons of human readers with other human readers yielded an average F1-score of 0.83±0.14, and comparisons of Yolov5 with human readers yielded an average F1-score of 0.84±0.12 (p=0.26, Welch’s T test). We therefore demonstrate that out of the provided tasks, DAPI detection provides the highest fidelity ground truth, and were unable to show a difference between Yolov5 and human readers in this task.
Triple-negative breast cancer (TNBC) is defined by a lack of biomarkers in the tumor. This inherent lack of targets results in a lack of effective therapeutic tools. However, immunotherapies have shown promise in treating TNBC. Here, we present computer vision methods for automatic detection of immune cells and larger immune structures in TNBC. We demonstrate accurate cell detection and segmentation in highly-multiplexed, whole-slide images of TNBC biopsies. Additionally, we show preliminary spatial analyses that identify and characterize tertiary lymphoid structures within the tumor. Ultimately, we hope to implement these methods to predict responders and non-responders to immunotherapy regimens for TNBC.
Triple-negative breast cancer (TNBC) is an aggressive subtype of breast cancer defined by the lack of hormone receptor overexpression. TNBC patients are at a higher risk of recurrence than patients with other breast cancers. As this disease disproportionally affects young women of color, there is an urgent need to address this health inequity by improving disease detection, prognostication, and therapy guidance. Currently, TNBC patients are expected to have better prognosis if a biopsy analysis shows more tumor-infiltrating immune cells. However, immunotherapies such as PD-L1 checkpoint blockade have shown variable efficacy. Studies on the immune constituents of a tumor with high phenotypic resolution have largely been reliant on tissue-destructive methods. Immunofluorescence microscopy, which conserves the spatial distribution of cells, is traditionally limited in collecting high numbers of colocalized antibody markers, which limits the phenotypic specificity of a spatial analysis of tumor immunity. To better understand the immune landscape of TNBC, we have collected highly multiplexed immunofluorescence microscopy images of 19 TNBC samples with 20 cellular markers using an iterative staining and imaging method. Here, we show the generalizability of pre-trained convolutional neural networks to T cell segmentation in TNBC, and further improve these algorithms through fine-tuning. We demonstrate a significant improvement in sensitivity after fine-tuning with domain-specific data (p < 0.05, Mann-Whitney U-test with Bonferroni correction). Additionally, we demonstrate that augmenting the fine-tuning dataset with training images from a different pathology can significantly improves cell detection performance.
Systemic lupus erythematosus (SLE) is a complex, systemic autoimmune disease with many clinical presentations including lupus nephritis (LuN), or chronic inflammation of the kidneys. Current therapies for SLE are only modestly effective, highlighting the need to better understand networks of immune cells in SLE and LuN. In this work, we assess the performance of two convolutional neural network (CNN) architectures –Mask R-CNN and U-Net— in the task of instance segmentation of 5 immune-cell classes in 31 LuN biopsies. Each biopsy was stained for myeloid dendritic cells (mDCs), plasmacytoid dendritic cells (pDCs), B cells, and two populations of T cells, then imaged on a Leica SP8 fluorescence confocal microscope. Two instances of Mask R-CNN were trained on manually segmented images—one on lymphocytes (T cells and B cells), and one on DCs (pDCs and mDCs)—resulting in an average network sensitivities of 0.88 ± 0.04 and 0.82 ± 0.03, respectively. Five U-Nets, one for each of the five individual cell classes, were trained resulting in an average sensitivity of 0.85 ± 0.09 across all cell classes. Mask R-CNN yielded fewer false positives for all cell classes, with an average precision of 0.76 ± 0.03 compared to the U-Net object-level average precision of 0.43 ± 0.12. Overall, Mask R-CNN was more robust than the U-Net for segmenting cells in immunofluorescence images of kidney biopsies from lupus nephritis patients.
Lupus nephritis (LuN) is an inflammatory kidney disease characterized by the infiltration of immune cells into the kidney, including T-cells, B-cells, and dendritic cells. Here, we combine high-dimensional immunofluorescence microscopy with computer vision to identify and segment multiple populations of cells. A U-Net was trained to segment CD4+ T-cells in high-resolution LuN biopsy images and subsequently used to make CD4+ T-cell predictions on a test-set from a lower-resolution, high-dimensional LuN dataset. This produced higher precision, but lower recall and intersection over union for cells in the low-resolution dataset. Further application of U-Nets to immune cell segmentation will be discussed.
Several disease states, including cancer and autoimmunity, are characterized by the infiltration of large populations of immune cells into organ tissue. The degree and composition of these invading cells have been correlated with patient outcomes, suggesting that the intercellular interactions occurring in inflamed tissue play a role in pathology. Immunofluorescence staining paired with confocal microscopy produce detailed visualizations of these interactions. Applying computer vision and machine learning methods to the resulting images allows for robust quantification of immune infiltrates. We are developing an analytical pipeline to assess the immune environments of two distinct disease states: lupus nephritis and triple-negative breast cancer (TNBC). Biopsies of inflamed kidney tissue (lupus) and tumors (TNBC) were stained and imaged for panels of 20 markers using a strip-reprobe technique. This set of markers interrogates populations of T-cells, B-cells, and antigen presenting cells. To detect T cells, we first trained a U-Net to segment CD3+CD4+ T-cells in images of lupus biopsies and achieved an object-level precision of 0.855 and recall of 0.607 on an independent test set. We then evaluated the generalizability of this network to CD3+CD8+ T cells in lupus nephritis and CD3+CD4+ T cells in TNBC, and the extent to which fine-tuning the network improved performance for these cell types. We found that recall increased moderately with finetuning, while precision did not. Further work will focus on developing robust methods of segmenting a larger variety of T cell markers in both tissue contexts with high fidelity.
Lupus nephritis (LuN) is an autoimmune disease characterized by chronic kidney inflammation, which can lead to loss of kidney function, known as end-stage renal disease. The cellular mechanisms causing this progression are not well-defined. Radiomic texture analysis was used to identify image features of biopsies from ESRD+ and ESRD- LuN patients. Each biopsy was stained with 6 markers to identify 5 cell classes in fluorescence confocal microscopy images. Image features associated with the CD20 stain (B lymphocytes), image summary metrics of mean and standard deviation, and 4 GLCM features were identified as most effective in classifying ESRD+/- biopsy images.
Lupus nephritis (LuN) is a manifestation of systemic lupus erythematosus defined by chronic infiltration of immune cells into the kidneys—particularly lymphocytes and dendritic cells (DCs). Ultimately, our goal is to characterize the cellular communities associated with progression to kidney failure. To accomplish this, we have generated a dataset of fluorescence confocal microscopy images of kidney biopsies from 31 LuN patients that have been stained for two T-lymphocyte populations, B-lymphocytes and two DC populations. We are using convolutional neural networks (CNNs) with a Mask R-CNN architecture to perform instance segmentation on these five classes. This multi-class instance segmentation task is hindered by an inherent class imbalance between lymphocytes and DCs, with DCs being much less prevalent. Here we discuss methods for managing class imbalance to achieve comparable instance segmentation of both DCs and lymphocytes in LuN biopsies. A network trained to identify all 5 classes yielded higher sensitivity to DCs when the training set was filtered to contain images with all 5 cell classes present. Average DC sensitivity on an independent test set improved from 0.54 to 0.63 with filtered training data. DC segmentation improved further when the network was trained specifically for DC classes. Average DC sensitivity reached 0.91 when trained separately from lymphocytes, with average Jaccard index of DCs improving from 0.69±0.2 to 0.76±0.2. Accurate segmentation of all cell types relevant to LuN pathogenesis enabled in-depth spatial analysis of the immune environments that result in renal failure in LuN patients.
Significance: Lupus nephritis (LuN) is a chronic inflammatory kidney disease. The cellular mechanisms by which LuN progresses to kidney failure are poorly characterized. Automated instance segmentation of immune cells in immunofluorescence images of LuN can probe these cellular interactions.
Aim: Our specific goal is to quantify how sample fixation and staining panel design impact automated instance segmentation and characterization of immune cells.
Approach: Convolutional neural networks (CNNs) were trained to segment immune cells in fluorescence confocal images of LuN biopsies. Three datasets were used to probe the effects of fixation methods on cell features and the effects of one-marker versus two-marker per cell staining panels on CNN performance.
Results: Networks trained for multi-class instance segmentation on fresh-frozen and formalin-fixed, paraffin-embedded (FFPE) samples stained with a two-marker panel had sensitivities of 0.87 and 0.91 and specificities of 0.82 and 0.88, respectively. Training on samples with a one-marker panel reduced sensitivity (0.72). Cell size and intercellular distances were significantly smaller in FFPE samples compared to fresh frozen (Kolmogorov–Smirnov, p ≪ 0.0001).
Conclusions: Fixation method significantly reduces cell size and intercellular distances in LuN biopsies. The use of two markers to identify cell subsets showed improved CNN sensitivity relative to using a single marker.
Triple-negative breast cancer (TNBC) is an aggressive subtype of breast cancer characterized by a high level of inflammation. In this work, we present a computational method for assessing immune activity across many populations of immune cells. For this analysis, we have a dataset of 18 TNBC biopsies, each stained with a ‘strip-and-reprobe’ approach consisting of multiple rounds of staining and imaging the tissue. Each sample is stained with 8 separate antibody panels, resulting in ~20 different stains per sample, plus tissue autofluorescence images. Biopsy sections are imaged across a ~4mm x 5mm area on a Caliber ID confocal microscope with a pixel size of 221 nm, resulting in image composites of ~27,000x27,000x20 pixels per sample. Using radiomic texture analysis, we have analyzed a subset of these images to determine which texture features are predictive of inflammation. From 3 of these samples, we have selected inflamed and uninflamed regions of interest (ROIs) for three lymphocyte markers: 1) CD4, 2) CD3, and 3) CD8. We have computed 20 texture features, specifically gray level co-occurrence matrix features, for each of these 42 ROIs and their corresponding tissue autofluorescence images. From this analysis, we have found multiple features that successfully differentiate inflamed tissue from uninflamed tissue. These features will be calculated across the full dataset to analyze the overall TNBC immune architecture of these samples as diffuse or compartmentalized, and which cell populations tend to coaggregate.
We demonstrate an instance segmentation method with Mask R-CNN using a ResNet-101 plus Feature Pyramid Network convolutional backbone to segment and classify T cells and antigen presenting cells (APCs) in multi-channel fluorescence confocal images. This network improves on our previous cell distance mapping (CDM) pipeline, which used a custom 10- layer convolutional neural network for cell segmentation. We have validated Mask R-CNN on two independent datasets of fluorescence confocal images: 1) mouse lymph node tissue, and 2) human lupus nephritis (LuN) biopsies. For dataset 1, mice were injected with fluorescent dendritic cells and two populations of fluorescent T cells. Mask R-CNN improved sensitivity averaged across all cell types from 0.88 to 0.94. Specificity improved from 0.92 to 0.95 across all cell types, and intersection over union score (IOU) improved significantly from 0.82 to 0.86 (p < 0.0001). Human LuN biopsies in dataset 2 were stained with two T cell markers and two APC markers, with separate staining panels to identify different populations of APCs. Mask R-CNN again improved segmentation and classification averaged across all cell types, increasing overall sensitivity from 0.72 to 0.76, specificity from 0.86 to 0.93, and significantly increasing IOU from 0.71 to 0.81 (p < 0.0001). Improved IOU scores are particularly important in CDM to be able to quantify cell shape for identification of functional interactions of immune cells. Mask R-CNN is therefore a superior method for instance segmentation of immune cells in microscopy images for image analysis of cellular function in pathological immune states.
Computer vision and deep learning are integral tools in the improvement of high-throughput analysis of cellular images. Specifically, optimization of algorithms for object detection and instance segmentation tasks are important in cellular image analysis to segment and classify multi-object, multi-class images. In this work, we employ an instance segmentation pipeline with Mask RCNN, using a ResNet-101 and Feature Pyramid Network convolutional backbone to segment and classify T cells and antigen presenting cells (APCs) in multi-channel fluorescence confocal images of lupus nephritis biopsies. This task was first performed on a dataset of fresh frozen biopsies stained for T cells (CD3 and CD4) and two APC populations: 1) myeloid dendritic cells (BDCA1 and CD11c), and 2) plasmacytoid dendritic cells (BDCA2 and CD123). The network achieved an average sensitivity of 0.82, specificity of 0.91, and Jaccard index of 0.79 across all cell types. However, relative to fresh frozen tissue, samples prepared through formalin fixation and paraffin embedding (FFPE) provide larger potential datasets for investigating immune activity. Training this same network architecture on an FFPE database of lupus nephritis tissue stained with the same antibody panel, the network achieved an average sensitivity of 0.82, specificity of 0.92, and Jaccard index of 0.77 across all cell types. In addition to working with FFPE tissue, it would also be beneficial to identify APCs with a single stain and image more cell types with a single staining panel. We have trained this network on a single-stained APC panel FFPE dataset to achieve an average sensitivity of 0.79, specificity of 0.86, and Jaccard index of 0.63 across all cell types. These three trained networks were used to assess differences in cell shape features between fixation and staining protocols.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.