1.

Introduction

Intravascular optical coherence tomography (IVOCT) has significant advantages for characterizing vascular lesions compared with other imaging modalities used by interventional cardiologists. This is significant as $\sim \mathrm{700,000}$ percutaneous coronary interventions (PCIs) are performed in the United States every year.¹ Although clinicians routinely use angiography for treatment planning to describe the vessel lumen, angiography does not provide specific information regarding vascular wall composition, except in the case of severely calcified lesions.² Intravascular ultrasound (IVUS) can identify the location of coronary calcium but cannot assess calcium thickness because the signal reflects from the calcium/soft tissue interface, which results in an acoustic shadow following the calcium front surface.³ IVOCT, however, provides the location and often the thickness of a calcification.⁴ In addition, IVOCT has a superior resolution (axial: 15 to $20\mu \mathrm{m}$; lateral: 20 to $40\mu \mathrm{m}$) compared with IVUS (axial: 150 to $200\mu \mathrm{m}$; lateral: 200 to $300\mu \mathrm{m}$).^5,6 Thus, IVOCT is the only intravascular imaging modality with the resolution and contrast sufficient for identifying thin-cap fibroatheromas,^7–9 which are significant prognosticators of plaque rupture and subsequent myocardial infarction.^10–13 Previous studies have suggested that IVOCT can successfully identify coronary fibrous, lipid, calcium, macrophage deposition, and plaque erosion.^14–17

We focus on the application of stent treatment planning with IVOCT and determine the “fibrocalcific” and “fibrolipidic” portions of a lesion. Hard, fibrocalcific coronary plaques can restrict stent expansion, necessitating the use of prestent lesion preparation (atherectomy, cutting balloon, ultrahigh balloon pressures, etc.).¹⁸ Softer, fibrolipidic lesions may not preclude stent expansion. However, when confronting such cases, appropriate determination of stent sizing and its geographic landing site is necessary to ensure that the edge of a stent does not land in a fibrolipidic region prone to vascular dissection or stent rupture. A challenge for live-time stent treatment planning, however, is that IVOCT generates $\sim 500$ image frames in a single $\sim 2.5\text{-}\mathrm{s}$ pullback scan. Visual determination of fibrolipidic and fibrocalcific portions of a lesion requires careful consideration of image features by an expert, making detailed manual analysis of every image frame time-consuming, labor-intensive, and certainly inappropriate for live-time use in the clinic.⁵ This creates a need for automated analysis of IVOCT images.

Previous studies have achieved promising results for automated IVOCT plaque classification. Several groups, including our own, have demonstrated that the “optical attenuation” coefficient can help identify coronary plaque types.^19–22 Others have shown that A-line peaks can be used to discriminate normal from diseased vessel regions.^23,24 Automated “pixel-wise” plaque classification was achieved by several groups. Ughi et al.²⁵ incorporated optical attenuation and texture features with a random forest classifier to identify fibrous, calcified, and lipid pixels within an IVOCT image. In addition, this group classified mature and immature neointimal tissue-covering stents.²⁶ Athanasiou et al.²⁷ utilized intensity and texture features to identify calcium, lipid, fibrous, and mixed tissues using k-means clustering and random forest classifier. Zhou et al.²⁸ used edge detection to identify adventitia and calcium tissue, and they further refined the calcium boundary using a level-set approach. In addition, they used wavelet and intensity features with a random forest classifier to find lipid and mixed tissues. In addition to voxel-wise segmentation, some groups have performed A-line plaque classification. Rico-Jimenez et al.²⁴ used a linear discriminant analysis algorithm to automatically identify A-lines containing fibrotic and fibrolipid plaques. Our group compared using a convolutional neural network (CNN) with a fully connected artificial neural network to classify fibrolipidic, fibrocalcific, and other A-lines.²⁹ Abdolmanafi et al.³⁰ assessed using deep learning methods to perform tissue classification²³ and to identify pathological formations in IVOCT images. Moreover, Gessert et al.³¹ used CNNs to identify IVOCT frames that contain plaque. Although all of the aforementioned studies were promising, some limitations exist. (1) In pixel-wise, semantic plaque classification, there is a conundrum in labeling and in evaluation, as the back border is not seen in lipidous plaques (Fig. 1). (2) It is unclear that all reports use a sufficiently large base of support in the image to capture a priori knowledge of vessel wall morphology (e.g., fibrous tissue followed by a lipidous region). (3) Very few studies consider histological verification. (4) Many studies have a limited number of images, limiting the ability to generalize.

Fig. 1

Large plaques relatively unseen by IVOCT due to high attenuation of lipid. Examples of lipid/necrotic tissue (*) are shown in cryo-images on panels (1a) and (2a) and in corresponding IVOCT images on panels (1b) and (2b). In highly attenuating lipidous plaque, the IVOCT signal is only visible for a few hundred micrometers beyond the plaque border. This is one justification for using our A-line plaque classification approach rather than the more common pixel-wise classification methods.

We contend that A-line classification approaches have advantages compared with pixel-wise approaches. We consider the A-line a more natural analysis element for IVOCT images, as it can capture important plaque characteristics (e.g., the rapid fall off of the signal in lipid or the sharp edges of a calcified lesion). Moreover, when labeling pixels, analysts have no way of knowing the extent of lipid from an IVOCT image (Fig. 1). This is because the IVOCT signal quickly decays in areas containing highly attenuating tissues, such that the detected IVOCT signal is only visible within tens to a few hundreds of microns beyond the plaque border. Hence, the depth of a lipidous plaque and the pixels to label in a lipidous plaque are quite arbitrary. In addition, for calcium and lipid plaques, pixel-wise segmentation approaches can sometimes produce granular plaque boundaries, which do not coincide with what is observed histologically for such tissues. With A-line plaque classification, we can identify plaque angular boundaries and feed results into previously developed plaque boundaries segmentation algorithms.^7,8,32,33 These algorithms implement border smoothness and connectivity constraints to better identify plaque boundaries.

This study includes several important contributions. First, and most important, we evaluated an A-line classification algorithm that uses the most comprehensive set of handcrafted features ($\sim 5000$) to date. We included many features previously used for IVOCT plaque analysis^23,26–28 and introduced innovative types, including vessel-lumen morphology, which evaluates the irregular lumen shape often present in calcifications, and three-dimensional (3-D) digital edge and texture, which can encompass information from nearby IVOCT image frames. Second, we created two large, labeled in-vivo and ex-vivo datasets ($\sim 60$ pullbacks, $\sim 7000$ image frames), which include a very wide variety of lesions, giving an opportunity to better test generalizability of algorithms. The unique ex-vivo dataset is labeled with the aid of carefully registered, cryo-imaging data,³⁴ reducing potential labeling bias introduced by human expert labeling of in-vivo images. The in-vivo dataset ensures that our work translates to real patient data. Third, we selectively tested the impact of our intuitive feature groups (e.g., morphological and 3-D features) on performance, aiding intuitive interpretation. Fourth, we evaluated classifiers having a common or different set of features for each plaque type. Fifth, we applied a conditional random field classification noise cleaning method previously reported by us²⁹ on en-face views, creating visualizations for rapid evaluation of classification quality. Finally, we demonstrate proof of concept to use A-line classification to automatically initiate plaque segmentations, which enables fully automated fibrolipidic and fibrocalcific boundary assessments.

2.

A-Line Plaque Classification Algorithm

Our A-line plaque classification approach involved several steps. We preprocessed the input IVOCT images to reduce speckle, to correct for catheter optics, and to determine an appropriate tissue region for processing ( Supplementary Material, A.1). Then, we extracted numerous handcrafted features, including vascular lumen morphology, optical physics, A-line peaks, and two-dimensional (2-D)/3-D digital edge and texture. We classified A-lines as fibrolipidic (from the lumen, fibrous tissue followed by lipid), fibrocalcific, or “other.” We used feature selection to choose the best features among the large number of proposed features. We compared two distinct multiclass classification strategies: a three-class (3C) approach and dual binary (DB) classifiers (fibrolipidic and fibrocalcific). The 3C approach looked for fibrolipidic and fibrocalcific A-lines simultaneously; the DB classifiers looked for A-lines from the two groups separately and then merged the results. In both cases, conditional random field postprocessing was used to reduce classification noise.

2.1.

Classification Feature Extraction

2.1.1.

Lumen morphology

Lumen morphological features were implemented to describe the size and shape of the lumen, which tends to correlate with hemodynamic shear stress and potentially contributes to atherosclerotic plaque formation.^35,36 Features were determined from the $(x,y)$ anatomical view. We computed the “change in lumen area” ($\mathrm{\Delta }{\mathrm{LA}}_Z$) along the longitudinal ($z$) direction, by comparing the current slice lumen area with each of its adjacent $\pm 3$ slices. The absolute lumen area of the current frame was not included because it depended more on the vessel under study than on disease. “Lumen eccentricity” was used to assess deviations from a circular shape, as often found in the presence of plaque, compared with very circular normal regions (Fig. 2).

Fig. 2

Noncircular lumens with calcified and lipidous plaques. (a) Fairly normal, circular lumen. (b) Very eccentric lumen in vessel containing two calcified lesions (*). (c) Eccentric lumen with a nearly concentric lipid plaque.

We calculated the “frame lumen eccentricity” (FLE), which is a single value describing lumen eccentricity in the current frame. We also computed the “change in frame lumen eccentricity” across adjacent slices ($\mathrm{\Delta }{\mathrm{FLE}}_Z$) by comparing the current frame FLE with the FLE of each of its adjacent $\pm 3$ slices. Individual “A-line lumen eccentricity” (ALE) was determined in an effort to localize A-lines containing plaque (Fig. 3). To calculate ALE, we determined the center of mass and area of the segmented lumen. We then created a perfectly concentric circle having the same area as the lumen, centered at the center of mass. We calculated the signed distance between each A-line lumen boundary point and the corresponding circle boundary along the radius from the center of mass, giving a vector of ALE values.

Fig. 3

A-lines containing calcified/lipidic plaque tend to deviate from a circular lumen contour. A typical vessel with calcified plaques in anatomical $(x,y)$ view (1a) and (2a) and the corresponding $(r,\theta )$ view (1b) and (2b), respectively, is shown. Within these representations, A-lines containing calcifications are highlighted by a dashed yellow line placed behind the calcifications. The top row shows the nonannotated lumen, and the bottom row shows the vessel with a superimposed circular lumen boundary. Both positive and negative excursions from circularity are seen in (2b).

Because we noticed that large eccentric calcifications often resulted in a flat lumen in our dataset, we computed a best-fit line for lumen boundary points and extracted corresponding features (sum of squared residuals, goodness of fit, and line magnitude) based on the residuals between the points and the fitted line (Fig. 4). We also compared the values for the same A-line position for each frame with the values obtained from its adjacent $\pm 3$ slices. These features are summarized in Table S1 in the Supplementary Material.

Fig. 4

A linear line segment fit to the lumen border has a smaller deviation in the calcified region than in more normal tissue. Anatomical $(x,y)$ IVOCT image containing calcium plaque (*). Panels show (a) original image, (b) best-fit line using 1/8 of the points around the lumen border entered on the normal region, and (c) same line length fit to the calcified region.

The last lumen morphological feature is “$R-\theta$ lumen curvature” (Fig. 5). Although listed in the morphological group, this feature could also be considered in the optical group. A region from an IVOCT image can appear dim, not because of the presence of plaque but because the IVOCT imaging beam is obliquely incident on the tissue surface resulting in light being reflected away from the transducer.³⁷ As a surrogate, we computed the lumen slope in $(r,\theta )$ images, where a slope of 0 corresponds to a vertical lumen boundary giving a perpendicular incidence of the beam on the tissue. Large positive or negative slopes corresponded to an oblique angle of incidence.

Fig. 5

Oblique beam incidence due to an eccentric catheter can cause superficial attenuation (SA) of the IVOCT signal. (a) Anatomical $(x,y)$ IVOCT image with the corresponding $R-\theta$ view shown in (b) vessel containing calcified plaque (*). (c) Corresponding fluorescence and (d) color cryo-images are shown. Yellow arrows show the trajectory of the IVOCT signal from the imaging catheter. We observe SA of the IVOCT signal near the point of entry. This could potentially be misconstrued as attenuating plaque, although no plaque is present. This SA region in the IVOCT image is near a high gradient of the lumen boundary, as can be seen in panel (b).

2.1.2.

Optical

The optical features implemented were inspired by the physics of IVOCT image acquisition and the tissue optical properties characteristic of each plaque type.³⁸ Optical attenuation was included (Fig. 6) because this feature has been well established in a number of previous studies.^19,20,22,26 We used the method developed by Vermeer et al.²¹ to determine the optical attenuation. Using a discretized differential equation, they solved the optical attenuation at each pixel as

Eq. (1)

$$\mu (i)\approx \frac{I[i]}{2\mathrm{\Delta }\sum _{i+1}^{\infty }I[i]},$$

where $\mu (i)$ is the attenuation at a given pixel, $I[i]$ is the intensity at the given pixel, $\mathrm{\Delta }$ is the pixel size, and the summation is the sum of all intensities “deeper” than the current voxel.

Fig. 6

Optical attenuation map of coronary plaque. A representative fibrolipidic (1) and fibrocalcific plaque (2) are shown in the $(r,\theta )$ view. (a) An intensity image and (b) the corresponding attenuation map, as obtained using the Vermeer method. Lipid plaques (dashed yellow line) tend to have high attenuation near the fibrolipid boundary. Calcified plaque (yellow asterisk) tend to have lower attenuation values compared with surrounding fibrous tissue.

Intensity features for each A-line were extracted from linear data, as we expect tissues with heavily calcified or lipid tissues to have low image intensities.^14,16 For intensity and optical attenuation features, we calculated the mean, median, standard deviation (SD), minimum, maximum, skewness, and kurtosis values over the following ranges of tissue, observed immediately after the lumen boundary: 0.0 to 0.25 mm; 0.25 to 0.5 mm; 0.5 to 0.75 mm; 0.75 to 1.0 mm, as well as over the total 1.0 mm.

Additional optical features were included. For each A-line, we obtained the distance between the catheter and the lumen boundary because larger distances could contribute to diminished image intensity. The distance to background intensity describes the depth at which the signal reached a value for background, as determined using Otsu’s method.³⁹ We determined this value by starting at the far depth of the image and searching toward the lumen to find the first pixel above this threshold. This feature should detect highly attenuating fibrolipidic regions, which are consistent with a rapid decay of the IVOCT signal. All optical features used are summarized in Table S2 in the Supplementary Material.

2.1.3.

A-line peaks

We analyzed prominent peaks in each IVOCT A-line (Fig. 7), which tend to correlate with plaque type.^23,24 Briefly, A-lines corresponding to normal tissue have a prominent first and second peak, and the signal tends to monotonically decrease. Fibrocalcific A-lines have several prominent peaks, and the signal does not decay monotonically. Finally, fibrolipidic A-lines tend to have only one prominent peak near the A-line boundary, and then the signal rapidly decays to baseline intensity. We low-pass filtered each A-line using a zero-phase shift finite impulse response filter with order 50 and cut-off frequencies of 0.1 and 0.01 of units $1/\mathrm{pixel}$ (pixel size $\sim 5\mu \mathrm{m}$). The cut-off frequency affects smoothing of the signal. To identify local peaks, we used the “findpeaks” function in Matlab with a minimum intensity difference required of ${10}^{-4}$ between a peak and its neighbors. Several features were determined, as described in Table S3 in the Supplementary Material.

Fig. 7

A-line peak features can facilitate discrimination between fibrocalcific, fibrolipidic, and other A-lines. Yellow A-lines are plotted from left to right in $(x,y)$ corresponding $(r,\theta )$ images, displayed graphically on the right. Peaks are denoted by asterisks (*). Normal vessels have two characteristic peaks corresponding to the initial intimal and back adventitial borders (top right). Further the signal appears to be monotonically decreasing. Lipid lesions tend to have one characteristic peak followed by a fast decay of the signal (middle right). Fibrocalcific A-lines tend to have a variety of peaks, and a signal without a monotonic decrease (bottom right).

2.1.4.

Digital edge and texture

We used 2-D digital edge and texture features because calcified plaques tend to have visually pronounced edges and a more prominently textured appearance than lipidous plaques.^14,16 Texture features came from the Leung–Malik filter bank,⁴⁰ which is a multiscale, multiorientation filter bank containing 48 filters, including multiorientation edge, Gaussian, and Laplacian of Gaussian filters. We also used the Schmid filter bank,⁴¹ which consists of 13 rotationally invariant filters. Finally, edge gradients were added (i.e., Laplacian, horizontal and vertical Sobel, and horizontal and vertical Prewitt). The parameters for each kernel are presented in Table S4 in the Supplementary Material. To process these 2-D features, all helically sampled A-lines over a large vessel segment were merged to form a large 2-D ($r,\theta$) array, for example, consisting of about $\mathrm{50,000}\times 1000\text{pixels}$ for a 100-frame segment. The array was pixel-shifted and only the first 1.0 mm of tissue in the wall was filtered. We referred to this image as ${\text{Concatenated}}_{R\theta }$. We then convolved this image with our various 2-D digital edge/texture filters to derive our filter responses. We computed the mean, median, SD, minimum, maximum, skewness, kurtosis, and energy values over the following ranges of tissue: 0.0 to 0.25 mm; 0.25 to 0.5 mm; 0.5 to 0.75 mm; 0.75 to 1.0 mm, as well as over the total 1.0 mm.

We implemented the 3-D digital edge and texture features (i.e., Laplacian, Laplacian of Gaussian, Sobel, and Prewitt). To do this, we reshaped our ${\text{Concatenated}}_{R\theta }$ image to form a 3-D $(r,\theta ,z)$ matrix and convolved this image with the various 3-D digital/edge filters from our filter bank. Following 3-D convolution, the $(r,\theta ,z)$ matrix was reshaped back to the ${\text{Concatenated}}_{R\theta }$ image, and feature values were calculated in the same manner as for the 2-D edge/texture features described above. The descriptions are summarized in Table S5 in the Supplementary Material.

3.

Experimental Methods

4.

Results

4.1.

A-Line Classification Results

We determined the optimal features and tested the performance of each one of the 10 combinations of classifiers and feature selection schemes that we developed. Feature selection was performed to find the optimal number of features for each classifier. The results are shown for the 3C-SVM classifier in Fig. S2 in the Supplementary Material. After feature selection, we performed fivefold cross validation and testing on our held-out dataset. The best 3C approach was the 3C-SVM. Using fivefold cross validation on the training data, the 3C-SVM classifier gave an overall classification accuracy of 74.60%, with percentage sensitivity/specificity for each class: other (76.90/88.77), fibrolipidic (88.33/83.78), and fibrocalcific (69.31/93.10). Moreover, on the held-out dataset the 3C-SVM classifier gave an overall classification accuracy of 76.52%, with percentage sensitivity/specificity for each class: other (81.43/89.59), fibrolipidic (94.48/87.32), and fibrocalcific (74.82/95.28). The confusion matrices and the table containing these results are shown in Fig. S3 and Table S1 in the Supplementary Material.

For the DB approaches, we found that the DB-SVM classifier using mRMR feature selection worked best for fibrolipidic classification and that the DB-SVM classifier using Wilcoxon feature selection worked best for fibrocalcific classification. When performing fivefold cross validation of the training data, the DB-SVM classifier using mRMR feature selection for fibrolipidic classification gave an overall classification accuracy of 80.14%, with percentage sensitivity/specificity for each class: other (80.59/79.68) and fibrolipidic (79.68/80.59). Moreover, on the held-out dataset, the DB-SVM classifier had an overall classification accuracy of 85.46%, with percentage sensitivity/specificity for each class: other (80.31/90.60) and fibrolipidic (90.60/80.31). Moreover, when performing fivefold cross validation of the training data, the DB-SVM classifier using Wilcoxon feature selection for fibrocalcific classification gave an overall classification accuracy of 80.03%, with percentage sensitivity/specificity for each class: other (87.90/72.03) and fibrocalcific (72.09/87.90). Moreover, on the held-out dataset, the DB-SVM classifier had an overall classification accuracy of 81.99%, with percent sensitivity/specificity) for each class: other (90.59/73.38) and fibrocalcific (73.88/90.59). The confusion matrices and the tables containing these results are shown in Figs. S4 and S5 and Tables S2 and S3 in the Supplementary Material.

In addition, we assessed the utility of including morphological and 3-D features for A-line plaque assessment (Fig. 8). We used the best 3C, binary fibrocalcific, and fibrolipidic classifiers and observed the effect of removing our 3-D and morphological features on overall classification accuracy on our held-out test set. For all classifiers, we found a reduction in overall classification accuracy following the removal of both 3-D and morphological features, suggesting the utility of incorporating both feature groups for fibrolipidic and fibrocalcific plaque classification.

Fig. 8

Added benefit of morphological and 3-D feature groups. For our best 3C classifier (3C-SVM) and our best binary fibrocalcific (DB-SVM-Wilcoxon) and fibrolipidic (DB-SVM-mRMR) classifiers, we assessed the overall classifier accuracy with and without the use of morphological and 3-D features. Performance was tested on our held-out test set. In virtually all the cases, we saw a decrease in classifier performance when 3-D and morphological features were removed, compared with when they were present. This suggests the value of using morphological and 3-D features for plaque assessment.

4.2.

Classification Noise Cleaning

We integrated the output from the best-performing classifiers from Secs. 4.1 and 4.2 into the CRF classification noise cleaning approach described in Sec. 2.2 (Fig. 9). For the 3C-SVM model, grid searching suggested the following optimal parameters: ${\omega }^{(2)}$ as 0.75, ${\sigma }_{\gamma }$ in the $\theta$ direction as 1.5, and ${\sigma }_{\gamma }$ in the $z$ direction as 1.5. For the fibrocalcific DB-SVM using Wilcoxon feature selection, we set ${\omega }^{(2)}$, ${\sigma }_{\gamma }$ in the $\theta$ direction, and ${\sigma }_{\gamma }$ in the $z$ direction as 0.5, 1.5, and 1.5, respectively. For the fibrolipidic DB-SVM using mRMR feature selection, we set ${\omega }^{(2)}$, ${\sigma }_{\gamma }$ in the $\theta$ direction, and ${\sigma }_{\gamma }$ in the $z$ direction as 0.5, 1.5, and 1.5, respectively.

Fig. 9

Classification noise cleaning using the en-face $(\theta ,z)$ view. CRF implementation smooths the classification results to eliminate isolated A-lines from being classified as plaque. This is evident for our best classifiers: 3C-SVM), binary fibrocalcific (SVM-Wilcoxon), and binary fibrolipidic (SVM-mRMR). We show a representative case for each classifier: (a) initial classification results, (b) CRF cleaned results, and (c) ground-truth annotations by experts. We visually observed that CRF cleaning made classification results more similar to those of ground-truth labels. The following CRF parameters were used for 3C, DB-fibrocalcific, and DB-fibrolipidic, respectively: [$\sigma \mathrm{\Theta }$: 1.5; $\sigma z$: 1.5; $\omega$: 0.75], [$\sigma \mathrm{\Theta }$: 1.5; $\sigma z$: 1.5; $\omega$: 0.5], [$\sigma \mathrm{\Theta }$: 1.5; $\sigma z$: 1.5; $\omega$: 0.75].

For the 3C-SVM approach, CRF cleaning improved the overall classification accuracy to 81.58%, with percentage sensitivity/specificity for each class: other (81.43/89.59), fibrolipidic (94.48/87.32), and fibrocalcific (74.82/95.28). Improvement in classification performance after CRF cleaning was also observed for the DB classifiers. For the fibrolipidic classifier, CRF cleaning improved the overall classification accuracy to 85.52%, with percentage sensitivity/specificity for each class: other (84.20/94.58) and fibrolipidic (94.48/84.20). For the fibrocalcific classifier, CRF cleaning improved the overall classification accuracy to 91.25%, with percentage sensitivity/specificity for each class: other (95.71/76.26) and fibrocalcific (76.26/95.71). These results are also displayed in Fig. S6 and Table S9 in the Supplementary Material.

We finally merged the cleaned results for the fibrolipidic and fibrocalcific classifiers to perform an apples-to-apples comparison of their performance with that from the 3C classifier (Fig. 10). After merging, for the DB approaches, we achieved an overall accuracy of 80.19%, with percentage sensitivity/specificity for each class: other (79.90/89.31), fibrolipidic (93.63/85.03), and fibrocalcific (73.52/96.19). Thus, the performances of the two approaches were nearly identical, with a slight edge given to the 3C-SVM classifier. The fact that the two approaches obtained similar performances yields credence to the same underlying feature set from which the two approaches were developed.

Fig. 10

Comparison of 3C and combined binary fibrolipidic and fibrocalcific classifiers for multiclass classification on the held-out test set. After classification noise cleaning, we merged the results of the DB classifiers to enable an apples-to-apples comparison with the 3C approach for multiclass classification. For the 3C-SVM approach, we achieved an overall accuracy of 81.58%, with percentage sensitivity/specificity for each class: other (81.44/89.59), fibrolipidic (94.48/87.32), and fibrocalcific (74.82/95.28). For the combined DB classifiers, we achieved an overall accuracy of 80.19%, with percentage sensitivity/specificity for each class: other (79.90/89.31), fibrolipidic (93.63/85.03), and fibrocalcific (73.52/96.19). Overall, we obtained very similar values for both our multiclass approaches, with the 3C-SVM giving slightly better performance.

4.3.

Successful and Challenging Classification Cases

Overall, our classification approach worked very well in the majority of cases. We show results from the cleaned 3C-SVM classification approach. Fibrocalcific cases with sharp borders, a textured appearance, and very eccentric lumens were easily classified. Figure 11(a) shows a $\sim 150\text{-}\mathrm{deg}$ fibrocalcific lesion with a textured appearance and visually sharp luminal and abluminal boundaries. Figure 11(b) shows a $>180\text{-}\mathrm{deg}$ fibrocalcific lesion with a very eccentric lumen, textured appearance, but only luminal boundaries that were visibly sharp. In both cases, however, our classifier gave near-perfect assessments of the lesion angular boundaries. Fibrolipidic cases in which the signal initially peaks and then decreases were also correctly classified. Figure 11(c) shows a $\sim 270\text{-}\mathrm{deg}$ fibrolipidic lesion with a sharp peak near the luminal boundary followed by a rapid decay of the signal. Figure 12(d) shows a near-360-deg fibrolipidic lesion with a characteristic initial peak followed by a precipitous decay of the signal. In both fibrolipidic lesions, a near-perfect agreement of plaque angular boundaries between manual and automated assessments was observed.

Fig. 11

Successful classification results of our 3C-SVM classifier on our held-out test set. (a)–(d) Agreement between expert annotated (outer ring) and automated (inner ring) assessments of plaque angular boundaries. In (a) and (b), we see two frames successfully classified as fibrocalcific, with the near-perfect agreement of boundaries between manual and automated assessments. In (c) and (d), we see two representative frames successfully classified as fibrolipidic. Again, we observe a near-perfect agreement of boundaries between manual and automated assessments.

Fig. 12

Bland–Altman assessment demonstrates a strong agreement between the manual and automated assessments of fibrocalcific plaque boundaries across all frames in our held-out test set. The mean value of agreement was $7.70\mathrm{deg}\pm 38.97\mathrm{deg}$. No apparent bias was observed regarding plaque angle, which suggested that we can accurately predict the angle of both small and large calcified lesions. Only 141 of 1737 frames were outside the 95% confidence interval of agreement.

Because fibrocalcific angle assessment is critical for clinical treatment planning, we assessed agreement across all frames in our held-out test set between manual and automated analysis (Fig. 12). Bland–Altman analysis of the fibrocalcific angle demonstrated a very strong correlation between the expert and automated assessments. On average, the difference between the automated and manual assessments was only $7.7\mathrm{deg}\pm 39.0\mathrm{deg}$. Only 141 of the 1737 images assessed in the held-out test set were outside the 95% confidence interval. No apparent bias was observed regarding plaque angle, which suggested that we can accurately predict the angle of both small and large calcified lesions.

Across all the 1737 images in our held-out test set, we only confronted a few challenging cases (Fig. 13). However, we believe that such cases would be difficult for an analyst as well. Figure 14(a) shows an image that contains both fibrocalcific and fibrolipidic A-lines. The classifier correctly identifies both lesions. However, one part of the lesion referred to as fibrocalcific by the analysts is designated as other by the classifier. This is because a clear back border is not visible in this region, and it is possible that this region of the plaque is mixed. Figure 14(b) shows a lesion that was judged to be fibrolipidic by the analysts. However, the classifier designates one portion of the lesion as fibrocalcific because this region does not decay clearly, and the fibrocalcific portion does appear to have some textured appearance and a slightly eccentric shape following the initial signal peak. In both cases, it is possible that this case was either misclassified by our experts or contained some combination of both lipidic and calcific plaque.

Fig. 13

Challenging cases for classification. (a), (b) Agreement between expert annotated (outer ring) and automated (inner ring) assessments of A-line classification. In (a), a case that contains both a fibrocalcific and a fibrolipidic lesion is shown. The classifier successfully detects both lesions. However, a small portion of what analysts deemed as calcium was classified by automated assessment as other (denoted by *). However, this region of misclassification is missing a visible abluminal border for the fibrocalcific lesion. Thus, it could potentially be a mixture of both fibrolipidic and fibrocalcific lesions. On the right, we have a case deemed by experts as a concentric fibrolipidic lesion. However, automated assessment deemed one portion as fibrocalcific. In this case, the plaque boundary does not appear to have a bright peak followed by precipitous decay of the signal, potentially indicating that the lesion is also a mixture of fibrolipidic and fibrocalcific plaque.

Fig. 14

Transition of A-line classification results into semiautomated plaque segmentation approaches. We demonstrate proof of concept to output our A-line classification results into previously developed fibrocalcific and fibrolipidic boundary segmentation approaches. This enables fully automated plaque classification and segmentation. (1) A representative fibrocalcific plaque example. (2) Representative fibrolipidic plaque. In both examples, (a) contains a volume rendering with vessel wall shown in orange, and (b) and (c) show a representative frame before and after plaque boundary segmentation, respectively. In (1c), calcium plaque is shown in white. (2c) Lipid cap thickness in color heatmap. Lumen border is outlined in blue in both cases.

5.

Discussion

In this study, we used numerous handcrafted features and expansive clinical in-vivo and ex-vivo validation datasets (totaling $\sim 7000$ images) to develop automated methods to identify fibrolipidic and fibrocalcific IVOCT A-lines. We incorporated features developed in previous studies that assessed tissue optical properties,^21,22 A-line peaks,^23,24 and 2-D texture features.^25–27,46 In addition, we developed features based on vascular lumen morphology and 2-D/3-D digital edge and texture. We investigated two strategies for multiclass classification: a 3C approach and DB classifiers. The 3C approaches tried to simultaneously detect fibrolipidic and fibrocalcific A-lines. The DB approaches tried to detect fibrolipidic and fibrocalcific A-lines independently and then merge results together. Our 3C approach using a multiclass SVM classifier and mRMR feature selection with a common set of features for fibrolipidic and fibrocalcific A-lines had the best performance. Following classification noise cleaning (Fig. 13), this approach had an overall classification accuracy of 81.58% for all A-lines in a held-out test set ($\sim 1800$ images), with percentage sensitivity/specificity for each class: other (81.43/89.59), fibrolipidic (94.48/87.32), and fibrocalcific (74.82/95.28). In addition, fibrocalcific angle detection between manual and automated assessments over our entire test set as assessed using Bland–Altman analysis was $7.7\mathrm{deg}\pm 39.0\mathrm{deg}$ (Fig. 12). Again, this indicates strong performance of our automated approach. In addition, we demonstrated proof of concept for streamlining classification output with existing fibrolipidic and fibrocalcific boundary segmentation methods (Fig. 14). Fully automated IVOCT plaque characterization could facilitate live-time stent treatment planning and offline assessment of drug and biologic therapeutics. The automated results could be input into finite element models to predict the clinical outcomes of using stents in certain lesions.^47,48 Thus, from A-line segmentation results, we have an opportunity to give recommendations on proper stent sizing, location, and potential plaque modifications. Moreover, the software provides an opportunity for offline assessment of drug and biologic therapeutics. That is, the software developed could help facilitate analysis of plaque progression/regression in offline assessments of patient data.

We believe that our approach of combining A-line classification with segmentation methods is advantageous compared with pixel-wise classification. This is because IVOCT data are acquired as a series of one-dimensional A-lines, which makes A-line analysis a natural means of assessing IVOCT images. Moreover, pixel-based classification is severely compromised in areas containing highly attenuating tissue types (Fig. 1). Furthermore, pixel-based classification can produce granular borders, which do not reflect the plaque boundaries that are histologically observed. Using A-line classification with 3-D classification noise cleaning, we locate the general location of plaque. Then, we apply plaque boundary segmentation to the location to find the exact location and to quantify plaque burden. Not only does this approach produce biologically observe plaque boundaries, but it also mimics how IVOCT image assessment was performed by our analysts.

Incorporation of morphological and 3-D features into our classification approach improved the overall classification accuracy (Fig. 11). Morphological features were inspired by previous studies, which suggest that vascular lumen morphology changes coincides with plaque incidence.^35,36 Digital 2-D/3-D edge features accounted for the difference in edge strength observed between the fibrolipidic and fibrocalcific boundaries.^14,16 The 3-D features accounted for the biologically observed, volumetric continuity of blood vessels and associated plaques. The improvement in overall classification accuracy by incorporating these feature groups suggests that future studies would benefit by considering such features for IVOCT plaque classification. Several groups, including our own, are investigating deep learning approaches for IVOCT image analysis and plaque classification.^23,29,30 Hybrid methods, which combine such deep learning results with the handcrafted features used in our study, could further improve classification performance.

We varied our multiclass classification strategy, choice of classifier, and feature selection algorithms to optimize the classification performance (Figs. S2–S6 in the Supplementary Material). The decision to vary such parameters was based on previous studies⁴³ that suggested that optimizing these parameters affects classification performance. In the end, we merged our DB classifiers to obtain a multiclass result, which we compared with the results of 3C classification (Fig. 10). We found nearly identical performance using these approaches. The fact that all approaches performed similarly is evidence of the robustness of our feature set.

Classification noise cleaning, which combines CRF and morphological image-processing operations, smoothed classification results and improved classification true positive rates by $\sim 5\%$ (Fig. S6 in the Supplementary Material). This processing was performed on the en-face ($\theta -z$) view of data, which is an innovative way of viewing volumetric classification results for a vessel in a single 2-D view. Thus, our method ensured that classification results made sense from a biological standpoint, considering that plaques demonstrate volumetric continuity. We used a CRF model⁴⁴ for classification noise cleaning. This algorithm considers the classification probabilities but also enables incorporation of intensity information into determining cleaned classification results. In our approach, we only considered classification probabilities. Future studies can investigate whether incorporating en-face intensity data improves classification performance. In previous work, we generated intensity-based IVOCT en-face views⁴⁹ to facilitate stent segmentation. Such views could be incorporated into the CRF model.

Classification models were developed using both in-vivo and ex-vivo training data. This is advantageous because it instills confidence in our training data and annotation labels. Labels in ex-vivo data were obtained from accurately registered, cryo-imaging data, facilitating accurate labeling. The use of in-vivo data ensures that our classification approach is applicable to real patient data. Although our classification approach performed well in the majority of cases, we noticed several challenging cases in our held-out test set (Fig. 14). We believe that these challenging cases were potentially mixed lesions. Expansion of our ex-vivo training database could help with classifying such cases.

A number of software implementation optimizations were necessary while developing our classification approach. This study involved machine learning using very expansive datasets. In total, $\sim 2.3\mathrm{million}$ A-lines were analyzed, each as an individual sample. For each A-line, we extracted a total of $\sim 5000$ features before feature selection, most of which were of data type double. Thus, although the computer we used had 128 GB of memory, we needed to carry out frequent memory clearing when performing computations on the entire data set. For feature extraction, it was imperative to preallocate data matrices and perform computation on portions of data at a time. Matrix reshaping and concatenation to generate our ${\text{Concatenated}}_{R\theta }$ image and 3-D $(r,\theta ,z)$ matrix facilitated the computational extraction of digital edge and texture features. When performing classification model development and fivefold cross-validation training, it was necessary to subsample data. These computational limitations could be circumvented in the future using computers with vast amounts of memory. Finally, although the classification time on precomputed features for the 3C-SVM classifier was $\sim 3\mathrm{s}/\text{frame}$, classification feature extraction is a rate-limiting step. We could expedite feature extraction by experimenting with further feature reduction to see whether we can use fewer features to generate comparable classification results. Improving our hardware resources by increasing computer memory and using more powerful graphics-processing units can further reduce computation time.

6.

Conclusion

IVOCT is a promising intravascular imaging modality for assessing coronary plaque burden to perform stent treatment planning. Our algorithm automatically classifies fibrocalcific and fibrolipidic IVOCT A-lines, using a very large number of handcrafted features and is developed using expansive in-vivo and ex-vivo, cryo-image validated datasets. Use of our 3-D and lumen morphology features improved classification accuracy, suggesting the importance of these intuitive features. A-line classification results were streamlined to initiate plaque segmentation. The study results suggest that our classification approach is a viable step toward fully automated IVOCT plaque classification and quantification for both live-time treatment planning and offline assessment of drug and biologic therapeutics.