1.

Introduction

Conventional color image sensors, either charge-couple device or complementary metal oxide semiconductor, acquire images in three different channels, red, green, and blue (RGB). In colorimetry or spectrophotometry, these devices can have inaccuracy due to their low spectral resolution.¹ Multispectral imaging (MSI) or hyperspectral imaging (HSI) aims at capturing images with more than three channels across the electromagnetic spectrum. These techniques can enhance the accuracy of the spectral and color estimation of a surface.² The main difference between MSI and HSI is the number of sensing channels in a given spectral range. HSI typically has a higher spectral resolution but is an expensive technique that suffers from low spatial resolution, and it is mainly encountered in remote sensing applications.³ MSI is a trade-off^4,5 that is more appropriate for a more extensive application board, such as medical imaging⁶ or quality inspection.⁷

The polarization property is another light attribute that complements the intensity and frequency. Human eyes cannot differentiate different polarization states without extra devices. Polarization provides information about the direction of the transverse electric fields that comprises the electromagnetic radiation. The light can be fully or partially polarized or even unpolarized. The predominant type of polarization can be categorized as linear, circular, or elliptical. Polarization analysis can model the relationship between the polarization state of the light and the physical interaction with matter, after reflection, refraction, or transmission. Polarization imaging offers the possibility of capturing data in 2D and is useful in computer vision, e.g., to separate the diffuse or specular components,^8,9 analyze the spatial variation of the index of refraction,^10–12 or estimate the surface normals.^13,14

Some insects such as crickets or butterflies developed a vision system to perceive both spectral and polarization signals. This capacity helps them to catch their prey or with navigation.¹⁵ The combination of both modalities has gained interest in various applications, such as canopy and ice detection,^16,17 medical imaging,¹⁸ or object tracking.¹⁹ Spectral polarization imaging (SPI) also plays an effective role in imaging over critical weather or underwater conditions.^20,21

Based on a recent review that we conducted,²² snapshot SPI acquisition systems have been categorized as a division of amplitude^23–25 or polarization grating devices.^26–28 Table 1 summarizes the recent practical devices from the state-of-the-art systems. Many methods have significant complexity and require either modulation processes, high-cost optics, or moving parts that reduce the sampling resolution. Some imaging setups are also often big to accommodate for the optical paths. We believe that a high resolution and simpler spectropolarimetric imaging system is still to be investigated.

Table 1

Comparison of the existing and proposed snapshot SPI systems.

Polarization filter array (PFA) and color polarization filter array (CPFA) sensors, such as the Sony IMX250 MZR or IMX250 MYR,³³ have been integrated by several camera manufacturers in affordable commercial devices. However, these sensors do not have sufficient spectral resolution for accurate spectral sensing as they capture only three spectral channels. Based on these devices, this paper proposes a compact and cost-effective setup capable of a snapshot acquisition of multispectral polarization data. Compared with other snapshot systems presented in Table 1, this work has the advantage of a high spatial resolution, with a relatively good spectral resolution in the full visible range.

The paper is organized as follows. In Sec. 2, we present the hardware and the preprocessing steps of the developed prototype. In Sec. 3, we define the procedure to calibrate the spectropolarimetric data. We then describe the transformation of the data, before providing a discussion in Sec. 5 and a conclusion in Sec. 6.

2.

Hardware and Preprocessing

2.1.

Hardware

The proposed SPI setup consists of two Triton TRI050S-QC 5.0 MP color polarization cameras, manufactured by Lucid Vision Lab. These cameras embed the SONY IMX250 MYR sensor,³³ the properties of which are shown in Fig. 1(a). This is a division-of-focal-plane sensor, in which each pixel senses one spectral channel and one polarization channel. The filter arrays are composed of three spectral channels ($c\in R,G,B$) spatially organized in a Quad-Bayer arrangement³⁴ and four polarization channels ($p\in \{\mathrm{0,45,90,135}\}\mathrm{deg}$) arranged in a Chun pattern³⁵ [see Fig. 1(b)].

Fig. 1

The color PFA (CPFA) sensor: (a) sensor characteristics and (b) spatial arrangement of the SONY IMX250 MYR sensor. Each pixel is covered by one linear polarization filter and one spectral filter.

As shown in Fig. 1(d), the two cameras are in a stereo configuration. In front of each camera, a lens with a fixed focal length of 12 mm is added. They are configured not to perform any analog/digital gain or white balance corrections, to ensure that the raw data are linear and not corrupted by any preprocessing.

MSI usually requires complex and expensive acquisition devices. One can reduce the complexity by assuming that the reflectance spectrum is smooth across the visible wavelengths and thus by limiting the number of spectral channels. Examples in the literature demonstrate that MSI in the visible can be achieved using six channels with a relatively good spectral estimation.³⁶ A practical method is to use conventional RGB cameras combined with a set of spectral bandpass filters.^1,37,38 This is usually done in multiple shots, placing one filter at a time. We use a similar technique by combining each of the two cameras with a fixed bandpass filter, so the two cameras have different spectral sensitivities. The first bandpass filter is a blue–green BG39 filter (noted $bg$), and the second one is a yellow GG475 filter (noted $y$), both manufactured by Schott Glass Technologies. We selected the same filters as those of Berns et al.³⁷ The bandpass filters are directly mounted on the objective lenses. From these combinations, we obtain six spectral channels in the visible range. Figure 2 shows the spectral sensitivities of the global system, along with the acquisition configuration.

Fig. 2

(a) The CPFA camera spectral sensitivities (only the 0-deg polarization channel is shown because the four polarization channels are very similar). The sensitivity is obtained by characterizing the sensor with a spectral resolution of 10 nm from 370 to 750 nm. (b) Spectral transmission of the bandpass filters provided by the manufacturer. (c) Total spectral sensitivities of the cameras after combination. (d) Setup configuration.

A uniform, unpolarized light source is positioned approximately normal to the base of the configuration.

In the next sections, we will describe the processing of the images from the setup. The steps are shown in Fig. 3.

Fig. 3

Practical imaging pipeline.

3.

Calibration

Two calibrations are done after preprocessing: (1) a geometric stereo calibration to establish the correspondence between the image pair from the two cameras and (2) a spectral calibration to estimate the reflectance of the scene. (Using the polarimetric characterization in Ref. 42, we found that the polarization measurement is near to ideal for these sensors (very high extinction ratios of the micropolarizers), so there is no need to calibrate polarimetrically.)

3.2.

Spectral Calibration

3.2.1.

Spectral reconstruction method

The spectral properties of surfaces are approximated by a few basis vectors and described by a low-dimensional linear model.⁴⁵ Here, we describe the spectral calibration to estimate the reflectance data $\mathbf{r}$ of a scene from the vector $\mathit{\rho }$ containing the digital values ${\rho }_j$. We use the method based on linear regression that links the spectral reflectance $\mathbf{r}$ to the camera responses $\mathbf{I}$ directly⁴⁶ as follows:

Eq. (2)

$$\mathbf{r}=\mathbf{MI}\hspace{0.17em},$$

where the matrix $\mathbf{M}$ is a reconstruction operator and $\mathbf{I}$ is a vector containing the measured intensities by the spectral system. In our case, $\mathbf{I}$ is of size $[1\times 6]$ and contains the camera intensities $I_s$, where $s\in R_{bg},R_y,G_{bg},G_y,B_{bg},B_y$. This is equivalent to calculating the first Stokes component⁴⁷ from ${\rho }_j$ as follows:

Eq. (3)

$$I_s=\frac{{\rho }_{s,0}+{\rho }_{s,45}+{\rho }_{s,90}+{\rho }_{s,135}}{2}.$$

Here, $\mathbf{M}$ is estimated using a set of reference spectra, which are previously measured by a spectrometer. We selected the Xrite Macbeth ColorChecker PassPort (MCCPP), which is composed of two main charts with known spectra: (1) the ColorChecker classic chart used for training [24 patches, see Fig. 4(a)] and (2) the creative enhancement chart used for testing [26 patches, see Fig. 4(c)]. Visualization of the reflectance factors for all patches is shown in Figs. 4(b) and 4(d). We assume that the training set is representative enough of the surface reflectances, which will be estimated a posteriori by the system. Another assumption is that the reflection from the charts is purely diffuse, so they do not reflect any specular component. We call ${\mathbf{R}}_{\text{train}}$ and ${\mathbf{R}}_{\text{test}}$ the set of known training and testing reflectance data, respectively, that are obtained from the Spectral Library of Chromaxion website.⁴⁸

Fig. 4

(a) ColorChecker classic chart from the X-Rite ColorChecker passport used for spectral calibration. (b) Spectral reflectance of the classic chart from the Chromaxion website.⁴⁸ (c) Creative enhancement chart from the X-Rite ColorChecker passport used for testing. (d) Spectral reflectance of the creative enhancement chart. (e) Comparison of the spectral estimation of the reflectance from the creative enhancement chart. GFC are shown for each recovered spectrum ${\widehat{\mathbf{r}}}_{\text{test}}$.

The calibration of $\mathbf{M}$ is done using the pseudoinversion as follows:

Eq. (4)

$$\widehat{\mathbf{M}}={\mathbf{R}}_{\text{train}}{\mathbf{I}}_{\text{train}}^{+},$$

where the + operator is the right Moore–Penrose pseudoinverse operator, ${\mathbf{I}}_{\text{train}}$ is a $[6\times 24]$ matrix formed by column vectors containing averaged camera signals (over a $10\times 10\text{pixel}$ area) of the 24 patches, ${\mathbf{R}}_{\text{train}}$ is a $[36\times 24]$ matrix with the reference reflectance factors of the patches from 380 to 730 nm with a step of 10 nm, and $\widehat{\mathbf{M}}$ is a $[36\times 6]$ matrix that is an estimate of $\mathbf{M}$.

3.2.2.

Spectral estimation evaluation

To evaluate the performance of the spectral estimation, we apply the trained linear model to a set of camera responses from the creative enhancement chart [see Fig. 4(c)]. Thus, the spectral estimation is done on a set of 26 patches, different from the set used for training. The computation is performed on ${\mathbf{I}}_{\text{test}}$, which is a $[6\times 26]$ matrix containing the averaged camera responses, as

Eq. (5)

$${\widehat{\mathbf{R}}}_{\text{test}}={\widehat{\mathbf{M}}\mathbf{I}}_{\text{test}}.$$

To quantify the reconstruction error, we compute the goodness of fit coefficient (GFC) for each estimated patches with reflectance vector ${\widehat{\mathbf{r}}}_{\text{test}}$. This gives a scalar value, computed as follows:

Eq. (6)

$$\mathrm{GFC}=1-\frac{1}{N_w}\Vert {\mathbf{r}}_{\text{test}}-{\widehat{\mathbf{r}}}_{\text{test}}\Vert ,$$

where $N_w=36$ is the total number of samples in the considered wavelength range and $\Vert .\Vert$ is the two-norm of a vector. A GFC of 0 indicates a bad fit, whereas 1 is a perfect fit.

Figure 4(e) shows the comparison between the estimated and measured (reference) reflectance spectra of the 26 patches of the creative enhancement chart. We can see that the estimated spectra are close to the reference data. The worst GFC values occur especially when high reflectances are encountered (patches 1, 7, and 8 from the first line of patches). This can be due to the relatively low sensitivities (and thus a low SNR) of our system in this range of wavelengths.

As a comparison, we computed the GFC on the test chart using either three channels ($s=R_{bg},G_{bg},B_{bg}$) or six channels. We found that our six-channel system increases the GFC by an amount of 0.16 in average over the 26 patches.

4.

Data Transform and Database

4.1.

Data Transform

Here, we transform the calibrated data using three selected representations: (1) Stokes formalism, (2) reflectance computed from polarization filtered intensities, and (3) color image.

4.1.1.

Stokes

The Stokes parameters are determined from an average of intensity measurements over area, wavelength, and solid angle. If we consider a multispectral system with relatively narrow bands, the spectral dependence of polarization information can be considered to be an additional useful information so that each channel $s$ senses a different Stokes vector.⁴⁹ Thus, from ${\rho }_j$, we estimate the Stokes vector $\mathbf{S}$ at each pixel position and for each spectral channel $s$ as

Eq. (7)

$${\mathbf{S}}_s=\left[\begin{array}{c}{S}_{0,s}\\ S_{1,s}\\ S_{2,s}\\ S_{3,s}\end{array}\right]=\left[\begin{array}{c}{S}_{0,s}=I_s\\ {\rho }_{s,0}-{\rho }_{s,90}\\ {\rho }_{s,45}-{\rho }_{s,135}\\ 0\end{array}\right],$$

where $I_s=S_{0,s}$ is the intensity component computed from Eq. (3), $S_{1,s}$ is the difference between intensities measured through 0 deg and 90 deg polarizers, and $S_{2,s}$ is the difference of intensities through 45 deg and 135 deg polarizers. $S_{3,s}$ is not considered in this work because we can sense only linear polarization.

The degree of linear polarization (DOLP) represents the amount of linear polarization in the light beam. It takes a value between zero for nonpolarized light and one for totally linearly polarized light, with intermediate values referring to partial polarization

Eq. (8)

$${\mathrm{DOLP}}_s=\frac{\sqrt{S_{1,s}^2+S_{2,s}^2}}{I_s}.$$

The azimuth angle of linear polarization (AOLP) is also computed from the Stokes components. It represents the angular orientation of the main axis of the polarization ellipse with respect to the chosen angular reference used for the system:

Eq. (9)

$${\mathrm{AOLP}}_s=\frac{1}{2}\arctan \left(\frac{S_{2,s}}{S_{1,s}}\right).$$

In Fig. 5, we plot the luminance and ${\mathrm{DOLP}}_{G_y}$ of the 24 MCCPP patches used for spectral calibration. First, we can see that there is a low but significant polarization signature for the patches. Second, we can see that we have an inverse relationship between luminance and DOLP, especially by looking at the values of the last six neutral patches (19 to 24). It can be explained by the fact that darker patches have a much smaller diffuse component (and thus a higher specular component, which is more polarized) due to a greater absorption of light. This effect is also shown in Fig. 6, in which darker balls exhibit a higher degree of polarization.

Fig. 5

(a) Luminance and (b) ${\mathrm{DOLP}}_{G_y}$ plot of the 24 MCCPP training samples.

Fig. 6

(a)–(c) Stokes components $S_{0,s}$, $S_{1,s}$, and $S_{2,s}$ for spectral band $s=G_{bg}$. (d)–(e) ${\mathrm{DOLP}}_s$ and ${\mathrm{AOLP}}_s$ for spectral channel $s=G_{bg}$.

A visualization of the Stokes transform on the scene called “resin balls” is shown in Fig. 6.

4.1.2.

Reflectance

The reflection of light upon a surface can be analyzed by an additive model, in which the total intensity is a diffuse component (resulting from the subsurface reflection phenomenon) plus a specular component (resulting from the surface reflection). Existing methods for separating reflection components use a polarization filter rotated in front of the image sensor.^8,50 It is often assumed that the diffuse component tends to be weakly polarized compared with the specular component. In our case, we believe that the polarization intensity filtering can benefit the reflectance estimation when a significant specular reflection occurs.

To this end, we compute the polarization filtered intensities,⁵¹ called $U_s$, by the following equation:

Eq. (10)

$$U_s=I_s-\sqrt{{(S_{1,s})}^2+{(S_{2,s})}^2}.$$

This has the effect of removing the polarization component from the total intensity $I_s$. Thus, we can compute the spectral reflectance for any pixel position from vector $\mathbf{U}$ instead of vector $\mathbf{I}$ [as in Eq. (5)]:

Eq. (11)

$${\widehat{\mathbf{r}}}^{\prime }=\widehat{\mathbf{M}}\mathbf{U}.$$

To verify the effect of removing the polarized intensity component, we compute the reflectance using either $\mathbf{I}$ or $\mathbf{U}$. We selected $5\times 5\text{pixel}$ regions near occluding boundaries of the objects, where specular reflection is assumed to have a greater influence on the DOLP of the reflected light. Results are shown in Fig. 7 for four regions of interest. It appears that we get a better fit when using $\mathbf{U}$ for the four cases.

Fig. 7

Measured and estimated reflectance of four samples using both spectral images computed from $S_0$ and $U$ [Eqs. (2) and (11)], respectively. Regions of interest (RoIs) is patch of size $5\times 5$ from where the spectral estimation curves are averaged. (a) RoI of controller scene, (b) spectral estimation of controller RoI, (c) RoI of Plier scene, (d) spectral estimation of Plier RoI, (e) RoI of Chart_b scene, (f) spectral estimation of Chart_b RoI, (g) RoI of Toy_2 scene, and (h) spectral estimation of Toy_2 RoI.

4.1.3.

Color

From by pixel reflectance data $\widehat{\mathbf{r}}$, we compute the corresponding International Commission on Illumination (CIE) $XYZ$ values as

Eq. (12)

$$\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]={\mathbf{C}}^t\widehat{\mathbf{r}},$$

where $\mathbf{C}$ ($36\times 3$) are the CIE (1931) color matching functions (CMFs).^52,53 Then, we convert the CIE $XYZ$ values to sRGB⁵⁴ for visualization. The $XYZ$ to sRGB transform is typically achieved in two steps: a first linear transform of $XYZ$ values to linear RGB values, followed by a gamma correction of 2.2 to adapt for nonlinear behavior of monitors.

4.2.

Database

We provide a database of 28 scenes with different materials, including plastic, wood, and metal. The scene called “chart” is custom plastic patches manufactured with a 3D printer, with a varying layer height parameter. Figure 8 shows the database in sRGB representation, after the color transform from Sec. 4. The brand logos are removed from the database images.The images are cropped to select the relevant content of the objects.

Fig. 8

sRGB visualization of the database after data transform to sRGB. The 28 scenes are captured by the hardware system depicted in Sec. 2. (a) Mini balls, (b) plastic bottle, (c) metallic bottle, (d) candies, (e) plastic chart blue, (f) plastic chart white, (g) plastic chart red, (h) scissors, (i) ColorChecker, (j) cutter, (k) electronics, (l) inkwell, (m) painting_1, (n) painting_2, (o) pens, (p) plastic, (q) pliers, (r) polarizers, (s) resin balls, (t) screw driver, (u) tape, (v) controller, (w) toys_1, (x) toys_2, (y) vernier scale, (z) wood 1, (aa) wood_2, and (ab) wood 2.

The code and database folder is organized as follows:

• 1. Data folder:

  • a. Pairs of preprocessed images for each scene, after demosaicking, flat-field correction, and geometrical correction. They are multipage.tif files (one page per polarization angle).

• 2. Misc folder:

  • a. Spectral calibrated matrix and

  • b. CIE XYZ values.

Three MATLAB scripts are provided to transform the data to Stokes, reflectance, and color images, respectively. The dataset and code are available in a GitHub repository via this link⁵⁵ or can be sent on request to the corresponding author.

5.

Discussion

Our stereo setup offers the possibility to perform reflectometry by fusing observations of the polarization state of light reflected off the surface from two different views.⁵¹ In a binocular stereo configuration, the specular intensity can be different from the two sensors, depending on the shape of the object to be considered and the lightning configuration. As an example, our stereo setup can be configured to optimize the roughness measurement by modifying the vergence, the working distance, or the focal length of the camera lenses. In the database, this is unconstrained because surfaces in the scene are multiple and unknown. But in the case of measurement on samples with a priori knowledge (roughness range, convexity shape, and index of refraction), the setup can be slightly adapted.⁵⁶ This is not investigated in this work and will be considered in future work.

Some drawback can come from the disparity computation, especially when imaging thick objects. From the stereo point of view, typical problems of matching occur when (1) there are occlusions (no corresponding data in one of the two images), (2) there are specular highlights, or (3) a pixel pertains to a more or less flat region. To be successful, the matching algorithm needs textures (but not periodic); if there is nothing to really match inside an image pair, the algorithm cannot make correspondences efficiently. Another potential factor comes from the use of an image pair taken with different spectral sensitivities.⁵⁷ This produces a noisy disparity map and thus noisy Stokes/spectral/color representations of data. We believe that an evolved disparity map computation method that is spectrally invariant is still to be developed.

From our knowledge, computing the reflectance spectra from polarimetric Stokes data ($S_1$ and $S_2$) has never been investigated in the literature. We verified that polarization information can help the reflectance estimation, whereas previous work assumed that all of the surfaces measured by a spectral camera are mainly diffuse. Some other works used a rotated linear polarizer in front of the spectral camera to filter globally the specular reflection. In our work, this is done by pixel because we have the measurement of the state of polarization for each pixel. Nevertheless, this method is limited by the amount of polarization intensity that is able to be filtered, which varies with the object and lightning configuration, i.e., the angle of incidence/reflection of the light. We believe that a dedicated separation of diffuse/specular components using spectral and polarization, such as in a prior work,⁸ can further improve the reflectance estimation for specular surfaces. This is not included in this work but can be investigated as an additional processing block to be included in the framework from Fig. 3.

This work opens perspectives in several application fields, from the stereo acquisition of spectropolarimetric images to new image processing of the data. The output of our framework, which is transformed data (Stokes, reflectance, and color) can be extended to specific surfaces and be an input to computer vision tasks such as feature extraction using principal component analysis, semantic segmentation, or defect detection.

6.

Conclusion

We developed a snapshot spectropolarimetric acquisition system. It has the advantages of a high spatial resolution and a relatively good spectral reconstruction precision. We designed a full processing pipeline that consists of preprocessing, geometric calibration, spectral calibration, and data transform. We also provide a method to better estimate the reflectance of specular surfaces using a filtering process based on the Stokes components. A database of 28 high-resolution spectropolarimetric images is made available online, along with the code to transform the data.

The configuration of the proposed system makes it possible to reconstruct the 3D spectral information using two complementary strategies: (1) the polarization information (the so-called “shape from polarization” method) or (2) the stereo depth computation. Image processing using both polarization and stereo information is to be investigated in future work.