1.

Introduction

Abnormal distribution of skin absorbing chromophores (e.g., epidermal melanin and dermal hemoglobin) is a clinical feature typical for burns, bruises, and various kinds of skin malformations, including tumors.¹ Skin chromophore distribution maps can provide qualitative and quantitative information on the severity of pathology, its development, and/or skin recovery processes. Simple and fast skin chromophore mapping is of special interest for dermatologists, cosmetologists, oncologists, emergency clinicians, surgeons, and forensic experts, as it facilitates decision taking and further strategy development. However, the procedures with commercially available devices that can provide skin chromophore distribution maps (e.g., SIAscope)² are time-consuming, the equipment is expensive and is cable-connected to a computer, which limits its wide applications in personalized medicine and in the field conditions.

Each chromophore has its specific absorption spectrum,³ so the remitted light intensity from a spot of skin (containing one or more chromophores) depends on the detection wavelength and composition of the comprised chromophores. A set of several spectral images of the same skin area, captured within different wavelength bands, can be converted into clinically significant parametric images—chromophore distribution maps.⁴ There are two main approaches for obtaining spectral images: by narrowband spectral filtering of the image sensor at spectrally broadband illumination of the target area or by means of spectrally narrowband illumination. The first approach is widely used in hyperspectral imaging (HSI) systems that capture sequential images of the target at several tens of adjacent spectral bands, e.g., using interference filters,⁵ acousto-optical, or liquid crystal filters.⁶ Commercial and custom-designed HSI systems have been successfully used for skin chromophore mapping.^5,7–9 However, routine clinical applications of the HSI systems are limited due to their robustness, expensiveness, and the need for powerful computing resources to process numerous spectral images (including those eventually not containing useful information). Another advanced chromophore-sensitive technology, spatial frequency domain spectroscopy,⁹ faces similar limitations. The HSI image acquisition process may last from several seconds⁹ to a minute or longer,¹⁰ so movements of the skin target area during the procedure can cause image artifacts. Special image stabilization algorithms¹¹ could reduce or fully avoid the motion artifacts, but it takes extra time and computing resources. Additional problems arise if the extinction coefficient of a particular chromophore is notably changing within a single band wavelength interval, typically 10 to 20 nm or more; in such cases, calculations of complicated integrals comprising spectral features of the chromophore absorbance and the filtered HSI band shape are necessary.

The second approach, spectrally selective illumination, has been mainly used in skin multispectral imaging (MSI) systems where sequentially switchable nonoverlapping spectral bands of illumination (e.g., by several LEDs with different peak wavelengths at typical spectral half-bandwidth $\sim 20$ to 50 nm) are exploited for obtaining spectral images.^12,13 Advanced MSI systems are portable and self-sustained, much cheaper than HSI systems, and better adapted for clinical use. The MSI processing procedure is also faster due to lower number of the spectral images, typically between 3 and 8. Still, skin motion artifacts and processing of relatively broadband spectral images may cause serious problems.

To improve the performance of the MSI chromophore mapping, acquisition time of the spectral images and the spectral bandwidth of illumination have to be minimized. An optimal approach would be the single-snapshot technique dealing with monochromatic (single-wavelength) spectral images. A method for extraction of three monochromatic spectral images from a single snapshot RGB image data set, taken under simultaneous three-wavelength illumination, has been recently proposed¹⁴ and experimentally validated on color targets.¹⁵ The previous studies confirmed that separate readings of three RGB detection channels are sufficient to spectrally isolate reflected light at each illumination wavelength, if the crosstalk among those channels is properly taken into account.¹⁶ A table-top setup comprising RGB board-camera and fiber optic system for simultaneous illumination of target by the 473-, 532-, and 659-nm laser lines has been tested for skin chromophore mapping, as well.¹⁷ In the results, physiologically plausible chromophore maps of melanin, oxyhemoglobin, and deoxyhemoglobin in pigmented and vascular skin lesions have been obtained by a single snapshot. This laboratory setup, however, was not well-adapted for patient measurements, so new designs of compact/portable equipment for skin chromophore mapping in clinical environment are of essential interest.

Currently, there are about two billion smartphones in use worldwide,¹⁸ many of them in the pockets of doctors or other clinical personnel. Smartphones comprise high-resolution digital RGB cameras, white LED illuminators, multifunctional displays, and powerful data processors with the ability to transmit the data (including specific images) via mobile networks. This set of features makes smartphones attractive for potential diagnostic applications, e.g., evaluation of human skin condition. Several dermatology-specific mobile applications, including those intended for skin melanoma early detection, have been developed.^19–21 If aiming at skin chromophore distribution mapping, the spectral information contained in smartphone digital images, taken under the built-in white LED illumination, appears to be insufficient. Narrowband filtering of the white illumination (e.g., by a set of interference filters)²² could improve the situation, but more promising from this point seems the use of specially designed small size multispectral add-on illuminators, e.g., based on narrowband color LEDs.²³

This paper describes implementation of the RGB single-snapshot approach for the mapping of three main skin chromophores using a smartphone with attached compact three-wavelength laser illuminator. Image processing principles, design of the system, and the first results of its clinical tests are discussed below.

2.

Method

2.1.

Concept

The general concept of snapshot skin chromophore mapping at fixed wavelengths is illustrated in Fig. 1. Let us suppose that RGB color image of a skin pathology spot, surrounded by healthy skin, is captured under illumination that comprises only three equal intensity spectral lines at wavelengths ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ (the vertical lines in Fig. 1). With respect to the spectral sensitivity of an RGB image sensor and the cross-talk between its detection bands at the particular wavelengths, three monochromatic spectral images are extracted from the color image data set by the technique described in Refs. 1415.16.–17. The three-chromophore skin model²⁶ suggests that the dominant absorbers in skin at any of the fixed wavelengths ${\lambda }_j$ ($j=1$, 2, 3) are oxyhemoglobin (further abbreviated as a), deoxyhemoglobin (b), and melanin (c)—see the crossings of their absorption curves with the vertical lines in Fig. 1. If the skin surface reflection is suppressed (e.g., by means of crossed polarizers), variations in chromophore composition would lead to changes of the diffusely reflected light intensities at each of the fixed wavelengths. Such variations in the pathology region relative to the healthy skin can be estimated by measuring reflected light intensities from equally sized regions of interest in the pathology ($I_j$) and the adjacent healthy skin ($I_{0j}$). The ratios $I_j/I_{0j}$ at each pixel or pixel’s group of three monochromatic spectral images contain information on the concentration increase or decrease of all three regarded chromophores $\mathrm{\Delta }{c}_i$ ($i=a$, $b$, $c$), which can be further mapped over the whole image area.

Fig. 1

Absorption of three main skin chromophores at three fixed wavelengths.^24,25

2.2.

Image Processing

The above-mentioned intensity changes at three exploited wavelengths due to absorption can be represented in terms of the Beer–Lambert–Bouguer law²⁷

Eq. (1)

$$\{\begin{array}{c}\ln \left(\frac{I_1}{I_{01}}\right)=-l_1[\mathrm{\Delta }{c}_a·{ϵ}_a({\lambda }_1)+\mathrm{\Delta }{c}_b·{ϵ}_b({\lambda }_1)+\mathrm{\Delta }{c}_c·{ϵ}_c({\lambda }_1)]\\ \ln \left(\frac{I_2}{I_{02}}\right)=-l_2[\mathrm{\Delta }{c}_a·{ϵ}_a({\lambda }_2)+\mathrm{\Delta }{c}_b·{ϵ}_b({\lambda }_2)+\mathrm{\Delta }{c}_c·{ϵ}_c({\lambda }_2)]\\ \ln \left(\frac{I_3}{I_{03}}\right)=-l_3[\mathrm{\Delta }{c}_a·{ϵ}_a({\lambda }_3)+\mathrm{\Delta }{c}_b·{ϵ}_b({\lambda }_3)+\mathrm{\Delta }{c}_c·{ϵ}_c({\lambda }_3)]\end{array},$$

where ${ϵ}_i({\lambda }_j)$ is extinction coefficients of three regarded chromophores at three exploited wavelengths and $l_j$ is absorption path length in skin at a particular wavelength. Chromophore concentration increase or decrease at each image pixel (or selected group of pixels) is found by solving the linear equation system (Eq. 1) with abbreviated measured quantities $k_j=\ln (I_j/I_{0j})$:

Eq. (2)

$$\begin{array}{c}\mathrm{\Delta }{c}_a=A_1·k_1+A_2·k_2+A_3·k_3\\ \mathrm{\Delta }{c}_b=B_1·k_1+B_2·k_2+B_3·k_3\\ \mathrm{\Delta }{c}_c=C_1·k_1+C_2·k_2+C_3·k_3\end{array}.$$

The coefficients $A_j$, $B_j$, and $C_j$ comprise numerical values of the corresponding chromophore extinction coefficients ${ϵ}_i({\lambda }_j)$ and absorption path lengths $l_j$ at the three exploited wavelengths. The respective extinction coefficients for oxyhemoglobin, deoxyhemoglobin, and melanin for our working wavelengths 448, 532, and 659 nm are presented in Table 1.

Table 1

The extinction coefficients of three main skin chromophores24,25 at three exploited wavelengths.

Chromophore	ϵ (448 nm) (1/mol.cm)	ϵ (532 nm) (1/mol.cm)	ϵ (659 nm) (1/mol.cm)
Melanin	1935	1170	541
Oxyhemoglobin	67,044	45,400	320
Deoxyhemoglobin	173,320	41,336	3227

The Beer–Lambert–Bouguer law is originally addressed to absorption of a light beam passing through a slab of substance with thickness $x$. If this law is adapted to the diffusely remitted light geometry, slab thickness $x$ has to be replaced by the mean path length $l_j$ of the backscattered photons inside the skin at a particular wavelength. To estimate the $l_j$ values with respect to light scattering properties, Monte Carlo (MC) simulations in a frame of the two-layered skin model²⁸ (epidermal and dermal layers followed by the infinite subcutaneous tissue layer) were performed. Optical parameters of the tissue layers for the considered wavelengths were averaged from the number of available sources for typical Caucasian skin.^29–33 See Table 2.

Table 2

Optical properties of biotissue layers used in the simulations (λ, wavelength, μs, scattering coefficient, μa, absorption coefficient, g, anisotropy factor, n, refractive index).

	λ (nm)	μs (mm−1)	μa (mm−1)	g	n
Epidermis	659	20	0.20	0.90	1.36
	532	27	0.60	0.85	1.36
	448	33	1.0	0.84	1.36
Dermis	659	18	0.12	0.90	1.40
	532	22	0.30	0.85	1.40
	448	27	0.65	0.84	1.40
Subcutaneous tissues	659	7	0.07	0.80	1.40
	532	10	0.40	0.76	1.40
	448	14	0.70	0.75	1.40

The simulations have been performed for various thicknesses of the epidermal layer (200, 150, and $100\mu \mathrm{m}$) and a dermis of $1800\mu \mathrm{m}$). In the simulations, uniform illumination beam with a diameter of 50 mm has been accounted. The diffusely backscattered (remitted) photons were collected from the same area accounting for the aperture angle of 10 deg. MC simulations were performed using a parallel (message passing interface) MC code for light propagation in biotissue-like media developed and verified in the earlier studies.^34–36 As the outcome, the distributions of the detected photons over their total path length and maximal penetration depth were generated. The mean path length and the mean penetration depth were calculated by weighted average taking into account the weight of the corresponding photon packets. The simulation results for skin with an epidermal thickness of $200\mu \mathrm{m}$ are presented in Fig. 2. MC simulations showed increased photon penetration depth and total path length with growing wavelength, the mean path length of the remitted photons being $\sim 4$ to 5 times longer than their average penetration depth. The MC calculation results are presented in Table 3. One can see that the mean path length of photons in skin at a fixed wavelength is quite insensitive ($<10\%$) to the twofold increase/decrease of the epidermal thickness. Consequently, different skin epidermal thicknesses should not seriously influence the data obtained by the proposed technique. The clinical images presented in Sec. 3 were processed using the MC-estimated mean path length values for 0.2-mm-thin skin epidermis.

Fig. 2

Monte Carlo simulated distributions of the detected photons over their total path length (a) and penetration depth (b) inside the skin (thickness of epidermis: $200\mu \mathrm{m}$) for three exploited wavelengths. The mean path length and the mean penetration depth were calculated by weighted average taking into account weight of the corresponding photon packets.

Table 3

MC-estimated penetration depths and path length in skin of the remitted photons. Last column: ratio between the mean path length and the mean penetration depth.

Wavelength (nm)	Epidermal thickness (mm)	Mean penetration depth (mm)	Mean path length in skin (mm)	The calculated ratio factor
448	0.20	0.16	0.83	5.19
	0.15	0.17	0.88	5.18
	0.10	0.18	0.92	5.11
532	0.20	0.29	1.50	5.17
	0.15	0.31	1.57	5.06
	0.10	0.32	1.64	5.13
659	0.20	0.75	2.93	3.91
	0.15	0.76	2.95	3.88
	0.10	0.76	2.98	3.92

To summarize, our algorithm for image processing includes the following steps:

• 1. RGB image comprising pathology and adjacent healthy skin is captured under illumination by three discrete wavelengths (disabling all automatic settings of the camera).

• 2. Three monochromatic spectral images are extracted from the digital image data set using the RGB crosstalk correction algorithm,¹⁴ with respect to actual relative intensities of the illumination lines.

• 3. Equal area regions of interest are selected in the pathology and healthy skin zones of the monochromatic images.

• 4. Pathology-to-healthy relative spectral reflectances $I_j/I_{0j}$ at the three exploited wavelengths and the values $k_j=\ln (I_j/I_{0j})$ are calculated for all image pixels or selected pixel’s groups.

• 5. The concentration increase/decrease of three chromophores in the pathology is calculated (2) for each pixel or pixel’s group.

• 6. Distribution maps of three skin chromophores are constructed and displayed.

2.3.

Design of the Illumination Unit

A compact smartphone-compatible three-wavelength illuminator has been designed, assembled, and tested in laboratory and clinics. The illumination unit comprised six laser modules, an optical element for laser beam management, and an electronics compartment. Figure 3(a) shows the design details and outlook of the operating prototype with a smartphone on it [Fig. 3(b)]. To ensure uniform three-wavelength illumination of around target area, an optical element with a flat ring-shaped laser diffuser was developed.³⁷ The illumination wavelengths 448, 532, and 659 nm were emitted by three pairs of compact 20-mW power laser modules (models PGL-DF-450 nm-20 mW-15011564, PGL-VI-1-532 nm-20 mW-15030443, and PGL-DF-655 nm-20 mW-150302232, Changchun New Industries Optoelectronics Tech. Co., Ltd). Laser modules [1, Fig. 3(a)—shown 3 out of 6] of each equal-wavelength pair were mounted at opposite sides on the internal wall of a hollow three-dimensional (3-D)-printed plastic shielding cylinder (2); the round bottom opening of this cylinder (diameter 40 mm) was in contact with skin and formed the field of view for the smartphone camera, situated 80 mm apart. All six coaxial laser beams were pointed to the 45-deg conical reflecting edge of a transparent disc (3) (beam collector) made of standard Plexiglas; after reflections they were turned radially toward the internal ring-shaped flat milky-Plexiglas diffuser (4). The upper and side surfaces of the collector/diffuser unit were mirrored by a vacuum-sputtered Al-coating. As a result, the flat diffuser (4) evenly illuminated the 65-mm distant skin target area simultaneously by three laser wavelengths. More details on the laser beam collector-diffuser are provided in Fig. 4.

Fig. 3

(a) 3-D model of the three-wavelength illumination unit and (b) the mobile prototype with smartphone on it. (1) Laser modules (three pairs, 448–532–659 nm), (2) shielding cylinder, (3) collector of laser beams, (4) flat ring-shaped diffuser of laser light, (5) sticky platform for the smartphone, (6) electronics compartment.

Fig. 4

Optical scheme of the laser beam collecting-diffusing element.³⁶ The arrow denotes one of six coaxial laser beams that are turned radial by 45-deg conical reflecting edge of the transparent disk 3 and then downscattered by means of the internal diffusing ring 4. Upper surfaces of both disks are Al-mirrored. Dimensions: external diameter 50 mm, inner-outer diameters of the diffusing ring 10 to 40 mm, thickness 3 mm.

The smartphone—model Google Nexus5, 8 Mpx image sensor SONY IMX179 with known RGB-sensitivities³⁸—was placed on a flat sticky platform (5) [Fig. 3(a)] with a round window for the smartphone rear camera, coaligned with the internal opening of the diffuser (4). The advantage of flat sticky platform design^23,37,39 is the ability to fix most models of smartphones or tablet computers, independent of their dimensions; it was implemented by means of a double-sided tape. The round camera window was covered by a film polarizer (high contrast linear polarizing film⁴⁰); another film with orthogonal direction of polarization covered the diffuser (4) from the bottom, so avoiding detection of skin surface-reflected light by the smartphone camera. The estimated spatial resolution of the imaging system was better than 0.1 mm.

A block diagram of the three-wavelength illumination device is shown in Fig. 5. Microcontroller (STM32F103) managed operation of the three-laser power modules via control of the laser power supply unit (PSU). Each pair of laser modules could be switched on and off independently by means of the control buttons. Two rechargeable batteries (Samsung 18650 3.7V Li-Ion, 2.8 Ah) and laser power supply circuits were placed in the 3-D-printed plastic compartment (6) below the platform (5). Target illumination intensity was adjusted to ensure linearity of photodetection by the smartphone camera; its automatic settings were switched off by the AZ Camera software, so the R-, G-, and B-outputs of the image sensor responded according to the spectral sensitivities of its three detection bands.³⁸

Fig. 5

Block diagram of the three-wavelength illumination unit, where PSU stands for power supply unit.

The above-described optical design ensured acceptable uniformity of target illumination. It was initially checked at all combinations of illumination wavelengths (448, 532, and 659 nm) and image sensor detection channels (R, G, B), using white paper as the target. Figure 6 shows the obtained illumination intensity distributions, detected at separated RGB output channels in two cases: (i) when only one wavelength was used for illumination (659 nm detected at R-channel, 532 nm detected at G-channel, and 448 nm detected at B-channel, the upper row) and (ii) when all three wavelength of illumination were simultaneously switched on (the lower row).

Fig. 6

Uniformity of white paper illumination at different illumination–detection combinations: (a) R-image at single-wavelength 659-nm illumination; (b) G-image at single-wavelength 532-nm illumination; (c) B-image at single-wavelength 448-nm illumination; (d–f) R-, G-, and B-images at simultaneous three-wavelength illumination. Color scale represents relative intensities. Diameter of the illuminated area is 40 mm.

3.

Results

After processing the clinical images, skin chromophore maps were constructed and changes of malformation’s chromophore content with respect to the adjacent healthy skin could be evaluated. Clinical models¹ assume that the pigmented skin lesions represent an abnormal increase of epidermal melanin and the vascular lesions—increased supply of dermal arterial blood (rich of oxyhemoglobin) to the superficial layers of skin. As initially expected, we observed notable melanin content increase in all cases of both pigmented malformations: nevi and seborrheic keratosis. Increase of oxyhemoglobin content and decrease of deoxyhemoglobin were observed in all examined vascular malformations—hemangiomas. For illustration, images for these clinical cases related to each of the examined three pathologies are presented in Figs. 7Fig. 8–9.

Fig. 7

(a) RGB image (scale bar: 5 mm) and the corresponding maps representing chromophore concentration changes for three cases of pigmented nevi: (b) oxyhemoglobin, (c) deoxyhemoglobin, and (d) melanin. Units of the color scale: millimoles (mM).

Fig. 8

(a) RGB image (scale bar: 5 mm) and the corresponding maps representing chromophore concentration changes for three cases of pigmented seborrheic keratosis: (b) oxyhemoglobin, (c) deoxyhemoglobin, (d) melanin. Units of the color scale: millimoles (mM).

Fig. 9

(a) RGB image (scale bar: 5 mm) and the corresponding maps representing chromophore concentration changes for three cases of vascular hemangioma: (b) oxyhemoglobin, (c) deoxyhemoglobin, (d) melanin. Units of the color scale: millimoles (mM).

The color photo of a malformation, taken under simultaneous three-wavelength illumination (a), is compared with the obtained concentration distribution maps of oxyhemoglobin (b), deoxyhemoglobin (c), and melanin (d). All pathology data are related to the adjacent healthy skin ($\mathrm{\Delta }c=0$) so that only increased and decreased concentrations are quantified by colors. Figure 7 shows how melanin content increases in nevi, without essential changes in the hemoglobin content. Similar responses were obtained from the other pigmented skin malformation, seborrheic keratosis (Fig. 8). Quite different chromophore composition changes were recorded for vascular hemangiomas (Fig. 9); melanin concentration in the pathology remained practically unchanged while the oxyhemoglobin content notably increased and the deoxyhemoglobin content decreased in comparison to healthy skin.

The color scales in Figs. 7–9 are calibrated in millimoles, so the increased or decreased chromophore content can be quantitatively analyzed and/or compared. It might help dermatologists to characterize the severity of pathology if clinical thresholds for particular chromophores in the malformation are established.

The grainy structure of chromophore maps at Figs. 7–9 is most probably caused by formation of laser speckles on the skin surface. This is a coherent effect and cannot be avoided even in scattered laser light.

4.

Discussion

Generally, the obtained results have confirmed viability of the proposed approach for single-snapshot skin chromophore mapping with a smartphone. The compact add-on illuminator design [Figs. 3(a) and 4] ensured uniform skin illumination simultaneously at three-laser wavelengths (Fig. 6). The previously developed RGB monochromatic imaging concept, complemented by MC simulations of photon trajectories in skin, was successfully applied for mapping of pathologic concentration changes of skin melanin, oxyhemoglobin, and deoxyhemoglobin (Figs. 7–9). Results of this study may facilitate further developments of relatively simple and compact optical equipment for clinical diagnostics and monitoring; they also highlight the directions of further research, related to yet unsolved problems.

One of them is insufficient quality of chromophore maps due to laser speckles formed on the illuminated skin. In order to obtain speckle-free images, incoherent spectral line sources could be used. As an option, we assembled a three-wavelength illuminator based on high frequency electrodeless spectral lamps filled with an Hg–Cd vapor mixture. However, such an illuminator required an additional high quality multibandpass filter which made the design too expensive, so probably more efforts are needed to develop other kinds of speckle-free multiwavelength illumination sources.

Another issue of further improvements is the skin absorption model; only three main skin chromophores were considered in this study, while bilirubin, $\beta$-carotene, lipids, and water may also contribute as skin absorbers.^3,29 Therefore, image-processing algorithms based on more advanced skin models (e.g., 4-chromophore or 5-chromophore) are to be developed in the future, along with further hardware developments. Smartphones comprise the conventional RGB image sensors, so only three output channels (R-, G-, and B-images) can be exploited; it limits to three the number of chromophores to be mapped by one snapshot. In this respect, a technology using several sets of simultaneous three-wavelength illumination should be developed in the future.

From the point of clinicians, the above-described technique would become useful for quantitative diagnostics only when a “gold standard” that sets the threshold (critical) values for increased chromophore concentrations, related to particular pathologies, will be available. To achieve that, serious clinical trials with a much larger number of patients and statistical analysis of the results have to be carried out. Severity of skin pathology could be quantified also in terms of chromophore concentration ratio “pathology/healthy”; further research in this direction would be necessary as well. In addition, results of the image processing—chromophore distribution maps—should appear on the screen of the used smartphone. Two approaches for that are to be examined: development of specific software application for particular (most popular) smartphone models or using a distant server that receives the captured images via mobile network and after processing sends back the calculated chromophore maps. The second option seems advantageous since it can be applied independently on the used smartphone model. As for potential hardware improvements, more handy for clinical use probably would be a separate multiwavelength illuminator which is first properly placed on the skin pathology and only then the smartphone attached for image capturing.

The mean path length in skin of the backscattered photons has been evaluated by means of MC simulations in this work. It would be useful to develop in the future techniques for direct determination of this parameter, e.g., using the photon time-of-flight measurement technique.⁴¹

5.

Conclusions

This study confirmed the possibility to map three main skin chromophores by a single snapshot of a smartphone camera under combined spectral line illumination. Millions of professionals nowadays use smartphones, so the proposed relatively simple and inexpensive approach potentially may find real clinical and forensic applications. The main challenges of the future work will be: (i) to minimize or fully avoid laser speckle artifacts in spectral images of the illuminated skin area; (ii) to improve the image-processing algorithms and hardware details; (iii) to expand the range of chromophores that can be mapped, e.g., to add bilirubin which shows up in cases of skin bruises and postsurgery recovery.