3.1.

Mask-Based Image Decomposition

Adaptive spatial filtering, such as the bilateral filter and sub-band coding by Laplacian pyramid, have been introduced for image decomposition.^13,22–24 These methods usually have computational complexity and calculation burdens.⁴ In JPEG baseline, DCT coefficients include information for extracting sub-band images, which represent detail, base, and surround.²⁵ In this study, a simple method of implementation for image decomposition in the compressed field is proposed. For detail preservation, an image is separated into two layers: detail layer and base layer, which represent the local texture and large features, respectively. The detail layer represents local high-frequency components and the base layer represents low-frequency components locally. This is shown in Fig. 8. The input image is decomposed using the bilateral filter. It shows the characteristics of the base layer and detail layer, which represent locally blurred images and local textures, respectively.

Fig. 8

Images decomposed using the bilateral filter: (a) input image, (b) base layer, and (c) detail layer.

In JPEG baseline, DCT coefficients are computed within an $8\times 8\text{pixel}$ block. An image is converted from the spatial domain to the frequency domain with an $8\times 8\text{pixel}$ block size. Thus, it is possible to separate frequency components locally by splitting DCT coefficients in the block. Figure 9(a) shows the DCT block and location of coefficients for band splitting. The top-left coefficient is a direct current (DC) component of the block image. We assign a DC component to the surround layer, set DC and the low-frequency components into the base layer and high-frequency components into the detail layer. This image decomposition is implemented with a masking method. Figures 9(b) and 9(c) are macro masks for extracting the base layer and detail layer. The use of DCT allows the integration of local tone mapping in JPEG baseline.

Fig. 9

Mask-based image decomposition in $8\times 8$ discrete cosine transform (DCT) block: (a) $8\times 8$ DCT block, (b) mask for base layer, and (c) mask for detail layer.

For analysis of the proposed DCT mask, we compare a DCT mask and bilateral filtering to separate base and detail layers. Detail layer images through bilateral filtering and DCT mask splitting are shown in Fig. 10. Blurring is distributed throughout strong edges across foreground regions (trees) and background regions (sky and grass). As a result, a halo artifact occurs around strong edges. Blurred white outlines near edges in the detail layer from bilateral filtering causes the halo artifact. In contrast, because a proposed detail separation is conducted in an $8\times 8$ DCT block, the region where halo artifacts appear could not be larger than an $8\times 8$ block. For pair comparison, tone mapped images are produced by the same tone mapping operator (TMO) for each separated base layer. As shown in Fig. 11, the halo artifacts differentially appear in result images. A DCT masking method, consequently, leads to diminished halo effects.

Fig. 10

Detail layer images (a) by a bilateral filter and (b) by a proposed DCT mask splitting.

Fig. 11

Decomposed intensity images with tone mapped base and preserved detail (a) by a bilateral filter and (b) by a proposed DCT mask splitting.

3.2.

Luminance-Adaptive Tone Mapping for the Base Layer

Some tone mapping functions are based on an electrophysiological model that predicts the response of photoreceptors (rods and cones) at any adaptation level.⁶ This has usually been adopted by other authors to model perceived brightness; also, our tone mapping function is based on this model. The shapes of functions are similar to an S-shaped curve in the logarithm domain.^26,27 The basis function is given by

This basis function has been inspired by a power-function response in CIECAM02 (Ref. 28), which presents the postadaptation nonlinearities of cone responses. Figure 12 shows a relation between luminance intensity and cone responses for different adapting luminances. If the adapting luminance is high, cone responses are right-shifted. Cones change their sensitive area to a higher-intensity region for a higher adapting luminance. This processing, called as luminance adaptation, is enacted in local cones on the retina. This is the reason that cone responses are applicable in local tone mapping.

Fig. 12

Cone responses according to adapting luminance.

A proposed tone mapping function follows the basis function, Eq. (1), with luminance adaptation processing. From Eqs. (2) to (6), analyzed brightness sensitivity properties are applied to local tone mapping. The proposed parameters are based on Stevens’ brightness functions and the analysis conducted by Lee et al. Tone mapping functions are given by

In Eq. (3), the compression level factor, ${\alpha }_{\beta }$, is defined as the degree of local compression. It has been formulated only for controlling the compression level without overall tone changing. In the threshold luminance analysis shown in Fig. 3, for identical contrast ratio perception under separate adapting luminance, the physical contrast ratio must increase for a lower adapting luminance. This is because the human visual system is more sensitive to luminance change when the adapting luminance is lower, so a higher physical contrast ratio is needed to keep local detail consistent in a dim surround viewing. In other words, to perceive a consistent brightness contrast regardless of variations in the adapting luminance, the image contrast should change in an exponential decay toward the higher adapting luminance. First, a contrast sensitivity factor, $\alpha$, which is a basic factor of ${\alpha }_{\beta }$, determines the contrast range of the image. It is derived from the physical contrast ratio according to the relative adaptation luminance at each white luminance in Fig. 4. It depends on the property that a high contrast ratio is required at a low adapting luminance, whereas at a high adapting luminance, a relatively low contrast ratio is sufficient. $L_{\mathrm{a}}$ is set to 0.2 $L_{\mathrm{w}}$. $L_{\mathrm{w}}$ is obtained from a Gaussian-blurred intensity image in which the max luminance is set to $2000\mathrm{cd}/{\mathrm{m}}^2$ for outdoor scenes. In order to calculate $L_{\mathrm{w}}$, DC coefficients in DCT blocks are used, which are represented as ${\text{Surround}}_{\mathrm{dct}}$ in Fig. 6. Then, $\alpha$ is weighted by factor $\beta$ of Eq. (5). A weighting factor, $\beta$, is designed to meet the compression balance and prevent intensity saturation at higher $L_{\mathrm{a}}$ or gray out at lower $L_{\mathrm{a}}$. $\beta$ restricts a compression range at higher and lower adapting luminances. Here, to reduce the effect on the average luminance of an image by $\beta$, a mid-point value ${\alpha }_{\mathrm{m}}$ of the overall $\alpha$ is fixed.

In Eq. (6), the luminance level factor, $\delta$, is designed to properly adjust an average luminance level of a resulting image, based on the analysis of average luminance for consistent brightness perception in Fig. 13. In viewing scenes with uniform luminance distribution from dark to bright, the average luminance values will have a linear relationship with the adapting luminance, which is defined as $\sim 20\%$ of the white luminance value of each scene. However, the human visual system has a nonlinearity property to perceive average luminance for adapting luminance. Figure 13 shows the difference between the physical average luminance and the perceived average luminance. A bold line represents median luminance values from visual threshold luminance values of Fig. 3 and a dashed line shows the physical average luminance for a uniform luminance distribution. From the analysis, although adapting luminance linearly changes, the perceived average luminance is not proportional to the changing ratio of the adapting luminance. This means that as the adapting luminance is lowered, human visions need a relatively higher average luminance than physical luminance to preserve average brightness; on the other hand, a relatively lower average luminance is needed for higher adapting luminance. The larger $\delta$ generates a lower average luminance level in the output image. On the contrary, if an image is exposed for a short period, the small $\delta$ makes the output image brighter. The parameter, $\delta$, is derived based on the ratios between the values of bold and dashed lines for various adapting luminances; then it is adjusted using images with broad adapting luminance ranges.

Fig. 13

Average luminance analysis for consistent brightness perception.

Figure 14 provides the resulting images with different values of ${\alpha }_{\beta }$ and $\delta$. First, the compression level factor ${\alpha }_{\beta }$ controls the overall dynamic range of the image. For a higher ${\alpha }_{\beta }$ (weighting factor $\beta$: 2.5), the dynamic range of a represented image is more compressed (the bright portions are dimmed and the dark portions are lightened), whereas for lower values of ${\alpha }_{\beta }$ (weighting factor $\beta$: 0.5), the compression is lower. Figure 14(c) has a smaller ${\alpha }_{\beta }$ value than Fig. 14(b), and the dynamic range of Fig. 14(c) is larger than Fig. 14(b). ${\alpha }_{\beta }$ is formulated for applying the visual contrast characteristic to a tone mapping function according to Lee’s analysis for Stevens’ brightness function, which is shown in Fig. 4. Human vision requires a higher contrast ratio at a relatively lower adapting luminance,. Second, a luminance level factor $\delta$ effectively corrects an underexposed or overexposed image. The represented image is toned down for a high mean value of the input image based on the experimental results of Fig. 3. As shown in Figs. 14(d) and 14(e), the change of $\delta$ affects the average luminance of the output image. Based on this analysis, the factor $\delta$ is formulated for cooperating subjective experiments. These two fitted functions in Eqs. (5) and (6) are shown in Fig. 15.

Fig. 14

Toning results: (a) intensity image, (b) $\beta =2.5$, $\delta =0.1$, (c) $\beta =0.5$, $\delta =0.1$, (d) $\delta =0.1$, $\beta =0.5$, and (e) $\delta =4$, $\beta =0.5$.

Fig. 15

(a) Contrast sensitivity factor, $\alpha$, for adapting luminance $L_{\mathrm{a}}$. (b) Luminance level factor, $\delta$, for relative $L_{\mathrm{m}}$.

4.1.

Visual Gamma Correction

Overall tone reproduction through TMOs changes brightness contrast in images and the perceived lightness (or relative brightness) also changes as a function of different surround luminance.^1,19 In the experimental results of Bartleson and Breneman for complex stimuli, the exponent of the lightness function increases with increasing adapting luminance, so photographic images require gamma correction based on the estimated visual gamma. Photographic images typically viewed in dim surroundings are reproduced using a power function with a lower exponent value. Based on this, Lee et al. proposed the visual gamma given by

The visual gamma as a function of the adapting luminance means that gamma correction should be conducted adaptively with local luminance as for human vision. Therefore, we adopt the visual gamma as postprocessing after the proposed tone mapping. The output of tone mapping, $tm\text{Base}$, is gamma corrected according to the following equation:

4.2.

Sharpness Enhancement

In order to compensate sharpness loss by the procedure for JPEG baseline, we apply CSF-based sharpening gain, $R_{\mathrm{csf}}$, to an existing mask-based unsharpening method. CSF refers to the reciprocal of the minimum contrast ratio that human vision can perceive at each spatial frequency. In JPEG baseline, the sharpness enhancement is applied adaptively by luminance adapting the CSF properties based on the mask-based sharpness filter. We consider the contrast sensitivity of human vision for which a high contrast sensitivity means that objects are clearly visible. In order to design $R_{\mathrm{csf}}$, we compute the relative contrast sensitivity as a function of adapting luminance using certain maximum values at each adapting luminance: 5, 50, 100, 500, 1000, and $2000\mathrm{cd}/{\mathrm{m}}^2$. The maximum contrast sensitivity at $5\mathrm{cd}/{\mathrm{m}}^2$ is set as a reference point. Figure 16 shows the ratio of maximum values to the reference at each adapting luminance. The proposed gain, $R_{\mathrm{csf}}$, is fitted with a rational function using these points, which is given by

Fig. 16

Relative unsharp gain $R_{\mathrm{csf}}$ for adapting luminance.

The basis of the sharpness mask, $H(u,v)$, is shown in Fig. 17.²⁹ The final sharpness enhancement using the CSF properties is given as follows:

Fig. 17

Coefficients of the basis sharpness mask, $H(u,v)$.

4.3.

Color Compensation

Generally, during the tone mapping with a simplified s-curve, the ratio of RGB signals changes and color saturation would be reduced.³⁰ Although local tone mapping is applied to only the luminance channel, dynamic range compression generally results in an alteration of the ratio of chromatic channels and a reduction of color saturation. To correct this chronic defect of tone mapping, we adopt a simple method for color compensation which restores a ratio of color to luminance before tone mapping.³⁰ This method is given by

5.1.

Objective and Subjective Assessment

To conduct quantitative comparisons of the proposed method with existing tone mapping methods, several image assessment tools were employed, including the universal image quality index³² (UIQI), the no-reference perceptual quality assessment³³ (NRPQA), the colorfulness metric ratio^34,35 (CMR), and structural fidelity of tone mapped images³⁶ (S). According to the mathematical definition of UIQI, the closer the UIQI value is to one, the better is the image quality. Unlike UIQI, NRPQA does not need a reference image, as it is aimed specifically at no-reference quality assessment of JPEG compressed images considering blurring and blocking as the most significant artifacts. As such, it is suitable for DCT-based image evaluation. A higher NRPQA value indicates a better image quality. CMR indicates the extent of color in the resulting image relative to the reference image. $S$ indicates local structural fidelity measure based on structural similarity (SSIM),³⁷ which contains three comparison components: luminance, contrast, and structure. Compared with SSIM, the luminance comparison component is missing in $S$ since TMOs locally change original intensity. Using all four of these numerical assessments, we compared the proposed method with previous approaches, such as iCAM06,³ a photographic tone reproduction based on dodging and burning with a zone system³³ (PTR), integrated surround retinex model³⁸ (ISRM), and retinex-based adaptive filter method³⁹ (RAFM).

Resulting images for these methods are shown in Figs. 18Fig. 19Fig. 20–21. Evaluation results for the full set of images are shown in Fig. 22. Note that according to UIQI results, the proposed method trails PTR by a slight margin, but it is competitive with other methods. NRPQA, which is a perceptual assessment for JPEG compressed images, presents a more objective evaluation than UIQI. Note that the proposed method has the best NRPQA scores among all methods tested. The best score is assigned to the most robust method about blurring and blocking artifacts. According to CMR scores, the proposed method shows comparatively good performance and no halo artifacts, as is apparent in images processed using ISRM. Finally, from S scores, it is confirmed that the proposed method has structural fidelity equal to or higher than those of the other methods.

Fig. 18

Result images for img3.

Fig. 19

Result images for img4.

Fig. 20

Result images for img5.

Fig. 21

Result images for img10.

Fig. 22

Evaluation results using three metrics: universal image quality index, no-reference perceptual quality assessment, colorfulness metric ratio, and structural fidelity of tone mapped images.

In addition to objective evaluation, we conducted the psychophysical experiment based on score rating. An original image is first presented; then reconstructed images by each method on a gray background are simultaneously shown on a display device: LG 47LM6700. Participants in the experiment are instructed to rate a score with 0-to-10 range for each attribute: global tone, local contrast (halo), sharpness, and color (naturalness). In the experiment, the total number of collected scores is $7(\text{images})\times 6(\text{methods})\times 4(\text{attributes})\times 6(\text{participants})=1008$. The average scores and standard deviations are presented as color bars and error bars in Fig. 23, respectively. The result shows that the proposed method is highly rated on the psychophysical experiment.

Fig. 23

Subjective evaluation using preference assessment.

Our overall assessment, based on qualitative comparison of the entire set of test images, confirms that the proposed tone mapping method produces colorful, high-contrast images with strongly enhanced details. In addition, according to subjective comparison, the proposed method has good preference scores for four-view, such as global tone, local contrast, sharpness, and color. All resulting images by the proposed method and the original images are shown in Fig. 24.

Fig. 24

Result images: input images (top) and out images (bottom) by the proposed method.

5.2.

Computation Time

We compute the computation time of the methods with the test setup as shown in Fig. 25. Considering the novelty of the proposed method that is inserted in JPEG baseline, JPEG encoding and decoding are conducted after tone mapping for the previous methods. For different resolutions ($853\times 480$, $1280\times 720$, $1920\times 1080$), computation times in MATLAB® are listed in Table 1 (CPU: Intel i7-2600K 3.40 GHz, RAM: 4 GB). In Table 1, the computation time of our method is faster than those of iCAM06 and RAFM, but similar to those of PTR and ISRM. Compared with iCAM06 and RAFM adopting time-consuming tasks for edge preserving and anti-halo, such as the bilateral filter and anisotropic Gaussian functions, our method improves the edge resolution and halo artifact while saving a lot of computation time.

Fig. 25

Test setup for checking runtime.

Table 1

Computation time of methods in MATLAB® (in seconds).