1.

Introduction

Computational cameras with multifocusing functions have combined optics and signal processing techniques to overcome limitations in a pure digital signal processing approach, such as in Ref. 1. The multiple color-filter aperture (MCA)^{2, 3} system is one such computational camera technique, by which a single camera system can estimate depth information of objects at different distances and perform multifocusing. The existing MCA system, however, exhibits several problems, including low exposure, color mixing, deviation of color convergence, and divergence of light rays. These problems result in nonideal point-spread functions (PSFs), which deteriorate the overall quality of the image even after color channel registration and fusion (CRF). In this work, we present: 1. fundamental principles of the MCA computational camera, 2. the automatic CRF process, and 3. a real-time image restoration system for improving the quality of images acquired by the MCA system. For the image restoration process, we first estimate the PSF based on the method proposed in Ref. 4, and then design the truncated constrained least-squares (TCLS) restoration filter using the estimated PSF. Most image restoration filters tend to amplify noise and produce ringing artifacts, and the proposed TCLS filter is not an exception. For removing such artifacts, we additionally propose a spatially adaptive noise smoothing (SANS) algorithm using adaptively weighted mixing between the color-channel-aligned image and the restored image by the TCLS filter.

2.

Automatic Color Channel Registration and Fusion for the Multiple Color-Filter Aperture Camera

The aperture of an optical system is the opening device that adjusts the amount of incoming light. Generally, the center of an aperture is aligned to the optical axis of the lens, and the convergence pattern on the imaging plane forms either a point or a circular region. However, if the center of the aperture is not aligned to the optical axis, the image formation of the sensor deviates from the optical axis. As shown Fig. 1, the MCA system consists of three apertures with R, G, and B color filters. The main advantage of the MCA system is that it provides additional depth information of objects at different distances, as shown in Figs. 1, 1, 1. At the cost of additional depth information, the image acquired by the MCA system contains a certain amount of out-of-focus blur, which results from color misalignment. For solving the color misalignment problem, manual registration of color channels and fusion methods have been proposed in Refs. 2, 3.

Fig. 1

(a) The MCA camera system, (b) an object is on the in-focus position b, (c) an object is on the near-focus position c, and (d) an object is on the far-focus position a.

We propose cluster-based image segmentation and phase correlation matching (PCM) for automatically registering pixels in three color channels. The proposed cluster-based segmentation algorithm employs K-means clustering and block partitioning. K seeds in the clustering algorithm are automatically determined by using the hill-climbing method⁵ in the 3-D CIELab histogram of an input image, as shown in Fig. 2. Figure 2 describes the algorithms for the 1-D case. A resulting salient region is shown in Fig. 2. After labeling salient regions, the input image is partitioned into same-sized blocks, as shown in Fig. 2. These partitioned blocks generate the region of interest (ROI) map for the color shift, as shown in Fig. 2. We estimate the amount of color shift in the block of the ROI map by using the PCM proposed in Ref 6. The registered regions are shown in Fig. 2, and the finally fused image is shown in Fig. 2. Since the fused image can be considered a geometrically aligned version of the input image, we can further enhance the image by removing focusing blur as described in the following section.

Fig. 2

A sample procedure of the proposed CRF algorithm; (a) an input acquired by the MCA system, (b) the hill-climbing process, (c) the salient regions, (d) block partitioning, (e) new ROIs, (f) registered regions, and (g) the fused image.

3.

Image Restoration for Enhancing the Multiple Color-Filter Aperture Output Image

We proposed a digital image restoration approach for overcoming fundamental limitations in the original MCA camera. We first estimate the PSF by assuming that the out-of-focus blur results in an isotropic or circularly symmetric PSF, and that the input image contains an ideal step edge at the linear boundary of an object. Using the estimated PSF,⁴ we generate the frequency response of the constrained least-squares (CLS) filter. The full-support spatial domain filter coefficients are obtained by the inverse Fourier transform of the CLS filter. Since significant energy is concentrated in the main lobe and its vicinity, we can have a smaller support spatial-domain filter by an appropriate truncation. Base on empirical optimization for both performance and efficiency, we adopted the raised-cosine function for the truncation window. A similar finite impulse response (FIR) restoration filter has been proposed for the extended depth of field (EDOF) system in the Bayer domain.⁷ In this work, we used the $13\times 13$ truncated CLS (TCLS) filter with consideration for the size of the PSF.

Although the TCLS filter can remove out-of-focus blur with a limited amount of noise, there still exist undesired artifacts in the restored image. To completely remove such noise artifacts, we employ an additional space-adaptive mixing algorithm that takes detailed high-frequency regions from the restored image and takes flat regions from the input out-of-focus blurred image. For properly selecting one of two images, we compute the local variance at each pixel $(x,y)$ as

4.

Experimental Results

For evaluating the focusing performance of the proposed sets of algorithms, we used color images of size $1504\times 1000$ acquired by the MCA system with a lens focal length of $150\mathrm{mm}$ . The MCA output image, which contains multiple objects at different distances and the ROI map, is shown in Figs. 3 and 3, respectively. The result of CRF is shown in Fig. 3, where we note that the misalignment is corrected, while the out-of-focus blur still exists. Results of the TCLS restoration filter, the corresponding alpha map, and the alpha map-based SANS algorithm are respectively shown in Figs. 3, 3, 3.

Fig. 3

(a) The image acquired by the MCA system, (b) the ROI map image, (c) the shifted and aligned image by CRF, (d) the restored image by the TCLS filter, (e) the alpha map image, and (f) the result of the SANS.

5.

Conclusion

The original version of the MCA system exhibits inherent limitations such as a manual CRF process and remaining focusing blur after CRF. For overcoming these problems, we propose a complete solution of real-time restoration incorporated into the MCA system in the form of the FIR filter structure. Furthermore, the proposed method can fully or partially be applied to a computational camera such as the MCA or EDOF system. Experimental results show that the proposed algorithm provides high visual quality by amplifying details of the image.