1.

Introduction

Integral imaging (II), which was originally called integral photography, is one of the most attractive techniques in the field of three-dimensional (3-D) displays. It has many advantages over other 3-D display techniques. For example, it provides continuous viewpoints within a specific viewing angle in both horizontal and vertical directions and does not need any special glasses for viewers.¹ Compact system configuration is another advantage which makes it a practical alternative to volumetric or holographic 3-D displays. So far, researchers have overcome the problems of low 3-D resolution,^2,3 limited depth of field, etc.^4–11 How to make the II 3-D display accepted by home users is an urgent issue. In the conventional II, the parameters of micro-lens arrays (MLAs) used in both pickup and display processes should be identical. In practical use, we cannot expect all II 3-D display systems to have MLAs with the same specifications, and we could not prepare various sets of elemental image arrays (EIAs) for different II 3-D display systems. To be an ideal 3-D television technique, we have to make the EIA compatible with different II 3-D display systems for a huge number of home users.

The theoretical analysis on scaling rule between the primary and secondary pictures taking was firstly carried out by Okoshi.¹² His analysis was based on the assumption that the elemental images (EIs) are not scaled when shifted with respect to the optical axes of the corresponding micro-lenses. Park et al. discussed the scaling effects of the optical parameters difference between the pickup and display processes.¹³ They showed that the lateral magnification is given by the micro-lens size ratio and the longitudinal magnification is given by the focal length ratio. Then, they proposed a ratio-conserving scaling to eliminate distortion of the 3-D image in which the EIAs are scaled using the focal length ratio.¹⁴ But these methods cannot cope with a change in the number of micro-lenses. The EIAs obtained in the pickup process are digitally analyzed and full 3-D information of the object are extracted, and then the extracted 3-D information is transmitted to display systems, whose numbers and parameters of micro-lenses are entirely different for the pickup process.¹⁵ However, stereo matching is needed to detect the disparity between two or more EIs and the quality of the generated 3-D image will be degraded. The scaling was also achieved by controlling the spatial ray sampling rate of EIA in the pickup process¹⁶ and a digital magnification method using interpolation theory was proposed to increase the spatial ray sampling.¹⁷ The intermediate-view reconstruction technique using the same interpolation theory was used to increase the number of EIs (Ref. 18). But the parameters of the micro-lens are not changed in these techniques. Smart pixel mapping algorithm is proposed by Martínez-Corral et al.¹⁹ to resolve the pseudoscopic problem of the reconstructed 3-D image, which is also resolved by Jung et al. recently.²⁰

In this paper, we propose an II without image distortion using MLAs with different specifications in the pickup and display processes. The deduced pixel mapping algorithm not only resolves the pseudoscopic problem, but also generates the EIA with different lens specifications, and the reconstructed 3-D images do not have any distortions.

2.

Principle of the Proposed II

Figure 1 shows the schematic of the proposed II which has different specifications of MLAs in the pickup and display processes. In the pickup process, as shown in Fig. 1(a), the MLA and the EIA are called MLA 1 and EIA 1, respectively. The pitch and the focal length of MLA 1 are $p_1$ and $f_1$, respectively, and MLA 1 contains $M_1\times N_1$ micro-lenses. EIA 1 is recorded on the rear focal plane of MLA 1 and the pitch of EIA 1 is the same with that of MLA 1. In the display process, as shown in Fig. 1(b), the MLA and the EIA are called MLA 2 and EIA 2, respectively. The pitch and the focal length of MLA 2 are $p_2$ and $f_2$, respectively, and MLA 2 contains $M_2\times N_2$ micro-lenses. EIA 2 generated by the pixel mapping algorithm has different specifications from EIA 1, and the 3-D image reconstructed by EIA 2 and MLA 2 maintains the original size and location of the 3-D object without any distortions.

Fig. 1

Schematic of the proposed II using MLAs with different specifications in (a) pickup and (b) display processes.

Figure 2 shows the schematic of the proposed pixel mapping algorithm which includes the virtual display process and the virtual pickup process. Since the pickup process and the virtual display process form the standard configuration of II, EIA 1 and MLA 1 reconstruct a 3-D image that maintains the original size and location of the 3-D object, but with the reversed depth. But the pseudoscopic problem can be resolved by the pixel mapping algorithm because the combination of the virtual display process and the virtual pickup process is a modified version of the two-step pickup process. In the virtual pickup process, MLA 2, which is the same as the one used in the display process in Fig. 1(b) and which has different specifications from MLA 1, picks up the depth-reversed 3-D image. The distance between MLA 1 and MLA 2 is $L$. EIA 2 that has different specifications from EIA 1 can be obtained on the rear focal plane of MLA 2, and EIA 2 and MLA 2 have the same pitch of $p_2$. The resolutions of EIs in EIA 1 and EIA 2 are both $r\times r$.

Fig. 2

Schematic of the proposed pixel mapping algorithm. (a) Virtual display process and (b) virtual pickup process.

In practicae, the virtual display and virtual pickup processes are carried out by mapping all the pixels in EIA 1 to EIA 2 through the following mathematical relationships. As shown in Fig. 2, in the $m$’th row and the $n$’th line EI of EIA 1, a pixel in the $i$’th row and the $j$’th line is denoted as $I_1{(m,n)}_{i,j}$. The rays emitted from the pixel $I_1{(m,n)}_{i,j}$ are refracted by the micro-lenses in MLA 1 and MLA 2, and then arrive at the $i$’th row and the $j$’th line pixel of the $m$’th row and the $n$’th line EI in EIA 2. The pixel in EIA 2 is denoted as $I_2{(m\prime ,n\prime )}_{i\prime ,j\prime }$. EIA 2 can be generated by using the following mathematical relationships:

The distance $L$ between MLA 1 and MLA 2 determines the depth of the reconstructed 3-D image. Assuming that in the pickup process, the distance between the 3-D object and MLA 1 is $l_a$, the depth of the reconstructed 3-D image in the display process is

When $l_a=L$, the 3-D image will be displayed on the MLA 2 plane, and when $l_a>L$ or $l_a<L$, the 3-D image will be displayed behind or in front of the MLA 2 plane.

When the light rays emitted from the pixel in the adjacent EI in EIA 1 are refracted by MLA 1 and MLA 2 and arrive at EIA 2, the crosstalk pixel is produced, as shown in Fig. 3. So the pitches and the focal lengths of MLA 1 and MLA 2 should satisfy Eq. (7) to avoid the crosstalk pixel

Fig. 3

Crosstalk pixel caused by adjacent EI in EIA 1.

The uninformed pixel will be caused when pixels in EIA 2 have no corresponding pixels in EIA 1, as shown in Fig. 4. So the numbers of EIs in EIA 1 and EIA 2, $M_1\times N_1$ and $M_2\times N_2$, should satisfy Eqs. (8) and (9) to avoid the uninformed pixels

Fig. 4

Comparisons between the numbers of EIs in EIAs 1 and 2.

3.

Experiments and Results

In this experiment, a 3-D scene which consists of three plane images with different depths is built up as shown in Fig. 5. A camera array is used to simulate MLA 1. The $Z$-axis shows the distances between the characters and the camera arrays, and they are 60, 100, and 150 mm, respectively. Three experiments using the proposed II, conventional II without scaling, and straightforward scaling II in Ref. 11 are carried out. The specifications of MLA 1 and MLA 2 are listed in Table 1 in which the parameters of MLA 1 and MLA 2 in the proposed II satisfy the relationships in Eqs. (7)–(9). Three EIAs 2 obtained using the three II methods are shown in Figs. 6(a), 6(b), and 6(c), respectively.

Fig. 5

Three-dimensional (3-D) scene in the experiments.

Table 1

Specifications of MLAs 1 and 2 in the experiments.

Fig. 6

EIAs 2 generated by the (a) proposed II, (b) conventional II without scaling, and (c) straightforward scaling II.

The depth-based computational II reconstruction is implemented to reveal explicitly the cross sections of the reconstructed 3-D images along the longitudinal direction, so that the longitudinal magnification of the 3-D image can be readily determined. Since the distance $L$ between MLA 1 and MLA 2 is 100 mm, the reconstructed undistorted 3-D images should be located at the depths of $-50$, 0, and 40 mm, respectively. As shown in Figs. 7(a) and 7(b), the 3-D images reconstructed by our proposed II have the same depths without the pseudoscopic problem as the ones reconstructed by the conventional II and their longitudinal magnifications are both 1. However, using the straightforward scaling II as shown in Fig. 7(c), the reconstructed 3-D images have the longitudinal magnification of $f_2/f_1=0.86$.

Fig. 7

3-D images at different depths reconstructed by the (a) proposed II, (b) conventional II without scaling, and (c) straightforward scaling II.

The optical II 3-D display experiments are carried out to verify the lateral magnifications of the reconstructed 3-D images. Since the total resolution of each EIA 2 is pretty high, a high-resolution color printer, EPSON STYLUS PHOTO 1390, is used to print three EIAs 2. Three MLAs 2 are used to match with the three printed EIAs 2 and three II 3-D pictures are obtained as shown in Fig. 8. A ruler is used to measure the lateral size of the reconstructed 3-D images. As shown in Figs. 8(a) and 8(b), the lateral sizes of the characters “II”s reconstructed by our proposed II and the conventional II without scaling are both 67.5 mm. However, using the straightforward scaling II as shown in Fig. 8(c), the lateral size of the reconstructed characters “II” is 43.5 mm. So the lateral magnifications of the proposed II are 1 and the lateral magnification of the straightforward scaling II is about $p_2/p_1=0.644$.

Fig. 8

3-D images reconstructed by the (a) proposed II, (b) conventional II with no scaling, and (c) straightforward scaling II.

4.

Conclusions

In this paper, we propose an II in which MLA 1 in the pickup process and MLA 2 in the display process have different specifications. The pixel mapping algorithm that functions as a converter to transmit the pixels from EIA 1 to EIA 2 not only resolves the pseudoscopic problem but also generates the EIA with different lens specifications. We also deduce the mathematical relationship between EIAs 1 and 2, and the relationships between the parameters of EIAs 1 and 2, and between MLAs 1 and 2. As long as the parameters of MLA 1 and MLA 2 satisfy the relationships in Eqs. (7)–(9), the parameters of MLA 2 and EIA 2 can be selected arbitrarily, hence different EIAs 2 can be generated from EIA 1 for different II 3-D display systems. The experimental results demonstrate that the reconstructed 3-D images in the proposed II maintain the original lateral and longitudinal sizes of the 3-D object without any scaling and distortion. The proposed II could be an ideal candidate for 3-D television broadcasting in the future.