1.

INTRODUCTION

Depth sensing technology has been intensively researched for applications in industry, medicine, computer science, entertainment, and other sectors. Structured light illumination (SLI) technology is a common approach among various methods for determining the depth of the object’s surface. In an SLI system, a projector projects pre-encoded fringe patterns onto the object, the camera synchronously captures the deformed pattern, and the phase is calculated from those patterns. After calibration, the depth of the surface can be calculated according to the principle of triangulation [1].

In this paper, a simple and practical square-wave phase-shift encoding strategy and the corresponding decoding method are proposed. Compared with the traditional sinusoidal phase-shift encoding strategy, the proposed method is immune to the effects of gamma distortion. Moreover, the invalid phase is removed by the epipolar constraints. Experiments are conducted to validate the proposed method. The proposed approach produces a more accurate point cloud than traditional SLI according to the results.

2.

RELATED WORK

The projection of a single pattern or a set of patterns onto the surface of the scanned object, which is then synchronously captured by a single camera or a series of cameras, is the basic process of a structured light system. The patterns have been generated in such a way that codewords are assigned to a group of pixels. There is a direct mapping from the codewords to the corresponding coordinates of the pixel in the pattern since each coded pixel has its own codeword [2]. Timemultiplexing, spatial neighborhood codification, and direct codification are the three encoding strategies used in projection techniques [2]. The spatial neighborhood codification tends to employ a single pattern for reconstruction. The codeword that labels a point of the pattern is obtained from a neighborhood of the points around it. In some spatial encoding strategy, such as pseudo-noise sequence [3], M-array [4], color encoding [5], grid encoding [6] et al, a single pattern is generated and projected, whose advantage is that it requires less time for projection and is much easier to achieve real-time 3D reconstruction. However, the decoding stage may become difficult because the spatial neighborhood cannot always be identified, which causes reconstruction errors.

The temporal encoding strategy is one of the most often employed methods in structured light systems. A series of patterns are projected onto the surface of the scanned object in this case. This encoding scheme is capable of achieving higher measurement precision than the spatial neighborhood encoding strategy. In these encoding strategies, the most common schemes include phase shift [7], grayscale [8], and so on. Because multiple patterns are needed to project, it is not suitable for dynamic scenes.

3.

PROPOSED METHOD

Based on the triangulation between the camera and the projector, SLI is a widely used, high-precision, and active measurement method. Figure 1 shows the basic process of SLI. In this paper, we propose a square-wave encoding and decoding scheme. Moreover, with the epipolar constants, the invalid phase can be removed and the accuracy of the point cloud is improved.

Figure 1.

Schematic of SLI.

3.1

Classical structured light illumination

The classical phase-shift SLI usually adopts the encoding scheme in which the intensity changes sinusoidally along the horizontal or vertical direction. The coding pattern can be described by the following equation

where is the pixel intensity at the coordinate (x^p,y^p), A^p and B^p are two constants, the integer f is the spatial frequency of the encoded pattern, and H is the height of projector resolution, n and N are the phase shift index and the total number of phase shift steps, respectively. After the patterns are projected onto the measured object, the camera captures the deformed pattern synchronously, which can be expressed as

where is the brightness of the nth pattern at the camera coordinate (x^c, y^c), and A^c, B^c and ϕ are the average brightness, brightness modulation and phase, respectively. To simplify, the coordinates (x^p, y^p) and (x^c, y^c) are omitted from the equation below. The phase ϕ can be derived from

With phase unwrapping operation, the absolute phase can be obtained.

However, the classical sinusoidal encoding scheme is sensitive to gamma nonlinearity, noise, and multiple patterns are required to be projected, which takes a much longer time. Liu et al. [9] deduced the gamma distortion model in detail, and quantitatively described the relationship between gamma and phase error. The gamma distortion model is based on a one-parameter gamma function, which is described as

where represents the gamma distorted pattern captured by the camera, and γ is the distortion parameter. The phase error can be derived as

where G_m is

As can be seen from the above equation, the phase error is related to γ. Therefore, the classical sinusoidal encoding scheme suffers from gamma distortion. In the next section, a gamma-free encoding scheme is proposed.

3.2

The proposed bidirectional pattern encoding scheme

In this section, a square-wave encoding strategy for bidirectional scanning SLI is proposed for 3D reconstruction, which overcomes the shortcomings of the traditional sinusoidal encoding scheme. The projected pattern along the vertical direction can be described as

where is the light intensity of bright area in the square wave pattern, rect (•) is a function that generates a square wave, which is shown in the following equation

Similarly, the projected pattern along the horizontal direction is expressed as

where W is the width of projector resolution. Figure 2 illustrates the difference between the square-wave encoding scheme and the sinusoidal encoding strategy. As we can see, the square-wave pattern only needs two levels of intensity.

Figure 2.

The comparison of the sinusoidal coding scheme and the square wave coding scheme. , A^p = 127.5, B^p = 127.5, f = 1.

After projecting the square wave pattern, the pattern captured by the camera can be described as

where is the light intensity of a captured pattern. For the square wave pattern, similarly, the image distorted by gamma nonlinearity can be expressed as

Correspondingly, the phase distorted by gamma nonlinearity is derived as

With equation (11), the equation (12) can be simplified to

As we can see from the above equation, the phase ϕ has nothing to do with the gamma distortion coefficient γ. In conclusion, it is not affected by gamma distortion. Then, the original range of the phase ϕ, which is obtained with the equation (13), is[–π, π]. For convenience, we map its range to[0,2π]. In Figure 3, we visualize one column of the wrapped phase.

Figure 3.

One column of the wrapped phase ϕ, f = 4.

In phase-shift structured light illumination, high frequency is more effective for suppressing noise, nonlinearity, and other errors, so the high-frequency phase is generally used to reconstruct 3D point clouds. The phase ϕ calculated by equation (3) needs to be unwrapped according to its fringe order. For the wrapped phase above, its fringe order can be obtained by the following equation

where ϕ_l is the phase of base frequency, round(•) is the numeric rounding operation, and k is the fringe order. For multi-frequency phase-shift structured light technology, when the base frequency is used to unwrap the phase of a high frequency, then the result is used as the new base frequency phase to unwrap the phase of a higher frequency until the highest frequency is obtained. After the fringe order is computed, the absolute phase can be calculated by the following equation

We obtain the absolute phase at a sub-pixel level by fitting. The absolute phase is shown in Figure 4.

Figure 4.

One column of the absolute phase Φ.

3.3

Removal of invalid phase

The calibration matrices of the camera and projector are expressed as

To obtain a more accurate point cloud, we remove the invalid phase resulting from several factors. In the Liu’s research [10], the epipolar line is derived into a much simpler description:

where and can be obtained from the following equation

where the is the vector from the camera center to a 3D point in a world. It is given by

where

The is the optical center of the camera and computed by

The description of epipolar line in equation (17) does not require a fundamental matrix between the camera and the projector and can be transformed into

where A, B and C are

According to the epipolar constraint, the point (x^c, y^c) corresponding to the point (x^p, y^p) on the projection plane should be located on the epipolar line, that is, in an ideal situation, the distance from the point to the epipolar line should be 0. The distance from the point to the epipolar line is

When errors are caused by various reasons such as noise, lens distortion, and image saturation in the imaging process, the above relationship is not satisfied. When the distance between the phase point and the corresponding epipolar line exceeds the threshold value, it indicates that the point is too impacted by external and internal factors to reconstruct an accurate point cloud. As a result, the region including these points should be excluded. The method is simple and practical. Moreover, it also works for the classical sinusoidal encoding SLI system.

To obtain the point cloud, we need to calibrate the structured light system to get the projector and camera parameters. There are various calibration techniques for calibrating the structured light system. In this paper, we adopt the least-square solution method, which is easier to implement and is close to other methods in accuracy [11]. With the calibration matrices of the camera and projector, the point cloud of the scanned surface is computed by

where C and D are

4.

EXPERIMENTAL RESULTS

To validate the effectiveness of the proposed encoding scheme, we conducted two experiments in this section. In the first experiment, the invalid phase area is determined and labelled when scanning a sculpture. In the second experiment, a vertical plane is scanned, which consists of black and white grids to simulate the high contrast region. The invalid phase area is determined and removed. Then, the measurement errors of the plane are calculated for the proposed method and the classical method. The structured light system consists of a Casio XJ-M140 projector, an AVT Prosilica GC650C camera, and a computer. The resolutions of the projector and camera are 800×600 and 640×480 respectively, and the camera works in grayscale mode. Before conducting the experiment, we first calibrate the projector and camera as described to obtain the calibration parameter matrices of the camera and projector. The system suffers from the gamma distortion and we don’t remove it.

First, we generate a set of square wave patterns for projection, the frequencies are 1, 8, 64, 128, and the total number of phase shifts for each group is 8. We first scan the sculpture and compute the phase (x^p, y^p) according to the captured patterns. The result is shown in Figures 5 and 6.

Then, we compute the and according to the equation (9) and equation (10) in the Ref. [10]. For a SLI system, the value of are constants and the value of are functions of (x^c, y^c), which can be organized as look-up table. Therefore, these values only need to be computed once until the location relationship between the camera and the projector changes. After the above computation, we can obtain the coefficients of the epipolar line at (x^p, y^p). In Figure 7, The black area is the invalid phase are removed by the method in section 3.3, and the white area is the retained area that can be effectively reconstructed.

Figure 5.

x^p of a scanned sculpture.

Figure 6.

y^p of a scanned sculpture.

Figure 7.

Removal of invalid phase when scanning a sculpture.

In the second experiment, in order to further quantify the accuracy improvement of the proposed method for 3D reconstruction, a vertical plane is reconstructed with the proposed and classical methods. Moreover, the reconstruction errors are shown in Table 1. Figure 8 demonstrates a vertical plane with a checkerboard image in the central part. The black and white area of the plane simulates different adjacent high contrast regions. The proposed method can effectively detect the regions with large phase errors in the phase-shifted structured light. Figure 9 shows the reconstruction error of the vertical plane after removing the invalid point area by the proposed method. Meanwhile, we calculate the mean reconstruction error, maximum reconstruction error and RMS of the reconstruction error with the two methods, and the results are shown in Table 1.

Figure 8.

A vertical plane with a checkerboard in the central part.

Figure 9.

The reconstruction error map.

Table 1.

The error comparison.

From Table 1, we can see that the proposed method can effectively remove the points with large errors in the 3D reconstruction, and then improve the accuracy of the reconstruction in the gamma distorted systems. Moreover, the proposed method is simple and practical without complicated calculations and additional hardware. With the proposed method of removing invalid phase area, we can easily locate the area and reconstructs much more accurate point clouds.

5.

CONCLUSION

A square wave encoding strategy, which is gamma-free, is proposed for an SLI system. A framework including projection pattern design, practical decoding strategy, and removal of invalid phase is developed. Besides, gamma analysis for the square wave is also presented, which proves that it is not affected by gamma distortion. Moreover, a simple and practical method, which can remove the invalid phase and also works for the classical sinusoidal encoding SLI system, is proposed. Compared with the sinusoidal phase-shifting method, the proposed method has a much better performance in the low SNR scenes and is easy to implement. With the proposed method, we can obtain a more accurate point cloud.