Special Section on Active Electro-Optical Sensing: Phenomenology, Technology, and Applications

Demonstration on range imaging of 256×256 pixels and 30 frames per second using short wavelength infrared pulsed time-of-flight laser sensor with linear array receiver

[+] Author Affiliations
Shumpei Kameyama, Akihito Hirai, Masaharu Imaki, Nobuki Kotake, Hidenobu Tsuji, Yuichi Nishino, Yusuke Ito, Mikio Takabayashi, Yasuhisa Tamagawa, Yoshihito Hirano

Mitsubishi Electric Corporation, Information Technology R&D Center, 5-1-1 Ofuna, Kamakura, Kanagawa 247-8501, Japan

Masaharu Nakaji, Eitaro Ishimura

Mitsubishi Electric Corporation, High Frequency and Optical Device Works, 4-1 Mizuhara, Itami, Hyogo 664-0005, Japan

Opt. Eng. 56(3), 031214 (Nov 13, 2016). doi:10.1117/1.OE.56.3.031214
History: Received June 30, 2016; Accepted October 20, 2016
Text Size: A A A

Open Access Open Access

Abstract.  We demonstrated the range imaging with high resolution of 256×256  pixels and high frame rate of 30 frames per second (fps) using a short wavelength infrared pulsed time-of-flight laser sensor, which is suitable for long range imaging. We additionally demonstrated the long range imaging of more than 1 km and wide field of view imaging of 12  deg×4  deg, 768×256  pixels, and 10 fps. For these demonstrations, we developed the linear array devices of the aluminum indium arsenide avalanche photodiode array and silicon germanium bipolar complementary metal oxide semiconductor read-out integrated circuit array. We also deployed the flattop beam illumination optics with the beam division and recombination method and realized efficient line shape illumination.

The time-of-flight (TOF) imaging laser sensor is an attractive tool for many applications, including robotics, terrain visualization, augmented vision, reconnaissance, pedestal detection, and so on. For the laser TOF measurement, there are basically two methods, pulsed and continuous wave modulation. It is generally said that the former is suitable for long-distance ranging, and the latter is better for the precise ranging in short-distance ranging.1 Therefore, for the applications that require an imaging range of more than a few tens of meters, the pulsed method is considered to be better.

Lamberts2 first presented the pulsed TOF imaging laser sensor with a 1.06  μm Nd:YAG laser and a two-dimensional (2-D) scanner. The laser sensor with a 2-D scanner is widely used now especially for the survey application, but it is not useful for real-time imaging since it measures the TOF of only one point on the scene and takes a substantial amount of time to scan whole imaging areas. This is conceptually true for the cases of long range imaging that need a large scanning mirror. For example, we reported the imaging results using this type of laser sensor in the past literature,3 and the frame rate was just 2 Hz for the imaging of 128×128  pixels.

The TOF laser sensor with an array (2-D or linear) receiver is a solution for the real-time imaging, since this can measure the TOF of multiple points on the object instantly. Stettner and Bailey4 shows this type of TOF laser sensor with a 2-D array receiver, using a 1.06  μm Nd:YAG laser and silicon avalanche photodiode (APD) array. This concept has been deployed by many groups using the wavelength region of 0.53,5 1.06,6 and 1.57  μm.79 The 2-D array receiver can realize instant range imaging by flashing the scene using one transmitted pulse without any scanning. However, there are the following issues in 2-D array type. First, a very high transmitted pulse energy is necessary to get enough receiving power for each element. For example, in the case of 256×256  pixels array, the required pulse energy is 65,536 times larger compared with the one in the case of a single pixel [for the same measurable range, in cases of the same transmitted average power, same receiving aperture size, same receiving sensitivity, and same field of view (FOV)]. Second, the yield rate of the array devices tends to be low because of its too many elements in the 2-D array device. Third, the performance of the read-out integrated circuit (ROIC) is restricted because the size of each pixel must be small (e.g., 100  μm×100  μm10). Additionally, the scanning is consequently needed even for the 2-D array type in many cases of wide FOV since the FOV of the 2-D array receiver is limited by the size of the 2-D array and the F number of the receiving optics.

On the other hand, the TOF laser sensor with a linear array receiver has merits on these issues. The required pulse energy can be lower compared with the one for the 2-D array type. For example, in the case of 256×256  pixels imaging, the 256 pixels linear array is used, and the required pulse energy can be 1/256 compared with the one for the 2-D array case. The required average transmitted power can be the same if a high repetition rate laser is used for the linear array type. The efficiency of the laser transmitter is generally high for the low pulse energy and high repetition rate laser. It is much easier for the linear array type to get the array receiver of which all pixels in the array work well. The performance of ROIC can be better for the linear array type since the area of ROIC can be larger. This larger area enables the installation of larger numbers of components (transistors, capacitors, and so on) in a chip to realize the better noise characteristic and larger number of functions. Further, the sensor FOV can be flexible (in many cases, wider) easily by changing the scanning angle width. Eberle et al.11 reported this type of laser sensor with 384 pixels linear array mercury cadmium telluride (MCT)-based APD, and showed preliminary imaging results at the distance of 270 m using the wavelength of 1.57  μm. However, the MCT-APD does not work well at room temperature although its specification is very attractive for the high receiving gain and multifunctionality (i.e., both working for active and passive imaging). The operating temperature of the MCT-APD is not denoted in Ref. 11, but it is reasonable to consider that the temperature is around 70 K by referring to another laser sensor using MCT-APD.9 In addition, the maximum read-out rate is 4 kHz (corresponding to 1.54  MHz/pixel), and this restricts the sensor frame rate to 15 Hz (below the video rate) for 384×256  pixels imaging. The detail of the ROIC and the reason for this restriction cannot be known in the reference. Further, the transmitting optics was designed to form the flattop-like illumination only at the far-field and is not suitable for the flattop illumination for wide distance range which is an important function for imaging laser sensors.

To overcome these issues, we have developed (i) 256 pixels linear array APD with aluminum indium arsenide for the multiplication layer (AlInAs-APD) for short wavelength infrared (SWIR) band which works at room temperature, (ii) silicon germanium bipolar complementary metal oxide semiconductor (SiGe-BiCMOS) ROIC array with the read-out rate of 2.56  MHz/pixel. The AlInAs-APD also has the feature in its low excess noise factor compared with the APD with indium phosphide multiplication layer (InP-APD), which is used in the past laser sensor. The ROIC is an analogue type and has features in multifunctionality in addition to the high read-out rate. We have also deployed the flattop beam illumination optics with the beam division and recombination method12 and realized efficient line shape illumination which is necessary for the linear array type of laser sensor. The concept of our development is shown in Fig. 1. By combining these developments, we demonstrate the range imaging with a high frame rate of 30 frames per second (fps) and high resolution of 256×256  pixels. We additionally demonstrate the long range imaging of more than 1 km. To the best of our knowledge, the above-mentioned imaging specification of our sensor is the highest for the range imaging pulsed laser sensor which is suitable for long range imaging. For example, the number of pixels in Refs. 7 and 8 (2-D array type) is limited to 32×32 and 128×128. The number of pixels in Ref. 9 (2-D array type) is large and 320×256, but the imaging range is limited to 70 m. The imaging range in Ref. 11 (linear array type) is relatively long (270 m) with the large number of pixels of 384×256, but the frame rate is limited to 15 Hz as noted above. In the following sections of this paper, we explain the key devices that we have developed, and show some imaging results using the prototypes using the SWIR band of 1.06 and 1.55  μm.

Graphic Jump Location
Fig. 1
F1 :

Concept of the pulsed TOF imaging laser sensor with the linear array receiver.

Avalanche Photodiode Array

We developed the APD array by utilizing our guardring-free planer AlInAs-APD technology.13 The conceptual design, theoretical performance, and its comparison with the evaluated results are shown in the past literature.13 To our knowledge, there has been no literature on laser sensors using AlInAs-APD. This is a linear mode APD and can be operated well in the required temperature of 300 K. In addition, this has a feature of a low excess noise factor. In Ref. 13, it is shown that the excess noise factor of this APD is 2.9 at the multiplication factor of 10. The InP-APD is widely used in the laser sensors (e.g., in Ref. 7). As denoted in Ref. 13, the excess noise factor of InP-APD is larger, and the value is 5.5 for the same multiplication condition. This means that the multiplication factor can be set at higher value for AlInAs-APD than the value for the other one under the same noise level. This is an advantage in using AlInAs-APD for laser sensors, and can be achieved with the careful tuning of the bias voltage. Figure 2 shows the structure of the developed 256 pixels AlInAs-APD array. All layers are grown on a n-type InP substrate by the molecular beam epitaxy. An AlInAs multiplication layer, a p-field control layer, an InGaAs absorbing layer, and a p-window layer are grown on n-type distributed Bragg reflector (DBR). The top surface is coated with a passivating film, which acts as antireflection coating. The dimension of the receiving area, which is formed on the top surface, is 50  μm×35  μm. The element pitch is 50  μm. The current–voltage characteristic with a multiplication curve of APD is shown in Fig. 3. The responsivity dependence on wavelength is shown in Fig. 4. The photocurrent was measured by inputting the light into a pixel with a power of 34  μA and changing the bias voltage. The multiplication factor was obtained using the measured photocurrent and responsivity of the APD. The responsivity can be known from the flat photocurrent region in Fig. 3, by assuming that the photocurrent of this region corresponds to the multiplication factor of 1. In Fig. 3, it is shown that the breakdown voltage Vbr at a reverse current of 10  μA is about 40 V. The dark current Id at 90% of Vbr is as low as about 30 nA at room temperature. The maximum multiplication factor is over 100 and the factor at 90% of Vbr is about 10. The maximum photocurrent for the linear input–output operation is 0.1 mA at the multiplication factor of 10 and this is the index of the dynamic range. In Fig. 4, the responsivity at a wavelength of 1.55-μm band is 1.07  A/W, which corresponds to a quantum efficiency of 85%. Although the quantum efficiency is kept within the SWIR band (from 1 to 1.7-μm band), the DBR enhances the responsivity at a wavelength of 1.55-μm band by 15%. The capacitance at 90% of Vbr is 0.3 pF at a frequency of 1 MHz. The 3  dB cut-off frequency is more than 1 GHz at a gain of 10.

Graphic Jump Location
Fig. 2
F2 :

AlInAs-APD array structure.

Graphic Jump Location
Fig. 3
F3 :

Current–voltage characteristic with a multiplication curve of the APD.

Graphic Jump Location
Fig. 4
F4 :

Wavelength dependence of the responsivity.

Read-Out Integrated Circuit Array

The ROIC array is fabricated in 0.18  μm SiGe-BiCMOS process. Each element of the ROIC array includes the transimpedance amplifiers (TIAs), the peak intensity detector, and the TOF detector which is the time-to-amplitude converter (TAC) circuit. The transimpedance gain and receiving bandwidth of the TIA are about 15  kΩ and 100 MHz, and the equivalent input noise current density is about 2  pA/Hz0.5. These values agree with the theoretical ones, which were designed by considering the transimpedance resistance and noise characteristics of the BiCMOS transistor. The transimpedance gain and the bandwidth were obtained by inputting the sinusoidal current signal of different frequencies to the TIA with known current level and measuring the output voltage. The equivalent input noise current density was obtained by the measured noise spectrum and the transimpedance gain. Figures 5, 6, and 7 show the configuration of the element of the array, the timing chart for the operation, and the photograph of the ROIC array chip, respectively. The chip size is 6.7  mm×3  mm. A chip of ROIC array includes the multiplexers for read-out. The number of element of the array is 64. The read-out rate is 2.56  MHz/pixel and this performance is proved in the imaging experiment shown later in Sec. 4.1. Owing to this read-out rate, our TOF laser sensor can realize the frame rate of more than 30 Hz (potentially, 39 Hz) for 256×256  pixels. The reason for this realization of such high read-out rate is that this ROIC obtains the range and intensity data by analog method without any digital processing.

Graphic Jump Location
Fig. 5
F5 :

Configuration of the element of the ROIC.

Graphic Jump Location
Fig. 6
F6 :

Timing chart of operation of the ROIC.

Graphic Jump Location
Fig. 7
F7 :

Photograph of ROIC chip.

The operation of the ROIC is the following. At first, the ROIC is initialized by a reset signal. The gate for range measurement (GATE) signal turns on the peak detector and the negative feedback loop formed by an operational amplifier (Op-Amp), an analog MOS switch, and a capacitor. Then, after the negative feedback shifts to the stable state, the measurement starts by transmitted laser light and switching this GATE signal into high level. The maximum peak voltage can be detected while the GATE signal is high level. The output signal of the Op-Amp is used for generating sample and hold trigger (S/H TRG) with the comparator and the threshold voltage. During operating of the peak detector, the TAC operates in parallel. The voltage stored at the capacitor in TAC is charged by the current source after the GATE signal is at high level. At the moment of the detection of peak voltage, the S/H TRG switches to low level. The S/H circuit in the peak detector and the S/H circuit in TAC hold the peak voltage and the ramp voltage, respectively. Since the ROIC is an analogue type, the timing resolution and jitter are not restricted by the ROIC itself and are restricted by the characteristic of the external control circuit to input the timing signal to this ROIC.

For ideal signal detection, the threshold voltage (i.e., the negative input voltage) of the comparator should be slightly higher than the offset voltage of the Op-Amp (i.e., the positive input voltage of the comparator). In this case, the minimum signal detection level can be low. If the offset voltages are the same between the elements in the array, the ideal signal detection can be easily realized with a common threshold voltage for all elements. However, in actuality, the offset voltage has unavoidable dispersion between elements. Figure 8 shows this dispersion of the offset voltage. This offset voltage was measured by changing the setting of the threshold voltage and finding out the boundary condition for the peak detector to work. If the offset voltage is much higher than the threshold, S/H TRG is not output in the cases of small signals. This means that small signals cannot be detected. If the offset voltage is lower than the threshold, SH TRG continues to be output. In this case, the S/H circuit in the TAC does not hold the ramp voltage and consequently, the ranging function does not work. To resolve this issue, we have added the function of setting the optimum threshold voltage for S/H TRG. That voltage is configured by a digital-to-analogue converter (DAC). The DAC accesses the random access memory which is integrated in each ROIC element. Using this function, we can set the threshold slightly higher than the offset voltage of the Op-Amp for all elements and the ideal signal detection can be realized. The ROIC has another additional function in switching the measurement range width. That can be realized by controlling the current in TAC. The current is controlled by the external input of digital value as shown in Fig. 9. Figure 9(a) shows the designed measurement range width versus the digital value. Figure 9(b) shows the measured results of input time (corresponding to TOF) versus output voltage of the ramp, respectively. The ramp voltage characteristic was measured by changing the input timing of a pulse signal. The range width was obtained by measuring the time width from the start and end of the ramp. In Fig. 9(b), the two lines correspond to the ramp voltage in case of TAC=9 and 53, respectively. The range widths, which are obtained from Fig. 9(b), are 450 and 75 m (corresponding to 3000 and 500 ns) for TAC=9 and 53. These values agree with the designed ones shown in Fig. 9(a). In the developed ROIC, we can set the measurement range width depending on the flexible user’s request in time. If the ranging width is set to be short, the gradient of the time-to-amplitude in the TAC becomes steep and it contributes to the high precision ranging. On the other hand, when the ranging width is set to be long, it is suitable for long-range imaging. These additional functions (setting the optimum threshold voltage for S/H TRG and switching the measurement range width) are realized owing to the feature of the larger area of linear array concept which was described in Sec. 1.

Graphic Jump Location
Fig. 8
F8 :

Offset voltage of OP-Amp and threshold voltage of comparator.

Graphic Jump Location
Fig. 9
F9 :

Variable characteristics of the TAC by setting the digital value [(a) designed characteristics of the measurement range width in each digital value and (b) measured ramp voltage configured by digital value].

Array Receiver Package

The linear array receiver consists of the APD array and the ROIC array assembled in one package. The photograph of the package is shown in Fig. 10. The four ROIC chips (each has 64 elements) are located at both sides of the 256 pixels linear APD array. The ROICs and APD are wired to each other. The package size is 42  mm×42  mm.

Graphic Jump Location
Fig. 10
F10 :

Photograph of the linear array receiver package.

Transmitting Optics

In this sensor, it is important to illuminate the transmitted laser light uniformly on a scene along with the line shape FOV. For this purpose, we have developed also the flattop beam illumination optics with the beam division and recombination method12 and realized efficient line shape illumination which is necessary for the linear array type of laser sensor. Figure 11 shows the schematic of this optics. Note that this figure shows one-dimensional illumination and does not show the discussion about the direction perpendicular to the sheet. In this optics, a Gaussian beam is diverged by a concave lens. The diverged beam is divided and refracted by a prism. After passing through the prism, the divided Gaussian beams are crossed and overlapped to each other. As the results of this crossing and overlapping, intensity distribution within a combined beam becomes uniform. The feature of this optics is in keeping the flattop illumination for wide distance range. The beam shape for the perpendicular direction is adjusted to make the divergence angle smaller than the FOV of an array element. Here, this is the collimated Gaussian shape. Figure 12 shows the evaluated result of the developed optics with the design of the divergence angle of about 4 deg. In Fig. 12, the optical coating was optimized for the wavelength of 1.06  μm and the result was obtained using this wavelength. It is shown that uniform illumination is realized. The evaluation was done at the short distance of 2 m but it is shown in the past literature12 that the flat illumination can be kept for long range. Standard deviation of illuminated intensity within a designed illuminated area is about 4% except for the edge part, where distinct ripples appear because of a diffraction effect. This effect is not so large since the divided beams are the diverged ones.

Graphic Jump Location
Fig. 11
F11 :

Schematic for line direction of uniform illumination optics using beam division and recombination method.

Graphic Jump Location
Fig. 12
F12 :

Evaluated result of the distribution of illuminated intensity.

We developed the prototype using the key devices. The sensor configuration is shown in Fig. 13. The sensor has the array receiver, in which the APD array and the ROIC array are assembled in one package as shown in Sec. 2.3, the transmitting optics that can illuminate uniformly on an object, and the commercially available pulsed laser. We developed two prototypes of which one uses the 1.55-μm erbium-doped fiber laser and the other uses 1.06-μm diode-pumped Nd:YAG laser. The pulse energies of the former and the latter lasers are 0.2 and 1 mJ, respectively. Basically, the pulse energy is the only point that is different between the two prototypes, and the pulse widths and the pulse repetition frequencies are 8 ns and 10 kHz for both lasers. The purpose of using the former laser is the demonstration of the imaging using the eye-safe wavelength and the purpose for the latter is the demonstration of high performance (i.e., high resolution of 256×256  pixels, high frame rate of 30 fps, and long range) imaging using the higher energy laser. The aperture diameter of the receiving optics is 75 mm. The FOV of the array receiver is 4 deg for the vertical direction. These optical parameters are the same for both wavelengths, but the optical coatings are the customized ones for each wavelength. A horizontal dimensional scanning mirror is a galvano type. The personal computer (PC)-based signal processor includes a digital input–output (DIO) board, a DAC, and an analogue-to-digital converter (ADC). The transmitted laser light is transmitted uniformly by the uniform illumination optics. The received light is detected by the array receiver through the receiving optics. The intensity and TOF of a received light are converted to analogue voltages in each channel of the array receiver. These voltages are acquired by the ADC. The DIO board outputs control signals to the array receiver and a laser trigger to the laser head. This board also outputs a trigger and a clock signals to the DAC and the ADC. The DAC controls the scanning mirror through the scanner driver. The signal processor outputs an intensity image, a range image, and a three-dimensional (3-D) image by combining with angle information for each pixel. The vertical angle information of each pixel can be known from the linear FOV. The information for horizontal direction can be known from the scanner angle at the timing of each pulse shot. The internal view of the main container of the prototype is shown in Fig. 14. The photograph is shown only for 1.55  μmtype, but the basic configuration is the same for both wavelengths (i.e., 1.55 and 1.06  μm).

Graphic Jump Location
Fig. 13
F13 :

Configuration of the prototype.

Graphic Jump Location
Fig. 14
F14 :

Internal view of the main container.

Ranging Precision and Accuracy

Here, we show the evaluation result of the ranging precision and accuracy. The used prototype is the 1.55-μm wavelength type. The precision and accuracy are the same for both wavelength types, since the pulse width and the performance of the ROIC are the same. The range imaging result used in this evaluation is shown in Fig. 15. The setting of the range width of ROIC was roughly adjusted to the maximum range point in the imaging scene. This is same for all imaging results in the following section of this paper. The static scene was selected to evaluate the ranging precision and accuracy. This figure has 768×256  pixels with the FOV of 12  deg×4  deg, but these are not important points in this evaluation. This figure also shows the proof of the read-out rate of 2.56  MHz/pixel which was described in Sec. 2.2, since this figure shows that the read-out of 256 pixels is performed well in the pulse repetition rate of 10 kHz. We evaluated the ranging precision as the standard deviation of ranging results at the specific point (denoted in the figure) for successive 100 frames. The histogram of the ranging result is shown in Fig. 16. The standard deviation is 7 cm and this is the ranging precision of this system. This precision is dominated by the noise of the receiver. To obtain the range accuracy, we investigated the ranging results for the multiple pixels corresponding to a same building wall (denoted in this figure). These ranging results should be almost the same (<5  cm) by considering the measurement condition. For the evaluation of the accuracy, the ranging results of each pixel are averaged for successive 100 frames to reduce the instantaneous deviation. The histogram of the averaged result is denoted in Fig. 18. The standard deviation is 20 cm. Since the range of the system is calibrated at the mean value in the histogram of Fig. 17 using the ground truth, the standard deviation of 20 cm can be assumed as the ranging accuracy of the system. This accuracy is mainly caused by the calibration error concerning with the time-to-amplitude gradient of the TAC in ROC. These performances on the precision and accuracy are almost the same as the laser sensor shown in Ref. 14.

Graphic Jump Location
Fig. 15
F15 :

Range imaging result used in the evaluation of ranging precision and accuracy.

Graphic Jump Location
Fig. 16
F16 :

Histogram of the ranging results for the specific pixel.

Graphic Jump Location
Fig. 17
F17 :

Histogram of the ranging results corresponding to the same building wall.

Graphic Jump Location
Fig. 18
F18 :

Real-time imaging of a man running (256×256  pixels, 30 fps, intensity image).

Real-Time Imaging

Here, we demonstrate imaging experiments using the prototype model. The wavelength used here is 1.06  μm. The online frame rate is 30 Hz for imaging of 256×256  pixels. The FOV is 4  deg×4  deg, respectively. The condition of 10 kHz pulse repetition frequency and 256×256  pixels may seem to conflict with the frame rate of 30 Hz. The time region, corresponding to both ends of the horizontal scanning, is not utilized for the imaging and is consumed for the signal processing. This limits the current frame rate of 30 Hz, but this can be improved depending on the computing power of the signal processor. Figures 18, 19, and 20 show the real-time intensity, range, and 3-D imaging results of a man running, respectively. The time difference of images in these figures is 0.4 s. This corresponds to 2.5 Hz and not 30 Hz, but the figure clearly shows a running man without any image flow owing to the 30 fps frame rate. In Fig. 20, the ground data are removed for seeing the man clearly. The range of the scene is about 150 m. In Fig. 20, the color of the point cloud indicates the height.

Graphic Jump Location
Fig. 19
F19 :

Real-time imaging of a man running (256×256  pixels, 30 fps, range image).

Graphic Jump Location
Fig. 20
F20 :

3-D imaging acquired in the same experiment corresponding to Figs. 14 and 15.

To evaluate the quality of the obtained range and 3-D image, we tested the simple method of an object detection algorithm with intensity and 3-D data. The algorithm flow is shown in Fig. 21. The process is explained as follows. At first, the range and intensity data are obtained. Using the range data and the information of the angle, we reconstruct the 3-D data. In the next step, data screening is performed. In this process, using the threshold of intensity data and the data in 3-D coordinate, the data that have low intensity or that have the 3-D data standing outside the scope of observation area are screened. Then, we extract the pixels having the height within the object size and intensity that are higher than the threshold we defined. Moreover, we perform the clustering of the pixels with the method of the density-based spatial clustering of applications with noise15 which is clustering within the constant distance in Euclidean space. Finally, the cluster data are filtered by the cuboid whose size is defined by the real object size. This detection algorithm needs the range precision to divide between the object and the other clutter objects since this clustering and filtering method is directly linked to the range precision. The detection results for which the object is the human are shown in Fig. 22. The rectangles on the intensity image that correspond to Fig. 18 indicate the result of the object detection. Although Figs. 1820 were obtained in real time, Fig. 22 is the postprocessed result because of the current computation power of the PC. These figures show that the object can be detected even in the scene with some clutter.

Graphic Jump Location
Fig. 21
F21 :

Algorithm flow for object detection.

Graphic Jump Location
Fig. 22
F22 :

Intensity images with detected result (red rectangle indicated the result of detection).

Long-Range Imaging

Figure 23 shows a long-range imaging of a landscape obtained by the same prototype as Sec. 4.2. Although the quality of the imaging is worse than that of Figs. 18 and 19 because of the lower signal-to-noise ratio, it is shown that the imaging was basically performed successfully even at a long range of more than 1 km. The horizontal streaks in Fig. 23 are caused by the dispersion of the minimum detectable signal level (note that each vertical pixel corresponds to each pixel of the linear array receiver). The ROIC, which was used in this imaging experiment, did not have the additional function of setting the optimum threshold voltage for S/H TRG which is denoted in Sec. 2.2. This is the reason of the above-mentioned dispersion of the minimum detectable signal level. Note that, for the all other imaging results shown in this paper, we used the ROIC with this additional function.

Graphic Jump Location
Fig. 23
F23 :

Long range imaging results [(a) intensity image and (b) range image].

The validity of this imaging range is not studied in detail here, but basically, it can be confirmed to be reasonable by considering the laser radar equation16 and the sensor parameters denoted in Secs. 2 and 3.

Wide Field of View Imaging

The wide FOV imaging is the advantage of the linear array type compared with the 2-D array type as described in Sec. 1. We demonstrate this advantage by changing the scanning width of the galvano scanner. The used wavelength is 1.55  μm for this experiment. Figures 24 and 25 show the intensity and range imaging result with 10 fps and 768×256  pixels. The FOV is 12  deg×4  deg, respectively. It is shown that changing FOV worked well.

Graphic Jump Location
Fig. 24
F24 :

Real-time imaging of a man running (768×256  pixels, 10 fps, intensity image).

Graphic Jump Location
Fig. 25
F25 :

Real-time imaging of a man running (768×256  pixels, 10 fps, range image).

We demonstrated the range imaging with high frame rate of 30 fps and high resolution of 256×256  pixels using SWIR-pulsed TOF laser sensor which is suitable for long range imaging. We additionally demonstrated the long range imaging of more than 1 km, and wide FOV imaging of 12  deg×4  deg with 768×256  pixels and 10 fps. These performances are realized by the key devices that we have developed. The linear array AlInAs-APD array has a feature in the low excess noise and works well at room temperature. The advantage of the above-mentioned low excess noise has not been proved in this paper. However, this feature provides a future potential for better receiving performance, compared with the case of InP-APD, by careful tuning of the bias voltage. The SiGe-BiCMOS ROIC array is an analogue type and offers real-time read-out of intensity and range information with a high read-out rate of 2.56  MHz/pixel. This ROIC also has a function of optimizing the detection threshold for each pixel and tunable ranging width. We deployed the flattop beam illumination optics with the beam division and recombination method for the realization of efficient line shape illumination.

Garcia  E.  et al., “Low-cost three-dimensional vision system based on a low-power semiconductor laser rangefinder and a single scanning mirror,” Opt. Eng.. 40, , 61 –66 (2001).CrossRef
Lamberts  C. W., “Active imaging system: a long range scanned laser,” Appl. Opt.. 15, , 1284 –1289 (1976). 0003-6935 CrossRef
Kameyama  S.  et al., “3D imaging ladar with linear array devices: laser, detector, and ROIC,” Proc. SPIE. 7382, , 738201  (2009). 0277-786X CrossRef
Stettner  R., and Bailey  H., “Eye-safe laser radar 3D imaging,” Proc. SPIE. 4377, , 46 –56 (2001). 0277-786X CrossRef
Marino  R. M.  et al., “High-resolution 3D imaging laser ladar flight test experiments,” Proc. SPIE. 4377, , 138 –151 (2005). 0277-786X CrossRef
Albota  M. A.  et al., “Three-dimensional imaging laser radar with a photon-counting avalanche photodiode array and microchip laser,” Appl. Opt.. 41, , 7671 –7678 (2002). 0003-6935 CrossRef
Dries  J. C.  et al., “A 32×32  pixel flash laser radar system incorporating InGaAs PIN and APD detectors,” Proc. SPIE. 5412, , 250 –256 (2004). 0277-786X CrossRef
Stettner  R., and Bailey  H., “Large format time-of-flight focal plane detector development,” Proc. SPIE. 5791, , 288 –292 (2005). 0277-786X CrossRef
Borniol  E.  et al., “Active three-dimensional and thermal imaging with a 30-μm pitch 320×256 HgCdTe avalanche photodiode focal plane array,” Opt. Eng.. 51, , 061305  (2012).CrossRef
Donnelly  J. P.  et al., “1-μm Geiger-mode detector development,” Proc. SPIE. 5791, , 281 –287 (2005). 0277-786X CrossRef
Eberle  B.  et al., “Novel eye-safe line scanning 3D laser-radar,” Proc. SPIE. 9249, , 92490H  (2014). 0277-786X CrossRef
Tsuji  H.  et al., “Flattop beam illumination for 3D imaging ladar with simple optical devices in the wide distance range,” Opt. Rev.. 23, , 155 –160 (2016). 1340-6000 CrossRef
Yagyu  E.  et al., “Design and characteristics of guardring-free planer AlInAs avalanche photo diodes,” IEEE J. Lightwave Technol.. 27, , 1011 –1017 (2009).CrossRef
Roback  V. E.  et al., “3D flash ladar performance in flight testing on the Morpheus autonomous, rocket-propelled lander to a lunar-like hazard field,” Proc. SPIE. 9832, , 983209  (2016). 0277-786X CrossRef
Ester  M.  et al., “A density-based algorithm for discovering clusters in large spatial databases with noise,” in  Proc. 2nd Int. Conf. on KDD , pp. 226 –231 (1996).
Broome  K. W.  et al., “Demonstration of advanced solid state ladar (DSSL),” Proc. SPIE. 3065, , 148 –157 (1997). 0277-786X CrossRef

Shumpei Kameyama received his DE degree from Chiba University in 2011. Since 1995, he has been with Mitsubishi Electric Corporation and he started the development of laser remote sensing systems in 1999. His current research involves coherent Doppler lidar, differential absorption lidar, and range imaging laser sensor. He is a member of the Japan Society of Applied Physics, Optical Society of America, and Institute of Electrical and Electronics Engineers.

Akihito Hirai received his ME degree from Chiba University in 2007. In 2007, he joined Mitsubishi Electric Corporation, where he has been engaged in research and development of silicon microwave monolithic integrated circuits (Si-MMIC’s) at RADAR and communication systems. He is a member of the IEEE Microwave Theory and Techniques Society and the Institute of Electronics, Information, and Communication Engineers.

Masaharu Imaki received his DE degree from the University of Fukui in 2005. He has been involved in the development of Doppler lidar, Raman lidar, DIAL, and HSRL at the university. In 2007, he joined Mitsubishi Electric Corporation. His current research involves distance imaging sensors, DIAL, and Doppler lidar. He is a member of SPIE, the Japan Society of Applied Physics, the Laser Society of Japan, and the Institute of Electrical Engineers of Japan.

Nobuki Kotake received his DE degree from Nagoya University in 2007. Since 2007, he has been with Mitsubishi Electric Corporation and he started the development of laser remote sensing systems. His current research involves coherent Doppler lidar and range imaging laser sensors. He is a member of the Institute of Electronics, Information, and Communication Engineers and Meteorological Society of Japan.

Hidenobu Tsuji received his BS and MS degrees in physics from Tokyo Institute of Technology in 2004 and 2006, respectively, and his PhD in science from Tokyo Institute of Technology in 2011. He is a head researcher at Mitsubishi Electric Corporation. His current research interests include optics about laser sensors and laser sensor systems.

Yuichi Nishino has been with Mitsubishi Electric Corporation and he started the development of range imaging laser sensor since 2012. He is a member of SPIE.

Yusuke Ito received his PhD from Osaka University in 2015. He serves as a researcher at Mitsubishi Electric Corporation since 2013 and studies coherent Doppler lidar, range imaging laser sensors, and space optical sensor systems.

Mikio Takabayashi received his BE degree from Tokyo University in 1993. He joined Mitsubishi Electric Corporation in 1993 and he has engaged in microwave antenna systems and electro-optical sensor systems. He is a member of the Institute of Electronics, Information, and Communication Engineers and Institute of Electrical and Electronics Engineers.

Yasuhisa Tamagawa received his DE degree from Osaka University in 1998. He joined Mitsubishi Electric Corporation in 1987 and he has been an engineer of electro-optical imaging systems.

Masaharu Nakaji received his BS and MS degrees in electronic engineering from Nagoya Institute of Technology College, Nagoya, Japan, in 1999 and 2001. In 2001, he joined Mitsubishi Electric Corporation, Hyogo, Japan, where he has been engaged in the research and development on high-speed photodetectors.

Eitaro Ishimura received his BS degree in physics from Keio University, Tokyo, Japan, in 1985. In 1985, he joined Mitsubishi Electric Corporation, Hyogo, Japan, where he has been engaged in the research and development on high-speed photodetectors, electroabsorption modulators, and laser diodes. He is currently working on optical devices for communication systems, such as low-noise AlInAs avalanche photodiodes.

Yoshihito Hirano received his DE degree from Osaka University in 2001. Since 1985, he has been with Mitsubishi Electric Corporation and started the research on optical communication equipment, solid-state lasers, nonlinear optics, laser radars, and microwave photonics. He is a member of the Japan Society of Applied Physics, the Laser Society of Japan, the Institute of Electrical Engineers of Japan, and the Optical Society of America. He is also a senior member of IEEE/LEOS.

© The Authors. Published by SPIE under a Creative Commons Attribution 3.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.

Citation

Shumpei Kameyama ; Akihito Hirai ; Masaharu Imaki ; Nobuki Kotake ; Hidenobu Tsuji, et al.
"Demonstration on range imaging of 256×256 pixels and 30 frames per second using short wavelength infrared pulsed time-of-flight laser sensor with linear array receiver", Opt. Eng. 56(3), 031214 (Nov 13, 2016). ; http://dx.doi.org/10.1117/1.OE.56.3.031214


Figures

Graphic Jump Location
Fig. 1
F1 :

Concept of the pulsed TOF imaging laser sensor with the linear array receiver.

Graphic Jump Location
Fig. 2
F2 :

AlInAs-APD array structure.

Graphic Jump Location
Fig. 3
F3 :

Current–voltage characteristic with a multiplication curve of the APD.

Graphic Jump Location
Fig. 4
F4 :

Wavelength dependence of the responsivity.

Graphic Jump Location
Fig. 5
F5 :

Configuration of the element of the ROIC.

Graphic Jump Location
Fig. 6
F6 :

Timing chart of operation of the ROIC.

Graphic Jump Location
Fig. 7
F7 :

Photograph of ROIC chip.

Graphic Jump Location
Fig. 8
F8 :

Offset voltage of OP-Amp and threshold voltage of comparator.

Graphic Jump Location
Fig. 9
F9 :

Variable characteristics of the TAC by setting the digital value [(a) designed characteristics of the measurement range width in each digital value and (b) measured ramp voltage configured by digital value].

Graphic Jump Location
Fig. 10
F10 :

Photograph of the linear array receiver package.

Graphic Jump Location
Fig. 11
F11 :

Schematic for line direction of uniform illumination optics using beam division and recombination method.

Graphic Jump Location
Fig. 12
F12 :

Evaluated result of the distribution of illuminated intensity.

Graphic Jump Location
Fig. 13
F13 :

Configuration of the prototype.

Graphic Jump Location
Fig. 14
F14 :

Internal view of the main container.

Graphic Jump Location
Fig. 15
F15 :

Range imaging result used in the evaluation of ranging precision and accuracy.

Graphic Jump Location
Fig. 16
F16 :

Histogram of the ranging results for the specific pixel.

Graphic Jump Location
Fig. 17
F17 :

Histogram of the ranging results corresponding to the same building wall.

Graphic Jump Location
Fig. 18
F18 :

Real-time imaging of a man running (256×256  pixels, 30 fps, intensity image).

Graphic Jump Location
Fig. 19
F19 :

Real-time imaging of a man running (256×256  pixels, 30 fps, range image).

Graphic Jump Location
Fig. 20
F20 :

3-D imaging acquired in the same experiment corresponding to Figs. 14 and 15.

Graphic Jump Location
Fig. 21
F21 :

Algorithm flow for object detection.

Graphic Jump Location
Fig. 22
F22 :

Intensity images with detected result (red rectangle indicated the result of detection).

Graphic Jump Location
Fig. 23
F23 :

Long range imaging results [(a) intensity image and (b) range image].

Graphic Jump Location
Fig. 24
F24 :

Real-time imaging of a man running (768×256  pixels, 10 fps, intensity image).

Graphic Jump Location
Fig. 25
F25 :

Real-time imaging of a man running (768×256  pixels, 10 fps, range image).

Tables

References

Garcia  E.  et al., “Low-cost three-dimensional vision system based on a low-power semiconductor laser rangefinder and a single scanning mirror,” Opt. Eng.. 40, , 61 –66 (2001).CrossRef
Lamberts  C. W., “Active imaging system: a long range scanned laser,” Appl. Opt.. 15, , 1284 –1289 (1976). 0003-6935 CrossRef
Kameyama  S.  et al., “3D imaging ladar with linear array devices: laser, detector, and ROIC,” Proc. SPIE. 7382, , 738201  (2009). 0277-786X CrossRef
Stettner  R., and Bailey  H., “Eye-safe laser radar 3D imaging,” Proc. SPIE. 4377, , 46 –56 (2001). 0277-786X CrossRef
Marino  R. M.  et al., “High-resolution 3D imaging laser ladar flight test experiments,” Proc. SPIE. 4377, , 138 –151 (2005). 0277-786X CrossRef
Albota  M. A.  et al., “Three-dimensional imaging laser radar with a photon-counting avalanche photodiode array and microchip laser,” Appl. Opt.. 41, , 7671 –7678 (2002). 0003-6935 CrossRef
Dries  J. C.  et al., “A 32×32  pixel flash laser radar system incorporating InGaAs PIN and APD detectors,” Proc. SPIE. 5412, , 250 –256 (2004). 0277-786X CrossRef
Stettner  R., and Bailey  H., “Large format time-of-flight focal plane detector development,” Proc. SPIE. 5791, , 288 –292 (2005). 0277-786X CrossRef
Borniol  E.  et al., “Active three-dimensional and thermal imaging with a 30-μm pitch 320×256 HgCdTe avalanche photodiode focal plane array,” Opt. Eng.. 51, , 061305  (2012).CrossRef
Donnelly  J. P.  et al., “1-μm Geiger-mode detector development,” Proc. SPIE. 5791, , 281 –287 (2005). 0277-786X CrossRef
Eberle  B.  et al., “Novel eye-safe line scanning 3D laser-radar,” Proc. SPIE. 9249, , 92490H  (2014). 0277-786X CrossRef
Tsuji  H.  et al., “Flattop beam illumination for 3D imaging ladar with simple optical devices in the wide distance range,” Opt. Rev.. 23, , 155 –160 (2016). 1340-6000 CrossRef
Yagyu  E.  et al., “Design and characteristics of guardring-free planer AlInAs avalanche photo diodes,” IEEE J. Lightwave Technol.. 27, , 1011 –1017 (2009).CrossRef
Roback  V. E.  et al., “3D flash ladar performance in flight testing on the Morpheus autonomous, rocket-propelled lander to a lunar-like hazard field,” Proc. SPIE. 9832, , 983209  (2016). 0277-786X CrossRef
Ester  M.  et al., “A density-based algorithm for discovering clusters in large spatial databases with noise,” in  Proc. 2nd Int. Conf. on KDD , pp. 226 –231 (1996).
Broome  K. W.  et al., “Demonstration of advanced solid state ladar (DSSL),” Proc. SPIE. 3065, , 148 –157 (1997). 0277-786X CrossRef

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Related Book Chapters

Topic Collections

Advertisement
  • Don't have an account?
  • Subscribe to the SPIE Digital Library
  • Create a FREE account to sign up for Digital Library content alerts and gain access to institutional subscriptions remotely.
Access This Article
Sign in or Create a personal account to Buy this article ($20 for members, $25 for non-members).
Access This Proceeding
Sign in or Create a personal account to Buy this article ($15 for members, $18 for non-members).
Access This Chapter

Access to SPIE eBooks is limited to subscribing institutions and is not available as part of a personal subscription. Print or electronic versions of individual SPIE books may be purchased via SPIE.org.