1.1.

Related Work

Object tracking has been studied widely in the field of video surveillance. In tracking approach, there are two types of object tracking: multiple target tracking and single target tracking. In this paper, we focus on single target tracking. The purpose of our research is to automatically track the object’s bounding box in every frame by using a given bounding box provided by the user including the object of interest in a first frame.

In the current research, object tracking can be classified as follows.

First, deterministic methods⁸ typically track an object by performing an iterative search for the local maxima of a similarity cost function of the template image and the current image. Jurie and Dhome⁹ employed the color distribution, with a metric derived from the Bhattacharyya coefficient as the similarity measure, and used the mean-shift procedure to perform the optimization. The mean-shift algorithm¹⁰ is a popular algorithm for deterministic methods.

Second, the statistical methods solve tracking problems by taking the uncertainties of the measurements and model into account during object state estimation.¹¹ The statistical correspondence methods use the state space approach to model object properties such as position, velocity, and acceleration. Kalman filters¹² are used to estimate the state of a linear system when the state is assumed to have a Gaussian distribution. One limitation of the Kalman filters is the assumption that the state variables are normally distributed. Thus, the Kalman filters will give poor estimates of state variables that do not follow a Gaussian distribution. This limitation can be overcome by using particle filters,¹³ also known as condensation algorithms or sequential Monte Carlo methods, which are efficient statistical methods to estimate target states. Most recent studies^7,8,14–16 have attempted to apply particle filters to the tracking systems so that dependable object tracking results can be achieved. Yang et al.⁸ proposed hierarchical particle filters for tracking fast multiple objects by using integral images for efficiently computing the color features and edge orientation histograms. The observation likelihood based on multiple features is computed in a coarse-to-fine manner. Deguchi et al.¹⁴ employed the mean-shift algorithm to track the target and incorporate the particle filters into the mean-shift result in order to cope with a temporal occlusion of the target and reduce the computational cost of the particle filters. Khan et al.¹⁵ also employed particle filters and mean shift jointly to reduce computational cost and detect occluded objects by estimating the dynamic appearances of objects with online learning of a reference object. Sidibe et al.¹⁶ presented an object tracking method based on the integration of visual saliency information into the particle filter framework to improve the performance of particle filters against occlusion and large illumination variations.

For online learning, Klein et al.⁷ proposed a visual object tracking method using a strong classifier that comprises an ensemble of Haar-like center-surround features. This classifier is learned from a single positive training example with AdaBoost and quickly updated for new object and background appearances with every frame. Saffari et al.¹⁷ and Shi et al.¹⁸ proposed the online random forest (RF) for the object tracking by continuous self-training of an appearance model while avoiding wrong updates that may cause drifting.

The first challenge in object tracking is to build an observation model. The color histogram is a well-known feature for object tracking because it is robust against noise and partial occlusion. However, it becomes ineffective in the presence of illumination changes or when the background and the target have similar colors.¹⁶ A combination of color and edge features is also used for mutual complements.^8,13

The second challenge in object tracking is to design an estimation of the likelihood (distance) between the target object and candidate regions. Several types of distances, such as histogram intersection or Euclidean distance, are used to compute the similarity between feature distributions.⁸ The most popular method to estimate likelihood is using the Bhattacharyya coefficient as a similarity measure.^14,16

The third challenge in object tracking is to recognize and track objects in images taken by a moving camera, such as one mounted on a robot or a vehicle, because this is much more challenging than real-time tracking with a stationary camera. In moving camera applications, the background is not static and the appearance, pose, and scale of a human vary significantly. To track humans in a moving environment, Jung and Sukhatme¹⁹ proposed a probabilistic approach for moving object detection when using a single camera on a mobile robot in outdoor environments. Klein et al.⁷ proposed an object tracking method based on particle filters by adapting new observation models for object and background appearances changing over time in moving camera. Leibe et al.²⁰ integrated information over long time periods to revise its decisions and recover from mistakes by considering new evidence from different camera environments (such as static or moving cameras) and large-scale background changes. Kalal et al.²¹ proposed a tracking framework (TLD) that explicitly decomposes the long-term tracking task into tracking, learning, and detection. The detector localizes all appearances that have been observed so far and corrects the tracker if necessary. The learning estimates detector’s errors and updates it to avoid these errors in the future.

However, since much of the human tracking research based on CCD cameras has many limitations, especially for dark indoor and outdoor environments owing to poor illumination, a few algorithms for tracking humans in thermal images have been tried.

Li and Gong⁴ constructed the regions-of-interest histogram in an intensity-distance projection space model with a particle filter to overcome the disadvantage of insufficient intensity features in thermal infrared images. Padole and Alexandre⁵ used two types of spatial and temporal data association to reduce false decisions for motion tracking with thermal images alone. Xu et al.²² proposed a method for pedestrian detection and tracking with a single night-vision video camera installed on a vehicle. The tracking phase for the heads and bodies of pedestrians is a combination of Kalman filter prediction and mean shift.

Fernandez-Caballero et al.²³ proposed an approach to real-time human detection and tracking through the processing of thermal images mounted on an autonomous mobile platform. This method simply used static analysis for the detection of humans through image normalization and optical flow for enhancing the human segmentation in moving and still images.

However, there exist nonhuman target objects, such as buildings, cars, animals, and light poles, which have intensities similar to that of humans in thermal images.⁴ Therefore, it is very difficult to maintain correct tracking when humans overlap while crossing paths. To solve these problems, some recent tracking systems use additional information from color CCD cameras.²⁴ Leykin and Hammoud¹ proposed a system to track pedestrians by using the combined input from RGB and thermal cameras. First, a background model is constructed with color and thermal images. Then, a pedestrian tracker is designed using particle filters. Han and Bhanu²⁵ proposed an automatic hierarchical scheme to find the correspondence between the preliminary human silhouettes extracted from synchronous color and thermal image sequences for image registration without tracking. Cielniak et al.²⁴ proposed a method for tracking multiple persons with a combination of color and thermal vision sensors on a mobile robot. To detect occlusion, they proposed a machine learning classifier for a pairwise comparison of persons using both the thermal and color features provided by the tracker.

However, these human tracking methods based on thermal sensors or thermal and color sensors have the following typical disadvantages:

To improve the human tracking performance for moving cameras while minimizing the computation time for darkness, this study proposes a novel human tracking approach for thermal videos that are based on online RF learning and combination of a local intensity distribution (LID) with oriented center-symmetric local binary patterns (OCS-LBP). As shown in Fig. 1, we design a real-time RF, which is the ensemble of decision trees for confidence estimation, and confidences of the RF are converted into a likelihood function of the target state. In the initial stage, the target model is selected by the user and particles are sampled. In the second and third stages, subblock-based RFs are generated using the long-term positive and negative examples with LID and OCS-LBP features by online learning. The learned RF classifiers are used to detect the most likely target position in the subsequent frame in the fourth and fifth stages. Then, the RFs are learned again by means of fast retraining with the tracked object and background appearance in the new frame.

Fig. 1

Block diagram of the human tracking procedure using thermal images.

This human tracking method based on RF combined with an LID and OCS-LBP allows human tracking to be performed in near real-time with a mobile thermal camera. Moreover, the tracking accuracy increases compared with that of a conventional human tracking method for thermal images.

The remainder of this paper is organized as follows. Section 2 describes the target representation method using LID and OCS-LBP features. Section 3 introduces the basic human tracking method using particle filters. Section 4 introduces the proposed human tracking method that incorporates online RF learning to avoid tracking drift caused by pose variations, illumination changes, and occlusion. Section 5 presents an experimental evaluation of the accuracy and applicability of the proposed human tracking method. Section 6 summarizes our conclusions and discusses the scope for future work.

2.1.

Local Intensity Distribution

A color histogram based on distance is a frequently used feature for object tracking.^4,10,14 However, the major characteristic of a human body in a thermal image is high intensity without color information, and individual humans exhibit distinct temperatures. Therefore, intensity is a better feature than color to distinguish humans from a background and other objects. In this research, we divide the bounding boxes of a target model and a target candidate into adjacent $3\times 2$ subblocks to create a robust feature model for object occlusion as shown in Fig. 2. Partitioned subblocks are beneficial if the size of a box is relatively large. This was justified by the experiment of Khan et al.,¹⁵ which revealed that an object often contains multiple local modes and that partitioned subblocks track objects more correctly than a single box when occlusion occurs.

Fig. 2

Representation of oriented center-symmetric local binary patterns (OCS-LBP) histogram generation. Local OCS-LBP histograms generated from each subblock and their gradient orientation histograms.

In accordance with the research of Deguchi et al.¹⁴ and Comaniciu et al.,¹⁰ let ${\{{\mathbf{x}}_j^i\}}_{j=1\dots n}$ be the normalized pixel locations in the $i$’th subblock defined as the target model. The normalized LID is represented by $m$-component ($m$-bin) histograms and the LID of the $i$’th subblock of the target model is denoted by ${\mathbf{q}}^i={\{q_u^i\}}_{u=1...m}$, where $q_u^i$ is the $u$’th histogram component of $i$’th subblock. Since pixels will be more in the peripheral region than in the center, a normal histogram is affected by occlusions and interference from the background. Therefore, we use the Epanechnikov kernel $k(\mathbf{x})$, which is an isotropic kernel that assigns greater weights to pixels at the central points of the subblocks as follows:¹³

Finally, the probability of the features $u=1,\dots ,m$ in the target model of $i$’th subblock is estimated as

The LID of the $i$’th subblock of the target candidate centered at $y$ in the current frame is denoted by ${\mathbf{P}}^{\mathbf{i}}(\mathbf{y})={\{p_u^i(y)\}}_{u=1\dots m}$, where $p_u^i(y)$ is the $u$’th histogram component of the $i$’th subblock. Using the same Epanechnikov kernel $k(\mathbf{x})$ and different bandwidth, depending on the size of the candidate box, the probability of the $i$’th subblock in the target candidate is estimated as

2.2.

Oriented Center-Symmetric LBP

In human detection, texture features such as the histogram of oriented gradient (HOG)²⁶ and LBP (Ref. 27) are popular features to discriminate humans from backgrounds. Recently, the LBP texture operator has been successfully used in various computer vision applications, such as face recognition,²⁸ human detection,²⁹ and human tracking,³⁰ because it is robust against illumination changes, very fast to compute, and does not require many parameters.³¹ LBP describes the gray-scale local texture of the image with low computational complexity by using a simple method. The original LBP descriptor forms different patterns based on the number of pixels by thresholding a specific range of neighboring sets with the central gray-scale intensity value. Even though LBP are widely used as a texture operator, they produce rather long histograms. Ma et al.³² combined HOG and LBP to compute oriented LBP feature. First, they define the arch of a pixel as all continuous “1” bits of its neighbors. Then, the orientation and magnitude of a pixel is defined as its arch principle direction and the number of “1” bits in its arch, respectively.

CS-LBP (Ref. 33) uses a modified scheme comparing the neighboring pixels of the original LBP to simplify the computation while keeping the characteristics such as tolerance against illumination changes and robustness against monotonic gray-level changes. CS-LBP is different from LBP in that differences between pairs of opposite pixels in a neighborhood are calculated, rather than comparing each pixel with the center. This halves the number of comparisons for the same number of neighbors and produces only 16 ($2^4$) different binary patterns. However, since the original CS-LBP lose the orientation and magnitude information, we introduce a new lower-dimensional feature-oriented CS-LBP (OCS-LBP) using a different approach of oriented LBP.³²

In order to extract an oriented histogram of OCS-LBP from a subblock, gradient orientations are estimated at every pixel and a histogram of each $k$’th orientation in a neighborhood is binned using Eqs. (5) and (6). Each pixel influences the gradient magnitude for an orientation according to the closest bin in the range from 0 to 360 deg at 45 deg intervals. In Eq. (6), robustness is maintained in flat image regions by thresholding the intensity-level differences using a small value $T$ in Eq. (5), as follows:

In Fig. 2, gradient orientation is confirmed when the differences between pairs of opposite pixels in a neighborhood are over the threshold. For example, the absolute difference between the values of $n_0$ (130) and $n_4$ (80) is over the threshold $T$ and $n_0$ is greater than $n_4$, so the absolute difference (magnitude) is assigned to the zero bin. The gradient orientation histogram for each orientation $k$ of a subblock is obtained by summing all the gradient magnitudes whose orientations belong to bin $k$. After that, the final set of $k$ OCS-LBP features of a single subblock is normalized by the min-max normalization.

Using the same method with the LID, the bounding box of a target model and a candidate are divided into $3\times 2$ adjacent subblocks and OCS-LBP histograms are extracted from each subblock. The number of subblock is decided according to the experiment results of Ref. 7. In Ref. 7, the target object was divided by nonoverlap $2\times 2$ subblocks to make a robust target model about occlusion. However, we change the number of subblocks as $3\times 2$ based on the human body ratio. All local OCS-LBP histograms are then used for online learning of the RF classifier.

Prediction Step

Given all available observations $y_{1:t-1}=\{y_1,\dots ,y_{t-1}\}$ up to time $t–1$, the prediction state uses the probabilistic system transition model $p(x_t|x_{t-1})$ to make a posterior prediction at time $t$.

Updating Step

At time $t$, the observation $y_t$ is available, so the state can be updated using Bayes’ rule.

The candidate samples ${\tilde{x}}_t^i$ are drawn from an importance distribution $q({\tilde{x}}_t|x_{1:t-1},y_{1:t})$ and the weights of the samples.

In the case of bootstrap filters, particle weights are iteratively estimated from the observation likelihood.

Resampling Step

Since the probability of most samples is negligible, $K$ particles having large weights are statistically selected more often than others, and the posterior state vector ${\widehat{x}}_t$ is updated as the weighted average over the states of the particles.

4.1.

Initialization of Target Model

Particle filters are sequential Monte Carlo methods that recursively approximate the posterior distribution using a finite set of particles over time $t$. In this paper, we define the set of particles as $P{t}_t={\{p{t}_t^i\}}_{i=1..N}$, where the $i$’th particle ${\mathrm{pt}}_t^i$ at time $t$ consists of its weight ${\omega }_t^i$ and state vector

In the initial stage, the position and bounding box are manually selected and the state vector of the initial target ${\mathrm{Tr}}_1={[c{x}_1,c{y}_1,w_1,h_1,R{F}_1]}^T$ is then set automatically according to the user selection. $R{F}_t^i$ is the classifier determined by online learning at time $t$.

4.2.

State Prediction

In the prediction stage of the particle filters, $N$ particles are propagated through the second-order autoregressive motion model³⁵ to predict the particle positions. The center position of the $i$’th particle is interpolated from the previous position ($c{x}_{t-1},c{y}_{t-1}$), the average velocities at times $t–2$ and $t–1$, and white Gaussian noise [$G(0,{\sigma }_x^2),G(0,{\sigma }_y^2)$].

In the case of the second frame, only the velocity is linearly combined with the previous position and white Gaussian noise.

The box size of the $i$’th particle is linearly interpolated from the previous box size of the target object ($w_{t-1},h_{t-1}$) and white Gaussian noise [$G(0,{\sigma }_w^2),G(0,{\sigma }_h^2)$].

4.3.

Subblock-Based Random Forest Learning

An RF proposed by Breiman³⁶ is a decision tree ensemble classifier, with each tree grown using some type of randomization. This RF has a capacity for processing vast amounts of data, with high learning speeds, based on a decision tree.

For the learning of the initial RF, training data are constructed using a positive example that is selected by the user and two negative examples that are randomly sampled from the background of the first frame. In the second frame, the training data are increased to two positive examples and four negative examples. Negative samples are randomly selected from outside of a tracked object regardless of background cluttering. Training data are increased in the ratio $1:2$ until 15 frames. The memory capacity is 15 for positive examples and 30 for negative examples. Every new target is added to positive memory and the RF is learned using the limited number of positive examples until the 15th frame. In contrast, negative examples are updated as the background at each frame according to the increase in frames. After the 15th frame, we always keep the five positive examples from the 1st through 15th frames in order to avoid the template drift problem by modifying the idea of Klein et al.⁷ Moreover, the reminder of the positive memory is occupied by the new example and the oldest example is discarded, like in a queue, because the more similar history of the positive examples produces more confident classifiers.

In this research, each particle is divided into six subblocks as mentioned in Sec. 2.2, and two types of RF classifiers for the $i$’th subblock are learned using the LID and OCS-LBP extracted from the corresponding blocks in the 45 training examples.

Let $F$ be the set of RFs ${\{(r{f}_i)\}}_{i=1\dots S}$, where $S$ is the number of subblocks. The $i$’th RF, ${\mathrm{rf}}_i$, is represented as ${\mathrm{rf}}_i=({\mathrm{srf}}_i^{\mathrm{lid}},{\mathrm{srf}}_i^{\mathrm{ocs}})$. Here, we construct two RFs, ${\mathrm{srf}}^{\mathrm{lid}}$ and ${\mathrm{srf}}^{\mathrm{ocs}}$, for each subblock: one uses only the LID feature (${\mathrm{srf}}^{\mathrm{lid}}$) and the other uses only the OCS-LBP feature (${\mathrm{srf}}^{\mathrm{ocs}}$), rather than combining these into one feature vector according to the experiments of Ko et al.,³⁷ because the basic characteristics of the LID and OCS-LBP are different. Therefore, the total number of RFs at time $t$ is 12 ($2\mathrm{RFs}\times 6$ subblocks).

The learning of the RFs in $i$’th subblock at time $t$ is summarized below.

When the class label set is denoted by $C=\{\text{Positive},\text{Negative}\}$, a leaf node $n$ has a posterior probability, and the class distributions of $l$ trees, $p(C_i|l)$, are estimated empirically as a histogram of leaf nodes on a class label, $C_i$.

The depth of the trees is set at 20 according to the results of Ko et al.,³¹ and the number of trees is four each for ${\mathrm{srf}}^{\mathrm{lid}}$ and ${\mathrm{srf}}^{\mathrm{ocs}}$. The experimental results for deciding the appropriate number of trees are described in Sec. 5.3.

4.4.

Likelihood Estimation Using RFs

After a set of RFs is learned on positive and negative training examples of frame $t$, the observation likelihoods for each particle of frame $t$ are estimated using RF classifiers. The reference feature histogram, the LID, and the OCS-LBP of the $i$’th subblock of a test particle are applied to the corresponding ${\mathrm{rf}}_i=({\mathrm{srf}}_i^{\mathrm{lid}},{\mathrm{srf}}_i^{\mathrm{ocs}})$. The likelihood of the $i$’th subblock is estimated by combining the probabilities of ${\mathrm{srf}}_i^{\mathrm{lid}}$ and ${\mathrm{srf}}_i^{\mathrm{ocs}}$. The test image is used as input to the learned RF, and the probability distribution (likelihood) of the $i$’th subblock in the positive class is generated by ensemble (arithmetic) averaging of each distribution of all trees $L=(l_1,l_2,\dots l_T)$ using Eqs. (15) and (16).

Hence, the final likelihood of a particle is estimated from Eq. (17).

This process is continued iteratively until the likelihoods of all particles are computed.

Once the final likelihood ($P^j$) of the $j$’th particle is estimated, the weight ($w_t^j$) of the $j$’th particle at time $t$ is replaced by using the likelihood $P^j$ obtained from the RF and each weight is normalized.

The state of the current target is updated as the top $J(=15)$ particles having greater weight.

4.5.

Online Relearning of RF Classifiers

When a tracking human target is detected in a current frame, the RFs should be relearned using the updated history including positive and negative examples. The purpose of online RF learning is to avoid tracking drift caused by pose variation, illumination changes, and occlusion.

The basis of the proposed online RF learning is to compute the difference in target state between the current and previous targets and only relearn the RF for the current frame if this difference and the probability of the target satisfy the conditions. In this paper, the learning condition is adaptively changed by using Eq. (21) according to the variance in thermal intensity of a target region.

Online RF learning consisted of the procedures described below.

$c_t^k$: Center of a tracked target specified by the current state vector $\overline{\mathrm{Tr}}$ at time $t$.

OC: Counter for duration check of full occlusion ($\mathrm{OC}=0$)

In procedure 2.2, we use the intensity variance of the target region to determine the learning condition. This condition is based on the fact that the probability of the RF on the current target is low as the intensity variance of the target is high. The test result for a minimum RF learning threshold of 0.72 is described in Sec. 5.1. In procedure 2.4.3, we check the duration of full occlusion and occlusion counter (OC) increases its number whenever continuous full occlusion is occurring. Then, if the total number of OC is over the terminal condition, tracking is terminated in procedure 4. The terminal condition T2 is a changeable threshold according to the application and we set it as 30 frames.

5.1.

Tests on Minimum Threshold and Condition for RF Learning

In our study, the most appropriate minimum threshold to use in Eq. (21) for updating training data and RF learning was found to be 0.72 on the basis of several experiments. To determine the proper threshold for RF learning, four test data were selected from the test dataset shown in Table 1, namely, Videos 1 and 2 (OTCBVS data) and Videos 5 and 6 (our data). We selected the two videos from the OTCBVS data (i.e., Videos 1 and 2) because in these videos two persons walk in a cluttered background and become fully occluded by each other. We selected Videos 5 and 6 of our data because two persons walk in different directions and become fully occluded by a tree and each other. In the first experiment, the minimum threshold for RF learning was estimated by changing the value of the static threshold. As shown in Fig. 3, a minimum threshold of 0.72 for RF learning exhibited the best performance, with an average $A({\mathrm{GT}}_{ik},{\mathrm{ST}}_{jk})$ value of 80.3%. Therefore, 0.72 was adopted in Eq. (21) as the minimum threshold for RF learning.

Fig. 3

Experimental results for five possible pairs to determine minimum threshold for updating training data and random forest (RF) learning.

For RF learning, we imposed the learning condition [Eq. (21)] using the minimum threshold. The purpose of the condition [Eq. (21)] is to design an adaptive RF classifier depending on the variation of intensity. To verify the performance of the learning condition, we compared the average $A$ values of the static threshold determined in Fig. 3 with those of the learning condition [Eq. (21)] for the same four test data. As shown in Fig. 4, the adaptive learning condition exhibited the better performance for all four videos, with an average of 84.5 versus 80.3%.

Fig. 4

Verification of performance of learning condition by comparing the values of the proposed updating condition and a static threshold.

5.2.

Determination of Optimal Number of Particles

The main disadvantage of particle filters is the computational cost of using a large number of particles, even though particle filters are known to be robust in visual tracking through occlusions and cluttered backgrounds.¹⁵ Therefore, it is essential to find the proper number of particles by considering the computational cost. Figure 5 shows the results of experiments using five possible values for the number of particles. As shown in Fig. 5, even though 40 particles gave the shortest processing time, the tracking performance was the worst. In contrast, 120 particles gave the best tracking performance and relatively good processing time, so 120 was adopted as the number of particle filters.

Fig. 5

Five possible pairs of experimental results to determine number of particles.

5.3.

Determination of Optimal Number of Trees

The RF is known to be very fast in learning and testing as compared to other classifiers, i.e., the multiclass support vector machines.³¹ The important parameters of the RF are the depth of the trees and the number of trees, $T$. Although increasing the depth of the trees and the number of trees improves the performance, the runtime cost depends on the depth of each tree and the number of trees. In our study, we set the maximum depth of the trees at 20 according to the experiments of Ref. 31.

To determine the proper number of trees for a local RF, we used the same four test data and compared the tracking performance by changing the number of trees. As shown in Fig. 6, when the number of trees for a local RF was four, the tracking performance was the best and the processing time was relatively good. Therefore, we adopted four trees for a local RF. In this study, we constructed two RFs per subblock: one uses only the LID feature and the other uses only the OCS-LBP feature, so the total number of trees for a target is 48 ($2\times 4\times 6$).

Fig. 6

Five possible pairs of experimental results to determine number of trees per local RF.

5.4.

Performance Comparison for Online RF Learning Versus Static RF

Online RF learning is the main technique to track occluded humans and avoid tracking drift caused by pose variation, illumination changes, and occlusion in a cluttered background captured by a moving camera. To evaluate the effectiveness of the proposed online RF learning, we compared the tracking performance with online learning to that without online learning (static RF). A static RF is learned but once, when the user selects the human rectangle, and human tracking is performed by the static RF classifier without relearning.

Figure 7 shows a performance comparison of human tracking methods for the same four test data. As shown in Fig. 7, the online RF learning produced a better tracking performance with an average tracking success rate of 84.5% compared to 75.1%. The main reason for the higher tracking success rate of the proposed online RF learning is that the long-term full or partial occlusion between persons and tree is reflected in the training history and the RF learning.

Fig. 7

Tracking performance using the proposed method improved by 9.4% when compared to that using the static RF method.

5.5.

Comparisons Between Different Algorithms

To evaluate the performance of the proposed algorithm, the proposed method was compared with OCS-LBP with RF ($\mathrm{OCS}\text{-}\mathrm{LBP}+\mathrm{RF}$) and LID with RF ($\mathrm{LID}+\mathrm{RF}$). In addition, we evaluate three different types of related works: (1) LID with particle filters using thermal image⁴ ($\mathrm{LID}+\text{particle filters}$), (2) simple online RF learning using Haar-like feature,¹⁷ (3) TLD tracker²¹ that is known as a robust object tracking algorithm in a moving camera. The experiments were performed using the same dataset as described in Table 1. As shown in Fig. 8, the overall performance of our proposed approach exceeded that of the other two combinations, the particle filters,⁴ simple online RF,¹⁷ and TLD tracker,²¹ based on the $A$ percentages of 81.9, 69.6, 57.2, 69.9, 70.9, and 62.2%. From the results, we can infer that an individual intensity feature is not a distinguishing feature for human tracking in thermal images, particularly for cases of human occlusion. In contrast, the OCS-LBP feature produced reasonable tracking results even in thermal images. Even though simple online RF and particle filter produced the second and third best tracking performance of the other three methods, they still showed a few missing or false detection results when occlusions occurred. TLD tracker showed the worst tracking results, showing that learning and detection algorithm of TLD is not appropriate for human tracking in thermal image. The test results showed that for robust and practical tracking, the combination of two features is superior to the individual feature-based human representation model in thermal video.

Fig. 8

Overall performance comparison of the proposed tracking algorithm with five other methods using the same dataset.

For a more detailed evaluation of tracking performance, Fig. 9 shows comparisons between the proposed method and the two methods [particle filters⁴ with LID and simple online RF (Ref. 17)] in terms of the $A$ performance versus the frame number for Videos 1, 5, and 9. The ground truth of the target object is marked manually.

Fig. 9

Results of tracking the target object with three different methods using A score for (a) Video 1, (b) Video 5, and (c) Video 9.

In Videos 5 and 9, the tracking method based on particle filters⁴ and simple online RF (Ref. 17) lost the target object when occlusions occurred or camera is moving. In case of simple online RF,¹⁷ it showed the worst performance in Video 9 because it could not distinguish the real target from background when occlusions occurred. However, for all three videos, the proposed scheme had a significantly smaller error and more robust results than the other methods, regardless of the full or partial occlusion and the camera movement.

Figure 10 shows the computational speeds of the six methods. As shown in Fig. 10, the proposed method (at 10.2 fps) requires more computation time than the other methods (at 14.8, 18.8, 15, 20.2, and 22 fps) because it uses subblocks of particles, online RF learning, and two types of RF. When the online learning was not applied (static RF), the tracking speed was approximately the same as that of the $\mathrm{LID}+\mathrm{RF}$ ($\sim 19\mathrm{fps}$) and faster than that of the $\mathrm{CS}\text{-}\mathrm{LBP}+\mathrm{LID}$ using conventional particle filters. Simple online RF (Ref. 17) showed highest computational speed as 22 fps because it used simple learning RF with Haar-feature. Because the main reason for computational delay is the online RF learning for individual subblocks, optimization of the real-time learning may be considered for the next version.

Fig. 10

Comparison between computational speeds of five methods.

Figure 11 shows the tracking results obtained for Videos 4, 5, 9, 11, and 12 by using our proposed method. From the results in Figs. 11(a) to 11(e), we deduce that our proposed method accurately and robustly tracks moving objects, despite background clutter with similar intensity distributions [(b), (c), and (e)], object intersections [(a) to (d)], long-term full (or partial) occlusion [(a) to (d)], and camera movement [(c) and (d)].

Fig. 11

Human tracking results obtained using the proposed method. Red boxes are regions tracked by online RF learning, and blue boxes are regions tracked by the static RF due to the partial or full occlusion.

The complete video sequences can be viewed at the following webpage: http://cvpr.kmu.ac.kr.