1.

Introduction

Conventional fire detection systems were designed to detect smoke, heat, and radiant energy from a fire using infrared, optical, and ion sensors.^1–3 However, these methods have problems, namely, it is impossible to tell whether a fire is occurring until smoke or flame spreads to the detection range of sensors and detection takes a long time because a fire alarm is only issued after the flame’s influence exceeds a reference temperature or a set value. In addition, because the detection range is restricted to the local position where sensors are installed, the need for numerous distributed sensors and installation increases the cost. On the other hand, a fire detection system using image processing does not require any additional costs, as it uses surveillance cameras already installed in public places, roads, and tunnels. Thus, fires can be quickly detected without having to wait until flames spread to the sensors, and can be monitored over a broader range.

Early fire detection algorithms used rule-based color models.^4–6 They classify the image pixels as fire or not fire by using the RGB, hue, saturation and intensity, and YCbCr color models. However, these methods have many false classifications. For example, red clothes, moving cars, and lights will be recognized as fires because only color information was used. Han⁷ used color and motion information to present real-time fire detection. Töreyin et al.⁸ used temporal and spatial wavelet analysis to separate the fire regions from sequential images. Although they presented good experimental results, this approach is impractical in various environments due to many heuristic thresholds. Z. Teng⁹ modeled the sequentially changing image pixel values by a hidden Markov model framework. Ko et al.¹⁰ applied a radial basis function kernel to two-class support vector machines. This research that uses machine learning algorithms could have an over-fitting problem and are not suitable for various environments. In most of the previous studies, it is necessary to determine whether an individual pixel is a fire or nonfire pixel. However, red objects that have a hue as fire are often mistaken for fire when determining each image pixel. In addition, moving lights, such as automobile headlamps, can be mistaken for fires when using only motion information.

Therefore, we defined the color conditions of fire features in both hue, saturation and value (HSV) and RGB color space. The local features and motion information of fires were expressed using the mean, variance, and image difference. Moreover, temporally blinking properties were modeled by using crossing patches so that it was possible to reduce the false detections described above. We defined 22 image features to classify patches into fire or nonfire areas by using a random-forest-supervised learning algorithm. The additive Markov chain was introduced to reduce false alarms. Also, we designed an embedded surveillance device made from acrylonitrile butadiene styrene (ABS) for stable fire detection in outdoor environments.

All modules, including image features extraction and fire classification, are realized to test fire images of different conditions and verified to be robust in complex environments.

2.

Embedded Camera System

In general, to protect people from fires, fire surveillance systems require supervisors in monitoring rooms to observe videos transferred from CCTVs. Building a surveillance system requires considerable effort to install surveillance cameras, as many repeaters and wiring inside facilities are needed to transfer videos to monitoring rooms, causing increased installation costs. In addition, all images are concentrated on servers in monitoring rooms and so the structure of server computers becomes complex. Therefore, we designed an embedded system with a built-in pan-tilt-zoom (PTZ) camera to observe fire areas more effectively. The proposed system, which has a built-in embedded processor inside the camera, is capable of real-time image processing and consists of a main camera that can monitor fires over a wide range and a subcamera that can quickly zoom into fire areas when a fire occurs.

The early-designed device fixed each module by installing a fixed bracket inside a universal housing, as shown in Fig. 1(a) and a subcamera below the housing. Due to heat from the camera and embedded board, a prolonged operating time rapidly raised the internal temperature of the housing. In addition, the commercial housing suffered from the disadvantages of heavy weight and expensive cost. In the second system design, the cooling performance was improved by projecting the cooling section of the embedded board outside, as shown in Fig. 1(b). This system reduced weight by using an ABS resin housing and a digital output main camera. The ABS housing has the advantage of reducing production costs because it can be produced in large quantities by injection molding. Protection from dust and moisture was accomplished by using a rubber seal and waterproof connector, as various foreign bodies in the form of particulate could otherwise enter the system.

Fig. 1

Structure of fire detection system. (a) Fire detection system using commercial housing. (b) Customized system showing ABS resin housing and cooling conductor.

The whole operation of the fire detection system is depicted in Fig. 2. The main camera is a fixed type and acquires raw images to detect fire while monitoring the wide surveillance area. In addition, the installed industrial digital camera is able to remove the noise generated in the process of converting images from analog to digital. This can simplify the system and permit control directly by the program. A PTZ camera with maximum 10-magnification was selected for the subcamera so that it is possible to immediately magnify and transmit images of the area where a fire has occurred. The images acquired from the main camera are processed in the built-in embedded board, and thus fire areas are detected. When the fire areas are detected, magnified images of fires are acquired from the subcamera. The acquired images from the subcamera are converted into digital images by a frame grabber. Information, such as a gazing direction and scale, is transmitted by RS-422 communication to the main processor, for calculating PTZ motion data from the fire area detection of the main camera. Finally, the system transfers the fire location and magnified images of the fire area to the monitoring room. Figure 3 is a graphical user interface screenshot of the fire monitoring system and an image that detects fires and magnifies the detected areas.

Fig. 2

Operation process of the fire detection system.

Fig. 3

Magnified images by designating a fire detection area and using a PTZ camera.

Figure 4 shows CPU temperature change in both the customized and commercial housing during 24 h. The measurements were conducted after a detection algorithm operated for 1 h in each housing. It indicates that average improvement of cooling performance is 22.017°C.

Fig. 4

CPU Temperature changes in customized and commercial housing during 24 h.

3.

Fire Detection Algorithm

In previous studies,^6–10 information from each pixel in the image was used to determine whether fires exist in the images. Alternatively, our paper proposes a method to model the local characteristics of fire, represented in sequential images based on patches of an input image, and to detect a fire area by using a supervised learning algorithm. The random forest showed a higher recognition rate than other machine learning algorithms.¹¹ It can process substitution branching for lost data processing and handle both categorical and numerical data. In addition, it makes it unnecessary to normalize image features and easy to understand the importance of variables.

First, in order to model the local features of fires, the input image is divided into small $r\times c$ patches (P), as shown in Fig. 5. We generate the color conditions, motion, and blink information of fires as features in the patch.

Fig. 5

Image Patch by block division of image.

3.1.

Modeling Fire Features

In general, fires have specific colors in images. In this section, fire features were analyzed in RGB and HSV color space. For color features of fire in the RGB color space, it was observed that the R (red) color is larger than the threshold value $R_{\mathrm{Th}}$ and the R color value is the largest in the RGB channel. Moreover, in the HSV color space, it is shown that the H (hue) value of fire is 0 to 70 or 290 to 360. For example, the feature $f_3$ means the saturation will decrease with the increasing red component, and thus the expression indicates that saturation will decrease downward to zero when the red component increases toward the upmost value 255 (Refs. 12, 13). Therefore, the patch is likely to indicate a fire area when there are many pixels satisfying these fire conditions. Features ($f_i$) 1 to 4 show how many pixels satisfy each condition in the patch (Table 1).

Table 1

Pseudo-code for the feature calculation of f1∼f4.

$\text{Initiailize}{f}_1\sim f_4$

For all pixels in a patch

$\text{if}(P{(r,c)}_R>R_{\mathrm{Th}}),f_1++;$

$\text{if}(P{(r,c)}_R>P{(r,c)}_G>P{(r,c)}_B),f_2++;$

$\text{if}(P{(r,c)}_S\ge (255-P{(r,c)}_R)\times S_{\mathrm{Th}}/R_{\mathrm{Th}}),f_3++;$

$\text{if}(290\le P{(r,c)}_H\le 360\mathrm{or}0\le P{(r,c)}_H\le 70),f_4++;$

The pixels in the fire areas are similar to specific colors such as red, yellow and white, and the deviation of colors is relatively uniform. Therefore, Features ($f_i$) 5 to 10 are defined by calculating the color mean and the variance for each RGB channel.

Eq. (1)

$$P_C(r,c)\sim N({\mu }_C,{\sigma }_C^2),C\in \{R,G,B\}$$

Eq. (2)

$$f_{[5\dots 10]}=\{{\mu }_R,{\mu }_G,{\mu }_B,{\sigma }_R^2,{\sigma }_G^2,{\sigma }_B^2\}$$

Additionally, it is observed that fires appear to blink in sequential images. Using this, the number of blinks in a certain area accumulated over 10 frames is defined as a feature.

Eq. (3)

$$\mathrm{Diff}=\sum _{p_i\in \mathrm{black}\mathrm{area}}I(p_i)-\sum _{p_i\in \mathrm{white}\mathrm{area}}I(p_i)$$

Eq. (4)

$$\text{if}|{\mathrm{Diff}}_t-{\mathrm{Diff}}_{t-1}|>\mathrm{threshold},\mathrm{then}\mathrm{crossing}\mathrm{occurs}\mathrm{.}$$

We defined three cross extractors to calculate the brightness difference between the previous and subsequent patches as shown in Fig. 6. If the difference in brightness exceeds a threshold value, then it is considered to be a crossing count. $I(p_i)$ in Eq. (3) is a brightness value at the pixel $p_i$ in the image. In this paper, a total of nine features were generated by using three different thresholds deciding the cross state in order to know the level of blinking.

Eq. (5)

$$f_{[11\dots .19]}=\{{\mathrm{crossing}}_{\mathrm{th}1}^i,{\mathrm{crossing}}_{\mathrm{th}2}^i,{\mathrm{crossing}}_{\mathrm{th}3}^i\},i\in \{1,2,3\}\mathrm{.}$$

Fig. 6

Cross extractors.

Finally, this study used the feature that many motions could be observed between previous and present frames in fire areas. Automobile headlamps are known to generate strong noise, due to the dispersion of light when using the common image difference between previous and subsequent frames. Therefore, motion pixels are defined as features after the removal of the noise by using the average brightness of the three previous and subsequent frames.

Eq. (6)

$$f_{[20\dots 22]}=\sum _{r\in P}^r\sum _{c\in P}^c|\frac{P_C^t(r,c)+P_C^{t-1}(r,c)+P_C^{t-2}(r,c)}{3}-\frac{P_C^{t-1}(r,c)+P_C^{t-2}(r,c)+P_C^{t-3}(r,c)}{3}|,C\in \{R,G,B\}\mathrm{.}$$

Figure 7 is the result that depicts the features defined above in an image. The image shown in the upper right indicates the area meeting the color conditions of Features 1 to 4. In addition, the image shown in the lower left represents the motion of overlapping frames corresponding to features 20 to 22. An image shown in the lower right represents the crossing blocks between the previous and present frames.

Fig. 7

Feature extraction from a fire image. (a) Original image. (b) Color condition. (c) Motion information. (d) Crossing.

3.2.

Learning RF and Classifying Fire Features

In order to classify the features defined in the previous section into fire and nonfire areas, this study used the random-forest supervised-learning method.¹⁴

The input feature ($f$) reaches the leaf node after following the binary branch of decision tree ($t_n$). Each node ($N$) in the ensembles $t_1(f),t_2(f),\dots t_n(f)$ selects a feature minimizing the Gini impurity ($i(N)$) in order to determine the feature and threshold value that can classify the feature into a suitable class most effectively. Finally, probability [$P(c|f)$] of each class is calculated by the summation of each leaf node probability, as shown in Fig. 8.

Eq. (7)

$$i(N)=\sum _{j\ne i}P(c_i)\log P(c_j)\mathrm{.}$$

Fig. 8

Random forest.

The assembly of each ensemble classifier generated in this manner is called a random forest. When given ensemble classifiers, feature vector $F$ and class $C$, the feature vector is classified into fire as follows:

Eq. (8)

$$\text{if}[P(c=\mathrm{fire}|f)>P(c=\mathrm{Non}\mathrm{Fire}|f)]\mathrm{then}\mathrm{Fire}\mathrm{else}\mathrm{NonFire}$$

The whole process of learning and classification is described in Fig. 9. In order to learn fire, the input image is divided into the unit area. The 22 features defined in the previous section are extracted from the divided images. When learners input the sample groups $\{c,f\},c\in \{\mathrm{fire},\mathrm{NonFire}\}$ into a learning machine, including classes and features generated from the fire and nonfire images as shown in Fig. 10, the ensemble classifier and decision rule are generated by the random forest. When images are captured in real-time, the random forest decides the fire and nonfire area by the decision rule.

Fig. 9

Learning and classifying process.

Fig. 10

Image patches used for learning. Fire sample (a), nonfire sample (b).

4.

Experiments

To learn fire features, 10,000 and 40,000 samples of a fire and nonfire, respectively, were extracted from video images, such as in a tunnel, downtown, and mountain area. Numerical type feature data were used for the training. The classifier was generated by random-forest supervised-learning.

When using the learning algorithm, the selection of not only features but also samples used for learning can have a large impact on the classification results. For example, if most fire samples are red areas, the red areas will be detected well. However, it becomes difficult to detect the yellow and white areas. When learning only areas with active motion, it becomes difficult to detect the areas with the small motion of a flame. Therefore, the fire samples used for learning should be selected so that the numbers are evenly distributed according to their colors and motions. In selecting the samples of a nonfire area, learning was performed by dividing into several areas as follows: areas similar in color to fires without motion, areas different in color from fires with active motion, and areas completely different in color from fires without motion. In such cases, it is desirable to select a relatively large number of nonfire samples, because many errors mistaking nonfire for fire occur when the number of fire samples is relatively large.

In the first experiment as shown in Fig. 11, Movies 1 to 5 were used to compare the fire detection algorithm proposed by Töreyin, Ko, and the method proposed in this paper. Additionally, Movies 6 to 11 were used to perform experiments in more varied environments. The description of the images can be found in Table 2. True positive means that a fire image is correctly determined as a fire, true negative means that a nonfire image is correctly determined as a nonfire, false positive means that a fire image is misjudged as a nonfire and false negative means a nonfire image is misjudged as a fire.

Fig. 11

Movie samples used for comparative experiments. (a) Movie 1. (b) Movie 2. (c) Movie 3. (d) Movie 4. (e) Movie 5.

Table 2

Descriptions of experimental movies.

Overall results of the first experiment are shown in Figs. 12 and 13. The method proposed in this study shows a higher detection rate than the other two methods. In this case, the detection rate is the sum of the values of true positive and true negative. In Movie 1, it was not possible to consider the motion information properly because the frame rate of the video was lower than other images. In Movie 3, the detection rate was lower than other images because the size of the fire area was smaller than other patches used in the experiment. Furthermore, in Movie 5, the false alarm shown in the method by Ko did not occur in the experiment with the proposed method.

Fig. 12

Images of fire detection results. (a) Movie 1. (b) Movie 2. (c) Movie 3. (d) Movie 4. (e) Movie 6. (f) Movie 7. (g) Movie 8. (h) Movie 9.

Fig. 13

Result of comparative experiments of Töreyin’s and Ko’s algorithms and detection rate (sum of $\mathrm{TP}+\mathrm{TN}$).

As depicted in Fig. 14, the second experiment was performed by using fires in tunnels, inside buildings and automobiles, with a blinking ambulance light, continuously passing automobile headlamps, and tail lights likely to be mistaken for fires. As shown in Fig. 15, the detection rate was an average of 96.45% and we could reduce the average of false negative rate from 2.1% to 1.31% when applying the additive Markov chain. However, as shown in Fig. 16(a) and 16(b), false positives were shown because the fire area was small at the moment when the fire was extinguished in Movie 6 and as the fire started in Movie 9. Depicted in Fig. 16(c) and 16(d), the false negatives occurred because the color, motion and crossing feature were similar to fires. Movie 9 demonstrated whether the classifier works well when exposing a bag, hat, and fire extinguisher similar in color to flames in the image. The results showed that it worked well even when moving the hat and bag. Although there were many areas that had values similar to the fire features defined above in Movies 10 and 11, the classifier could work well because each of the feature values were relatively small.

Fig. 14

Movies used for analysis of detection rate. (a) Movie 6. (b) Movie 7. (c) Movie 8. (d) Movie 9. (e) Movie 10. (f) Movie 11.

Fig. 15

Result of detection rate performed in various environments. (a) Before applying additive Markov chain. (b) After applying additive Markov chain.

Fig. 16

Cases of false positive: (a) Movie 6, (b) Movie 9. Cases of false negative: (c) Movie 9, (d) Movie 11.

It is possible to recognize variable importance in the learning process when using random-forest supervised-learning. Figure 17 shows that the importance of 22 features is evenly distributed. In this case, Features 5 and 7 are relatively high because the classifier frequently used the mean value of red and blue color channels during the training. The rate of features ($1\sim 10$) related to the fire color are 51.2%. It means that the color features are more effective than the motion and crossing features ($11\sim 22$). However, the motion and crossing features perform the role of the fire features very well. It works to prevent miss-classifications when similar colors, such as automobile headlamps and red hats, appear in scenes. In the color features, a contribution of the RGB channel values ($5\sim 7$) is higher than the color conditions ($1\sim 4$). In the crossing features ($11\sim 19$), crossing features for small threshold ($11\sim 13$) have more influences on detecting fires.

Fig. 17

Variable importance of learned random forest.

The algorithm could process approximately 30 frames of $320\times 240$ image in Core 2 Duo 2.00 Ghz at our embedded system.

5.

Conclusions

In this paper, a fire detection system with the built-in embedded processor, the main camera, and the PTZ camera was designed to perform fire surveillance in various environments. In addition, fire features were modeled by color, motion, and crossing features. This paper proposed an algorithm that could detect fires precisely by using the random forest classifier, making it possible to reduce the false alarms that occurred in previous studies resulting from the complex background. We used an additive Markov chain method for a robust fire detection algorithm. The performances of this method were verified by comparative experiments with other methods in various environments. The proposed fire detection system can detect fires with high reliability in public places, such as buildings, roads and tunnels. Also, it minimizes casualties and property damages by early fire detection. In addition, the automatic surveillance system can reduce facility maintenance costs by reducing the number of monitoring personnel and the fatigue level of the supervisor. Therefore, our system contributes to increased safety in public places because potential fires can be monitored 24 h a day.