2.1.

Overview of the Proposed Approach

Figure 1 shows an overview of the proposed method. First, we obtain an RGB eye image. Then, we normalize the $I$ and $K$ values to the range [0, 1] in order to use them as inputs to the fuzzy system. Next, we obtain the output image from the fuzzy system combining each pixel of the $I$ and $K$ images. Each pixel of the output image has a range from 0 to 1, and this image is converted to one where each pixel has the range from 0 to 255 by simple linear scaling.

Fig. 1

Flow chart of the proposed method.

The image is then binarized using a specific threshold and we execute component labeling to select the biggest eye blob area. Then we project black pixels in the vertical direction and calculate the standard deviation of the vertical lengths of the projected black pixels. Finally, the eye openness or closure is classified based on the standard deviation.

2.2.

Eye Image Preprocessing

2.2.2.

Obtaining I and K images from eye region

The colors of the human pupil and the eyelashes vary from light brown to black—this is usually darker than the color of the skin. To classify dark and bright pixels and detect eye openness and closure, we propose fuzzy-based segmentation. Two features are used as inputs to the fuzzy system. The first input is the intensity ($I$) of the hue saturation intensity (HSI) color space.²⁹ The second input is black ($K$) from the CMYK color space.³⁰ The two inputs are calculated by Eqs. (1) and (2), respectively:

Eq. (1)

$$I=\frac{1}{3}(R+G+B),$$

Eq. (2)

$$K=1-\mathrm{MAX}(R,G,B).$$

In Eqs. (1) and (2), $R$, $G$, and $B$ are the red, green, and blue values of an RGB pixel. The I value is obtained by averaging $R$, $G$, and $B$. The $K$ value is obtained by Eq. (2). The input values of a fuzzy system should range from 0 to 1, whereas the range of $I$ and $K$ is from 0 to 255. Therefore, we perform the normalization of $I$ and $K$ so as to make their range from 0 to 1. For that, we obtain the histograms of the $I$ and $K$ values, respectively, and model the histogram as a Gaussian distribution. Then we set the minimum and maximum boundaries of $I$ and $K$ for normalization based on a three-sigma range (99.7%)³¹ in order to exclude the noise value in the range for normalization. That is, we obtain $-3\sigma$ and $+3\sigma$ as the min and max bounds from the histogram, and these bounds are obtained for each eye image. If a pixel value of the $I$ and $K$ image is over the max bound, it is set to 1, and if the pixel value is under the min bound, it is 0. Equation (3) represents this method:

Eq. (3)

$$b(x,y)=\{\begin{array}{ll}0& \text{if}a(x,y)\le \mathrm{Min}\text{bound}\\ 1& \text{if}a(x,y)\ge \mathrm{Max}\text{bound}\\ \frac{[a(x,y)-\mathrm{Min}\mathrm{bound}]}{(\mathrm{Max}\mathrm{bound}-\mathrm{Min}\mathrm{bound})}& \text{else}\end{array},$$

where $a(x,y)$ is a pixel value from the $I$ or $K$ image, and $b(x,y)$ is the normalized pixel value. From Eq. (3), we obtain the two inputs whose ranges are from 0 to 1 for the fuzzy system. Figure 2 shows the examples of $I$ and $K$ images and their histograms.

Fig. 2

$I$ and $K$ images obtained from an RGB image, and their histograms. (a), (f), (k), and (p) RGB images. (b), (g), (l), and (q) $I$ images. (c), (h), (m), and (r) $K$ images. (d) Histogram of (b) (non-Gaussian shape). (e) Histogram of (c) (non-Gaussian shape). (i) Histogram of (g) (shape similar to Gaussian). (j) Histogram of (h) (shape similar to Gaussian). (n) Histogram of (l) (shape similar to Gaussian). (o) Histogram of (m) (shape similar to Gaussian). (s) Histogram of (q) (shape similar to Gaussian). (t) Histogram of (r) (shape similar to Gaussian).

As shown in Figs. 2(d) and 2(e), although the histograms of some images are a little different from the Gaussian shape, most image histograms are quite similar to the Gaussian shape, as shown in Figs. 2(i), 2(j), 2(n), 2(o), 2(s), and 2(t). Therefore, our normalization based on a Gaussian shape does not produce an error. In addition, without this method of setting the minimum and maximum boundaries for normalization based on the assumption of a Gaussian shape, it is difficult to use other theoretical methods to determine the minimum and maximum boundaries. Therefore, we use the normalization method based on the assumption of a Gaussian shape in our research.

2.3.

Fuzzy-Based Eye Image Segmentation Method

2.3.1.

Definition of fuzzy membership functions

In general, it is difficult to determine an optimal threshold for image segmentation. In this study, we propose a fuzzy-based eye image segmentation method. On the basis of the assumption that the pupil and eyelashes are usually darker than the skin, we use the $I$ and $K$ values as the two inputs to the fuzzy system. Each of these inputs is in the range [0, 1]. The output value from the fuzzy system also ranges from 0 to 1.

The fuzzy membership functions used in this study are shown in Fig. 3. Generally, membership functions are used to represent the distribution of input or output values in a fuzzy system. As shown in Fig. 3, the low ($L$), middle ($M$), and high ($H$) areas are usually designed as being overlapped. We use these membership functions and fuzzy rules to obtain an output value through the defuzzification method. Table 2 shows the fuzzy rule table used in this study.

Fig. 3

Illustrations showing the input and output fuzzy membership functions: (a) the input fuzzy membership function for $I$ value; (b) the input fuzzy membership function for $K$ value; (c) output fuzzy membership function.

Table 2

Fuzzy rule table for obtaining the output value of fuzzy system.

Input 1 (I value)	Input 2 (K value)	Output value
$L$	$L$	$M$
$L$	$M$	$M$
$L$	$H$	$H$
$M$	$L$	$M$
$M$	$M$	$M$
$M$	$H$	$H$
$H$	$L$	$L$
$H$	$M$	$L$
$H$	$H$	$L$

As shown in Fig. 2, the eye region is usually darker than other areas, whose $I$ and $K$ values are low and high, respectively. Therefore, if the values of the $I$ and $K$ pixels are shown as lower and higher ones, respectively, the possibility that the given pixel belongs to the eye region is higher ($H$). On the contrary, if the values of the $I$ and $K$ pixels are shown as higher and lower ones, respectively, the possibility is lower ($L$). Based on these, we design the fuzzy rule table as shown in Table 2. Therefore, when the $I$ and $K$ values of the pixel are low and high, respectively, the output value is close to 1. In contrast, if the $I$ and $K$ values of the pixel are high and low, respectively, the output value is close to 0. However, when the environmental illuminator is bright, the $I$ value of eye pixels could be brighter than normal. The effect of camera blurring can also cause the $I$ value of eye pixels to be brighter than normal. Hence, we design the fuzzy rule table where the $I$ value of eye pixels can be $L$ or $M$ with the $K$ value of $H$.

2.3.2.

Obtaining the output value of fuzzy system by defuzzification method

As shown in Fig. 4(a), three outputs are obtained as $f_1(L)$, $f_1(M)$, and $f_1(H)$, respectively, through three membership functions of $L$, $M$, and $H$ with input 1 ($I$ value). In addition, three outputs are also obtained as $f_2(L)$, $f_2(M)$, and $f_2(H)$, respectively, through three membership functions of $L$, $M$, and $H$ with input 2 ($K$ value). For example, if the input value of $I$ is 0.538, $f_1(L)$, $f_1(M)$, and $f_1(H)$ are 0.0, 0.924, and 0.076, respectively, as shown in Fig. 4(a). If the input value of $K$ is 0.429, $f_2(L)$, $f_2(M)$, and $f_2(H)$ are 0.143, 0.857, and 0.0, respectively, as shown in Fig. 4(b). With these two pairs of three outputs, we can obtain the nine combinations of output [($f_1(L)$, $f_2(L)$], [$f_1(L)$, $f_2(M)$], [$f_1(L)$, $f_2(H)$], [$f_1(M)$, $f_2(L)$], [$f_1(M)$, $f_2(M)$], [$f_1(M)$, $f_2(H)$], [$f_1(H)$, $f_2(L)$], [$f_1(H)$, $f_2(M)$], [$f_1(H)$, $f_2(H)$] as shown in Table 3.

Fig. 4

Illustrations of obtaining the output of membership function: (a) outputs of input 1 ($I$ value) and (b) outputs of input 2 ($K$ value).

Table 3

Illustration of nine combinations of output values and the determined IV by Min or Max rule.

Index of pairs	Output of f1(·)	Output of f2(·)	IV
			Min rule	Max rule
1	0.0 ($L$)	0.143 ($L$)	0.0 ($M$)	0.143 ($M$)
2	0.0 ($L$)	0.857 ($M$)	0.0 ($M$)	0.857 ($M$)
3	0.0 ($L$)	0.0 ($H$)	0.0 ($H$)	0.0 ($H$)
4	0.924 ($M$)	0.143 ($L$)	0.143 ($M$)	0.924 ($M$)
5	0.924 ($M$)	0.857 ($M$)	0.857 ($M$)	0.924 ($M$)
6	0.924 ($M$)	0.0 ($H$)	0.0 ($H$)	0.924 ($H$)
7	0.076 ($H$)	0.143 ($L$)	0.076 ($L$)	0.143 ($L$)
8	0.076 ($H$)	0.857 ($M$)	0.076 ($L$)	0.857 ($L$)
9	0.076 ($H$)	0.0 ($H$)	0.0 ($L$)	0.076 ($L$)

Then, based on the Min or Max rule and Table 2, we can obtain the output values.^32–34 For example, in the first row of Table 3, [$f_1(L)$, $f_2(L)$] are 0.0 ($L$) and 0.143 ($L$), respectively. We take 0.0 and 0.143, respectively, if we use the Min and Max rule.^32–34 According to the fuzzy rule of Table 2, $L$ and $L$ become $M$. Therefore, we can finally obtain 0.0 ($M$) and 0.143 ($M$), respectively, by the Min and Max rule, as shown in Table 3. For convenience, we call these values of 0.0 ($M$) and 0.143 ($M$) as IVs in our paper.^32–34 Like this method, we can obtain the nine IVs as shown in Table 3. Then, the final output score can be calculated using the defuzzification method.^32–34,35 Detail explanations of the defuzzification method are as follows with Fig. 5.

Fig. 5

Illustration of the defuzzification methods used in this study: (a) first of maxima (FOM), last of maxima (LOM), middle of maxima (MOM), and mean of maxima (MeOM) and (b) weighted average method.

Figure 5 illustrates a number of defuzzification operators. In this study, we consider five defuzzification operators as follows.^32–34 Figure 5(a) shows the first of maxima (FOM) operator, which is represented as the smallest output score ($s_2$) obtained by the biggest IVs [$\mathrm{IV}(M)$ and $\mathrm{IV}(H)$]. The last of maxima (LOM) returns the biggest output score ($s_4$) obtained by the biggest IVs [$\mathrm{IV}(M)$ and $\mathrm{IV}(H)$]. To obtain a result using the middle of maxima (MOM) operator, we take the average of the output scores by FOM and LOM as [$(s_2+s_4)/2$]. The mean of maxima (MeOM) operator is the average of all the output scores by the biggest IVs [$\mathrm{IV}(M)$ and $\mathrm{IV}(H)$] as [$(s_2+s_3+s_4)/3$]. In the original weighted average method, the output score ($s_5$) is calculated by the weighted average method with the rectangles of $R_1$, $R_2$, and $R_3$ of Fig. 5(b).³⁵ Although the nine IVs are obtained by the Min or Max rule as shown in Table 3, the original weighted average method considers the rectangular regions defined only by the maximum IVs of each membership function [$\mathrm{IV}(M)$, $\mathrm{IV}(H)$, $\mathrm{IV}(L)$ of Fig. 5(b)], which cannot reflect the effect of other IVs. Therefore, we use RWAM, where all the rectangular regions by all the IVs are considered for calculating the output score.

The output scores given by defuzzification range from 0 to 1, and we rescale them from 0 to 255 by multiplying by 255. From that, we can obtain the eye image by the defuzzification method. As shown in Fig. 6, different images are produced depending on the selected defuzzification operator, Min or Max rule.

Fig. 6

Comparisons of images by fuzzy system: (a) original image; (b) by Min rule with FOM; (c) by Min rule with LOM; (d) by Min rule with MOM; (e) by Min rule with MeOM; (f) by Min rule with revised weighted average method (RWAM); (g) by Max rule with FOM; (h) by Max rule with LOM; (i) by Max rule with MOM; (j) by Max rule with MeOM; (k) by Max rule with RWAM.

Examining the images in Fig. 6, we can find certain characteristics. There are the images where the eye regions can be more separable from other areas than the original image, like Figs. 6(b), 6(d), and 6(e). Therefore, in order to classify eye openness and closure, we transform the output image by the fuzzy system of Fig. 6 into the binarized one as shown in Fig. 7. We compared the performances of various binarization methods as shown in Tables 4 and 5 with Figs. 10 and 12.

Fig. 7

Results of postprocessing step in the case of open eye: (a) by Min rule with FOM; (b) binarized image of (a); (c) result by component labeling and size filtering of (b); (d) vertical histogram of (c).

Table 4

Classification results table of eye openness and closure using database I (unit: %) (The smallest equal error rate is shown in bold type).

No.	Fuzzy system applied	Min/max rule	Defuzzification method	Binarization threshold is obtained by	Type 1 error	Type 2 error	Equal error rate
1	×			Gonzalez method	24.30	24.20	24.25
2				Otsu method	35.20	35.60	35.40
3				Static threshold	9.10	9.20	9.15
4	O	Min rule	FOM	Gonzalez method	26.80	26.60	26.70
5			LOM		49.20	27.30	38.25
6			MOM		6.20	6.10	6.15
7			MeOM		6.20	6.10	6.15
8			RWAM		33.20	33.20	33.20
9		Max rule	FOM		41.40	41.40	41.40
10			LOM		35.90	35.70	35.80
11			MOM		41.60	41.60	41.60
12			MeOM		38.30	38.20	38.25
13			RWAM		78.80	22.90	50.85
14		Min rule	FOM	Otsu method	14.90	14.90	14.90
15			LOM		22.00	21.90	21.95
16			MOM		7.50	7.40	7.45
17			MeOM		7.50	7.40	7.45
18			RWAM		39.00	39.00	39.00
19		Max rule	FOM		50.00	49.80	49.90
20			LOM		42.40	42.40	42.40
21			MOM		46.30	46.30	46.30
22			MeOM		42.80	42.50	42.65
23			RWAM		34.70	34.60	34.65
24		Min rule	FOM	Static threshold	2.70	2.80	2.75
25			LOM		41.00	41.20	41.10
26			MOM		2.70	2.80	2.75
27			MeOM		2.70	2.80	2.75
28			RWAM		3.60	3.70	3.65
29		Max rule	FOM		52.10	52.20	52.15
30			LOM		43.80	43.60	43.70
31			MOM		52.30	52.10	52.20
32			MeOM		32.20	31.90	32.05
33			RWAM		5.50	5.50	5.50

FOM, first of maxima; LOM, last of maxima; MOM, middle of maxima; MeOM, mean of maxima; RWAM, revised weighted average method.

Table 5

Classification results for eye openness and closure using database II (unit: %) (The smallest equal error rate is shown in bold type).

No.	Fuzzy system applied	Min/max rule	Defuzzification method	Binarization threshold is obtained by	Type 1 error	Type 2 error	Equal error rate
1	×			Gonzalez method	19.80	19.60	19.70
2				Otsu method	53.30	53.20	53.25
3				Static threshold	31.50	31.80	31.65
4	O	Min rule	FOM	Gonzalez method	38.60	38.60	38.60
5			LOM		60.00	34.00	47.00
6			MOM		14.80	14.40	14.60
7			MeOM		14.60	14.80	14.70
8			RWAM		54.90	54.60	54.75
9		Max rule	FOM		51.00	50.90	50.95
10			LOM		47.90	47.80	47.85
11			MOM		50.40	50.70	50.55
12			MeOM		48.20	46.60	47.40
13			RWAM		73.60	24.90	49.25
14		Min rule	FOM	Otsu method	10.20	10.50	10.35
15			LOM		41.90	41.90	41.90
16			MOM		13.00	13.20	13.10
17			MeOM		13.00	13.20	13.10
18			RWAM		49.80	49.70	49.75
19		Max rule	FOM		45.50	45.40	45.45
20			LOM		46.40	46.40	46.40
21			MOM		47.00	47.00	47.00
22			MeOM		45.60	45.60	45.60
23			RWAM		53.20	53.20	53.20
24		Min rule	FOM	Static threshold	10.40	10.50	10.45
25			LOM		51.70	51.80	51.75
26			MOM		10.40	10.50	10.45
27			MeOM		10.40	10.50	10.45
28			RWAM		15.70	15.50	15.60
29		Max rule	FOM		45.50	45.40	45.45
30			LOM		53.40	53.60	53.50
31			MOM		45.80	46.00	45.90
32			MeOM		40.40	40.40	40.40
33			RWAM		11.60	11.50	11.55

Fig. 8

Results of postprocessing step in the case of closed eye: (a) by Min rule with FOM; (b) binarized image of (a); (c) result by component labeling and size filtering of (b); (d) vertical histogram of (c).

Fig. 9

Experimental setup and collected images: (a) experimental setup; (b) collected images (left image is the original one captured at the Z distance of 2 m, and right three images are the cropped face ones from the left image); (c) collected images (left image is the original one captured at the Z distance of 2.5 m, and right three images are the cropped face ones from the left image).

2.4.

Classifying Eye Openness and Closure from a Binary Eye Image

With the binarized image, we perform component labeling. Figure 7(a) shows the image resulting from the Min rule with FOM. The image is easy to separate into regions of eye and skin. Figure 7(b) is a binarized one of Fig. 7(a) by static threshold. As shown in Fig. 7(c), we eliminate small noise areas using component labeling. Next, we obtain the biggest area from the binarized blobs as the eye region.

With the binarized image of Fig. 7(c), we can obtain the vertical histogram as shown in Fig. 7(d). As shown in Fig. 7(d), the mid area of the histogram shows a higher value, whereas the side ones show a lower value in the case of an open eye. However, both the mid and side areas of the histogram show the low value in the case of a closed eye as shown in Fig. 8.

Based on these, we use the standard deviation of the histogram (the length of the black pixels in the vertical direction) as the features for classifying open and closed eyes. If the standard deviation is above a specific threshold, we determine the state of the eye as openness. Otherwise, the eye is assumed to be closed.

3.1.

Experimental Results with Database I

To experimentally verify our classification method for eye openness and closure, we collected 6336 open eye images and 6294 closed eye images (database I). We captured the images from a distance of 2 to 2.5 m in an indoor environment, where each person watched TV at a distance. The images of database I were obtained using a Logitech C600 web camera equipped with a zoom lens, and the image resolution is $1600\times 1200\text{pixels}$.³⁶ The camera is positioned below the TV. Figure 9 shows the experimental setup and the examples of collected images.

To measure the accuracy of the classification of eye openness and closure, we conducted two experiments without and with our fuzzy-based method. As explained in Sec. 2.4, the image is binarized after the fuzzy-based fusion of $I$ and $K$ images in our research. Therefore, we compared three binarization methods: Gonzalez algorithm,³⁷ Otsu algorithm,³⁸ and static threshold²⁹ in the same image by the fuzzy-based method for fair comparisons. Table 4 lists the complete set of results.

We compared the equal error rate (EER) of all methods. The EER is calculated as the error rate at the moment when the type 1 error is most similar to the type 2 error. The type 1 error indicates the error rate of open eye images being incorrectly determined as closed eye images. The type 2 error denotes the error rate of closed eye images being incorrectly determined as open eye images. In the cases of the Min rule with LOM using Gonzalez method and the Max rule with RWAM using Gonzalez method, there exists a large difference between the EER and type 1 (type 2) error. That is because the type 1 and 2 error cases do not occur continuously according to the change of threshold for discriminating open and closed eyes.

As the results show, the lowest EER values were obtained by the Min rule with FOM, Min rule with MOM, and Min rule with MeOM using a static threshold through the fuzzy system. From these, we can confirm that the proposed method is superior to others.

Figure 10 shows the receiver operating characteristic (ROC) curves for the 10 highest ranked EER results obtained from Table 4. As shown in Fig. 10, the Min rule with FOM and static threshold, Min rule with MOM and static threshold, and Min rule with MeOM and static threshold outperformed the other methods. In case of the images by the Min rule with FOM, Min rule with MOM, and Min rule with MeOM, the same images are obtained through the binarization with static threshold, and consequent ROC curves for classifying the eye openness and closure are same as shown in Fig. 10.

Fig. 10

Receiver operating characteristic (ROC) curves for the 10 highest ranked equal error rate (EER) results from Table 4.

3.2.

Experimental Results with Database II

In order to measure the effect of the kind of database on the performance of our method, we conducted a further experiment on the classification of eye openness and closure using an open database (ZJU Eyeblink Database).³⁹ This database has 80 video clips with a resolution of $320\times 240\text{pixels}$. We used 20 of these video clips excluding the clips of wearing glasses and the images where eye detection failed. The resulting image set contained a total of 5376 images, with 4891 open eye images and 485 closed eye images (database II). Figure 11 shows the examples of images from open database.

Fig. 11

Examples of images from ZJU Eyeblink Database.

Table 5 presents the EER results obtained from database II. The EER results of the methods using our fuzzy system are better than those given by other methods.

In the cases of the Min rule with LOM using the Gonzalez method and the Max rule with RWAM using the Gonzalez method, there exists a large difference between the EER and type 1 (type 2) error. That is because the type 1 and 2 error cases do not occur continuously according to the change of threshold for discriminating open and closed eyes.

Figure 12 shows the ROC curves for the 10 highest ranked EER results obtained from Table 5. Although the EER by the Min rule with FOM and Otsu method is lowest as shown in Table 5, Fig. 12 shows that the overall accuracies by the Min rule with FOM and static threshold, Min rule with MOM and static threshold, and Min rule with MeOM and static threshold are highest in terms of the ROC curves such as the results of Fig. 10. From Fig. 12, we can confirm that the proposed fuzzy-based method outperformed other methods. In case of the images by the Min rule with FOM, Min rule with MOM, and Min rule with MeOM, the same images are obtained through the binarization with a static threshold, and the consequent ROC curves for classifying the eye openness and closure are same as shown in Fig. 12.

Fig. 12

ROC curves for the 10 highest ranked EER results from Table 5.

3.3.

Experimental Results Analysis

Figure 13 shows the images that resulted in good classification of eye openness and closure with database I. As shown in the images of Fig. 13(d), the result images of open eye are shown as discriminated from those of closed eye, and the consequent feature values [standard deviations of histogram (the lengths of the black pixels in the vertical direction)] of the open eye are larger than those of the closed eye. Therefore, the open eye can be discriminated from the closed eye.

Fig. 13

Images of good classification results (database I): (a) original images, (b) images by Min rule with FOM and static threshold, (c) binarized images of (b), (d) component labeling results of (c). (The left three images of each row are open eyes, whereas the right three ones are closed eyes).

Figure 14 shows the images that resulted in bad classification of eye openness and closure with database I. In the case of open eye images, bad classification occurred when the eye image is too blurred [the first and second images of the left three ones of Fig. 14(a)], or when reflections exist in the eyeball [the third image of the left three ones of Fig. 14(a)].

Fig. 14

Images of bad classification results (database I): (a) original images, (b) images by Min rule with FOM and static threshold, (c) binary images of (b), (d) component labeling results of (c). (The left three images of each row are open eyes, whereas the right three ones are closed eyes).

In the case of closed eye images, bad classification was due to image blurring [the first image of the right three ones of Fig. 14(a)], incorrect detection of eye region [the second image of the right three ones of Fig. 14(a)], or the incorrect selection of eyebrow by component labeling [the third image of the right three ones of Fig. 14(a)].

Figure 15 shows the images that resulted in good classification of eye openness and closure with database II. As shown in the images of Fig. 15(d), the result images of the open eye are shown as discriminated from those of the closed eye, and consequent feature values [standard deviations of histogram (the lengths of the black pixels in the vertical direction)] of the open eye are larger than those of the closed eye. Therefore, the open eye can be discriminated from the closed eye.

Fig. 15

Images of good classification results (database II): (a) original images, (b) images by Min rule with FOM and static threshold, (c) binarized images of (b), (d) component labeling results of (c). (The left three images of each row are open eyes, whereas the right three ones are closed eyes).

Figure 16 shows the images that resulted in bad classification of eye openness and closure with database II. In the case of open eye images, bad classification occurred when eyelid pixels were disconnected by image blurring [the first image of the left three ones of Fig. 16(a)], the eye is not widely opened and the eyebrow is incorrectly selected by component labeling [the second image of the left three ones of Fig. 16(a)], or the eye image is too dark [the third image of the left three ones of Fig. 16(a)].

Fig. 16

Images of bad classification results (database II): (a) original images, (b) images by Min rule with FOM and static threshold, (c) binary images of (b), (d) component labeling results of (c). (The left three images of each row are open eyes, whereas the right three ones are closed eyes).

In the case of closed eye images, bad classification happens when the eye is not completely closed [the first image of the right three ones of Fig. 16(a)], the eye region is not correctly detected [the second image of the right three ones of Fig. 16(a)], or the eye image is too dark and the eyebrow is incorrectly selected by component labeling [the third image of the right three ones of Fig. 16(a)].

It is usually difficult to evaluate the accuracy of eye segmentation because all the pixels of an accurate eye region should be manually obtained as the ground-truth data. Therefore, in our research, we measured good and bad responses of Figs. 13–16 based on the error of classifying eye openness and closure in Tables 4 and 5. That is, the good response means that the open or closed eye is correctly classified as an open or closed one, respectively, by our method. The bad response means that the open or closed eye is incorrectly classified as a closed or open one, respectively, by our method.

As shown in Eqs. (1) and (2), $I$ is obtained by averaging $R$, $G$, and $B$, whereas $K$ is obtained by selecting the maximum value among $R$, $G$, and $B$, and subtracting it from 1. For example, in the case of a gray pixel of medium level, we can assume that $R$, $G$, and $B$ are 1, 0.5, and 0, respectively, if the range of $R$, $G$, and $B$ is from 0 to 1, respectively. From that, $I$ of Eq. (1) is 0.5 ($1.5/3$), whereas $K$ of Eq. (2) is 0. In the case of white ($R=G=B=1$), I is 1 ($3/3$), whereas $K$ is 0. Furthermore, in the case of black ($R=G=B=0$), $I$ is 0 ($0/3$), whereas $K$ is 1.

By comparing these three cases, the gray pixel of the medium level ($R=1$, $G=0.5$, $B=0$) is represented as the white in $K$ value ($K=0$). whereas it is the gray pixel of the medium level in the $I$ value ($I=0.5$). The gray level of the surrounding skin of an eye can be regarded as the gray pixel of the medium level because its gray level is lower than the bright sclera and higher than the dark eyeball as shown in Fig. 17(a). Therefore, $K$ has the effect of making the gray pixel of the surrounding skin of eye close to white while maintaining the level of the dark eyeball. Consequently, the contrast between the surrounding skin of an eye and the dark eyeball in the $K$ image is increased more than that in the $I$ image, which can enhance the accuracy of segmenting the eyeball from the area of surrounding skin of the eye.

Fig. 17

Comparisons of binarization with $I$, $K$, and the image from our fuzzy-based method in the case of a large number of eyelashes in the original image: (a) original images of open and closed eyes, (b) an open eye, and (c) a closed eye.

However, the $K$ image has the disadvantage of making the gray pixel of the eyelid, eyelashes, and shadows (whose gray levels are also higher than the dark eyeball) close to white, which can cause the eyelid line to be erroneously segmented from the surrounding skin of the eye.

Therefore, we combine the $I$ and $K$ images using a fuzzy method, which enhances the advantages of both $I$ (less affected by eyelid, eyelashes, and shadows) and $K$ (enhancing the contrast between the eyeball and surrounding skin of eye). Thus, we can improve the final accuracy of the eye segmentation and the determination of whether the eye is open or closed, which is less affected by eyelashes and shadows based on fuzzy-based combining method.

We included experiments to show that these claims are correct as follows. Figures 17 and 18 show the comparisons of binarization with $I$, $K$, and the image from our fuzzy-based method in the case of a large number of eyelashes and shadows in the original image, respectively. We, thus, demonstrate that a more accurate binarized image of the eye region can be obtained by our fuzzy-based method (Min rule FOM) than by the $I$ or $K$ images.

Fig. 18

Comparisons of binarization with $I$, $K$, and the image from our fuzzy-based method when the shadows around the eye are included in the original image, for the cases of (a) an open eye, and (b) and (c) a closed eye.

In addition, as shown in Table 4 and Fig. 10 (database I), the average EER of the classification of eye openness and closure by our fuzzy-based combination method is $\sim 2.75\%$, which is much smaller than that obtained when not combining the $I$ and $K$ images (9.15%). In addition, as shown in Table 5 and Fig. 12 (database II), the average EER of the classification of eye openness and closure by our fuzzy-based combination method is $\sim 10.35\%$, which is also much smaller than that obtained when not combining the $I$ and $K$ images (19.70%). Thus, we found that our fuzzy-based combination method outperforms that using either the $I$ or $K$ image without combining them.

We included explanations and experiments for other races, especially African-Americans. Experiments were performed with 208 images of two African-Americans. As shown in Fig. 19, we found that a more accurate binarized image of the eye region can be obtained by our fuzzy-based method (Min rule FOM) than from the $I$ or $K$ images. In addition, the average EER of the classification of eye openness and closure by our fuzzy-based combination method is $\sim 2.8\%$, which is similar to that of Table 4 using database I. Therefore, we concluded that our fuzzy-based combination method is robust to images of other races.

Fig. 19

Comparisons of binarization with $I$, $K$, and the image from our fuzzy-based method in the case of African-Americans: (a) original captured image, and the cases of (b) and (c) an open eye, and (d) and (e) a closed eye.

We included the explanations and experiments in the case of pose variations (head rotation). Experiments were performed with 213 images of pose variations. As shown in Fig. 20, we found that a more accurate binarized image of the eye region can be obtained by our fuzzy-based method (Min rule FOM) than from the $I$ or $K$ images. In addition, the average EER of the classification of eye openness and closure by our fuzzy-based combination method is $\sim 2.9\%$, which is similar to that of Table 4 using database I. Thus, we found that our fuzzy-based combination method is robust to images of varying poses.

Fig. 20

Comparisons of binarization with $I$, $K$, and the image from our fuzzy-based method in the case of pose variations (head rotation): (a) original captured image, (b) an open eye, and (c) a closed eye.

We included explanations and experiments in the case of users wearing glasses. Experiments were performed with 304 images of users wearing glasses. As shown in Fig. 21, a more accurate binarized image of the eye region can be obtained by our fuzzy-based method (Min rule FOM) than from the $I$ or $K$ images. In addition, the average EER of the classification of eye openness and closure by our fuzzy-based combination method is $\sim 3\%$, which is similar to that of Table 4 using database I. Therefore, we found that our fuzzy-based combination method is robust to images of users with glasses.

Fig. 21

Comparisons of binarization with $I$, $K$, and an image by our fuzzy-based method in the case of users wearing glasses: (a) shows an open eye, and (b) and (c) show a closed eye.

To obtain the eye region from the input image, we first detect the face region in the input image. We used the widely used AdaBoost method for face detection.²⁵ AdaBoost is used to detect the ROI of the eye within a face. Rather than performing additional training procedures for the AdaBoost method with our own database, we used the AdaBoost algorithm provided from the OpenCV library (version 2.3.1), which was already trained.⁴⁰

As shown in Fig. 23, the eyes are so small that detection errors and processing time increase if the AdaBoost method is used to directly detect the eye regions from the entire image. We compared the results of eye detection by our method (eye is detected within the ROI of the eye in a detected face region), as shown in Fig. 22, and by the method where the eye is located in the entire image without face detection, as shown in Fig. 23. As shown in Fig. 23, the error cases of eye region incorrectly detected occur, whereas there is no error in Fig. 22.

Fig. 22

Examples of correct face and eye detection using our method.

Fig. 23

Examples of incorrect eye detection in the entire image without face detection.

We also compared the processing time of eye detection using our method and for the method where the eye is located in the entire image without face detection. Experimental results showed that the processing time for eye detection in the latter method was 1.102 s, which is much longer than in our method. [The processing time including face (58.67 ms) and eye (12.70 ms) detection as $\sim 71.37\mathrm{ms}$, as shown in Table 6.) Therefore, we performed eye detection within the ROI of a face.

Table 6

Processing time for each step of the proposed method per image (unit: ms).

We measured the accuracies of face and eye detection using our method. The accuracies are measured based on Eqs. (4) and (5):

In our research, although we use conventional fuzzy membership functions and defuzzification methods (FOM, LOM, MOM, and MeOM), we newly propose the fuzzy rule table shown in Table 2, which reflects the characteristics of $I$ and $K$ values for accurate eye segmentation. In addition, we newly propose RWAM (where all rectangular regions from all IVs are considered for calculating the output score, rather than maximum IVs) as the defuzzification method and compare the performances. Figures 3(a) and 3(b) show the input fuzzy membership functions for $I$ and $K$ values, respectively. Figure 3(c) represents the output fuzzy membership functions. In all cases, we used a simple linear (triangular) function. In previous studies, these membership functions were defined by the heuristic experiences of the researcher (not by experiments). We used the linear (triangular) membership function because it has been widely used in fuzzy-based methods^41–43 to consider the processing time and complexity of the problem to be solved. By determining the membership functions and fuzzy rule table heuristically (not experimentally), the conventional fuzzy-based method has the advantages of not requiring additional training procedures, which take considerable processing time, and being less affected by the type of training database.

In our research, we used only three thresholds (parameters). The first is the threshold for determining whether the eye detection was successful by sub-block-based template matching or adaptive template matching. The second is the threshold for binarization of the static threshold method of Sec. 2.4 and Table 4. The last is the threshold for determining eye openness and closure based on the standard deviation of Sec. 2.4. All of these thresholds were empirically determined by test and trial. The recall and precision of eye detection by our method were 99.8 and 99.5%, respectively, from which we conclude that the first threshold is appropriate. As shown in Table 4, the static threshold method using the second threshold outperforms the binarization of the Otsu and Gonzalez methods, from which we conclude that the second threshold is appropriate. As shown in Tables 4 and 5, our fuzzy-based method of determining eye openness and closure outperforms the other methods, from which we conclude that the third threshold is appropriate.

In our research, we performed experiments with two databases (databases I and II) in order to measure the performance of our method in various database environments (image resolution, Z distance between camera and user, etc.). As shown in Figs. 9(b), 9(c), and 22, although the image resolution of database I is as large as $1600\times 1200\text{pixels}$, the image resolution of the eye region is small, as small as $25\times 12\text{pixels}$, because the Z distance between the camera and the user was large when collecting this database. As shown in Fig. 11, the image resolution of database II is $320\times 240\text{pixels}$, and the image resolution of the eye region is also as small as $25\times 12\text{pixels}$. Although the image resolution of database II is smaller than that of database I, the resolutions of the eye region of databases I and II are similar because the Z distance between the camera and user is much larger in database I.

Most widely used face databases include eye regions with higher image resolution. In the case of the PAL database,⁴⁴ the eye region is larger than $60\times 30$ or $90\times 45\text{pixels}$. In the case of the AR database,⁴⁵ the eye region is larger than $\sim 55\times 25\text{pixels}$. Therefore, the image resolution of the eye regions of the databases in our experiments is lower than that of other face databases.^44,45

As shown in Table 4, the error of determining eye openness and closure is $\sim 2.75\%$ by our method. This means that about two or three frames per 100 successive frames of valid open eyes are incorrectly determined as closed eyes and skipped. In addition, about two or three frames among 100 successive frames of closed eyes are incorrectly determined as open eyes.

3.4.

Processing Time of the Proposed Method

To measure the processing time of the proposed method, we utilized a desktop computer with an Intel Core i7 processor 975 at 3.33GHz and 8 GB RAM. Table 6 lists the measurement results of the processing time with database I according to each step of Fig. 1. For measuring the processing time, the Min rule with FOM and static threshold is used because it gives the best performance on the experiment results. As a result, the average processing time of each image was $\sim 89.062\mathrm{ms}$; we, thus, confirm that our method can be operated at a fast speed [$11.2\text{frames}/\mathrm{s}$ ($1000/89.062$)].