2.1.

Draft Mark

Figure 1 shows two typical types of draft marks, representing 3.6, 3.8, and 4.0 m. The size of each draft mark is 10-cm high and 2-cm wide.⁷ Type 1 uses “M” to represent meter, while type 2 uses only numbers. The bottom of each draft mark indicates the draft. For example, if a waterline touches the bottom of “4M,” the draft is 4.0 m.

Fig. 1

Two typical types of draft marks: (a) type 1 and (b) type 2.

2.2.

Draft Mark Segmentation

The conventional draft mark segmentation method segments draft marks by Otsu’s binarization method. It may be effective for images consisting only of draft marks and a hull, because the grayscale histogram of the image becomes bimodal. However, almost all observed images include other regions than draft marks and the hull, such as sea surface and shadows.

Figures 2(a) and 2(b) show the observed image and its grayscale histogram, respectively. We captured the image from a wharf using a hand-held camera for adding camera shake effect, because surveyors often read drafts on a pitching and rolling boat. We can verify that the histogram is not bimodal due to the influence of the sea surface and shadows. Figure 2(c) shows the binary image obtained by Otsu’s binarization method, where the threshold value is 113. We see that the draft mark “2” above “2M” is not separated from the hull.

Fig. 2

Conventional draft mark segmentation. (a) Original image, (b) grayscale histogram, and (c) binary image.

2.3.

Waterline Estimation

The conventional waterline estimation detects a moving sea surface by a frame differencing technique. More concretely, it takes the difference of two consecutive frames, binarizes the difference image with threshold $\theta$, and then accumulates the binary difference images during $n$ frames. Then the conventional method binarizes the horizontal mean of the accumulated binary image to segment the sea surface and regards the upper edge of the segmented sea surface as the waterline. However, if the sea surface moves slowly, we cannot accurately detect the waterline from the frame difference. Moreover, the estimated waterline tends to be the highest during the observation time. This misestimation is caused by accumulation of binary difference images.

Figure 3 shows the result of the waterline estimation, where we set $\theta =60$ and $n=600$. Figure 3(a) is the accumulation of binary difference images of the video sequence captured in a towing tank. The white region in Fig. 3(b) denotes the segmented sea surface. Figure 3(c) shows the estimated waterline superimposed on a photograph taken in the absence of a wave. We see that the estimated waterline is higher than the true one.

Fig. 3

Conventional waterline estimation. (a) Accumulation of binary difference images, (b) segmented sea surface, and (c) estimated waterline.

3.1.

Draft Mark Segmentation

The procedure of draft mark segmentation is further divided into three steps: draft mark detection, local thresholding, and removing noise segments.

3.1.1.

Draft mark detection

Top-hat transform⁸ is one of the morphological operations for detecting white or black objects that are smaller than a structuring element. We use the top-hat transform to detect draft marks since they are thin white or black objects. The white top-hat transform for detecting white draft marks is defined by Eq. (1), and the black top-hat transform for detecting black draft marks is defined by Eq. (2):

Eq. (1)

$$\{\begin{array}{l}{I}_{\text{erosion}}(x,y)=\underset{\sqrt{m^2+n^2}\le p}{\min }I(x+m,y+n),\\ I_{\text{opening}}(x,y)=\underset{\sqrt{m^2+n^2}\le p}{\max }{I}_{\text{erosion}}(x+m,y+n),\\ I_{\text{white}}(x,y)=I(x,y)-I_{\text{opening}}(x,y),\end{array}$$

Eq. (2)

$$\{\begin{array}{l}{I}_{\text{dilation}}(x,y)=\underset{\sqrt{m^2+n^2}\le p}{\max }I(x+m,y+n),\\ I_{\text{closing}}(x,y)=\underset{\sqrt{m^2+n^2}\le p}{\min }{I}_{\text{dilation}}(x+m,y+n),\\ I_{\text{black}}(x,y)=I_{\text{closing}}(x,y)-I(x,y),\end{array}$$

where we use a circular structuring element of a radius $p$ in pixels, $I(x,y)$ is an input image, $I_{\text{white}}(x,y)$ is the white top-hat image, and $I_{\text{black}}(x,y)$ is the black top-hat image. The radius $p$ must be chosen to be larger than the stroke width of draft marks, i.e., 2 cm. The image $I_{\text{opening}}(x,y)$ is the result of the opening operation, which has the effect of filling small and thin white objects. The white top-hat image is the difference between the input image and its opening image. The image $I_{\text{closing}}(x,y)$ is the result of the closing operation, which has the effect of filling small and thin black objects. The black top-hat image is the difference between the input image and its closing image. Since some ships may have both white and black draft marks, we use both the white and black top-hat transforms to detect draft marks.

Throughout this paper, we set $p=Y/30$, where $Y$ is the height of the input image in pixels. When we choose the input image so that more than two draft marks are included, $Y$ is equivalent to more than 30 cm, and $Y/30$ is equivalent to more than 1 cm. For this reason, we can make the radius of the circular structuring element larger than the stroke width of draft marks by putting $p=Y/30$.

We have applied the draft mark detection method to the image of Fig. 2(a). Figure 5 shows the result of the draft mark detection. We show only the white top-hat image, because there are only white draft marks in Fig. 2(a). We see that draft marks are roughly segmented, but they are corrupted by noise. We therefore segment the draft marks clearly by local thresholding in Sec. 3.1.2.

Fig. 5

Result of the draft mark detection.

3.1.2.

Local thresholding

The assumption that the whole of an observed image can be classified into only two classes, i.e., draft marks and the ship’s hull, is not exactly true in practical applications, while local images around draft marks can be roughly classified into the two classes. We thus introduce a local thresholding for draft mark segmentation. More concretely, we select bounding boxes of the segments in the resulting image of the draft mark detection, and then binarize each of the local images enclosed by the bounding box with Otsu’s binarization method. The edges of the bounding box are parallel to the coordinate axes and pass through the topmost, bottommost, rightmost, and leftmost points of the segment.

Figure 6(a) shows the bounding boxes of the segments obtained in Fig. 5. Figure 6(b) shows the result of the local thresholding. We see that draft marks are clearly segmented, but also that there are still noise segments, such as under water draft marks.

Fig. 6

Local thresholding: (a) bounding boxes of the segments shown in Fig. 5 and (b) result of the local thresholding.

3.1.3.

Removing noise segments

We judge whether segments in the resulting image of the local thresholding are noise or not by checking if the segment satisfies the following inequalities:

Eq. (3)

$$T_1\le w/h\le T_2,$$

Eq. (4)

$$T_3\le s,$$

where $h$ and $w$ are the height and width of the segment in pixels, respectively, and $w/h$ is a ratio of sides. The ratio of sides of draft mark “1” is 0.2, which is the smallest of all draft marks. We thus set $T_1=0.1$. We set $T_2=1$, considering that the height of draft marks is longer than the width in almost all cases, and the height of underwater draft marks is contracted by refraction. Meanwhile, $s$ is an area of the segment. We set $T_3=10$, considering that too small segments are obviously noise.

We have used the judging method to remove noise segments, as shown in Fig. 6(b). Figure 7 shows the result. We see that almost all noise segments are well removed.

Fig. 7

Result of removing noise segments.

3.2.

Draft Mark Recognition

Figure 8 shows the flowchart of the proposed draft mark recognition. We shall call a string of draft marks as the draft mark string. In Fig. 7, “2M” consisting of the two draft marks “2” and “M” is the draft mark string. Recognition of the draft mark string is indispensable for draft reading. We thus distinguish the draft mark string and the single draft mark according to character connectivity⁹ and the predetermined size of the draft mark. More concretely, we distinguish them based on the following rules:

Fig. 8

Flowchart of the draft mark recognition.

Although plural white or black draft mark strings may be extracted by using the above rules, only the lowest draft mark string is used for draft reading. Draft marks in the lowest draft mark string are recognized by template matching using sum of squared differences. The matching templates are the images “0” to “9” and “M,” each of which height and stroke width are 10 and 2 cm, respectively. Only when the rightmost draft mark in the lowest draft mark string is “M,” draft marks “8,” “6,” “4,” and “2” are recursively searched below until no draft mark is found or “2” is found, according to the following rules:

3.3.

Waterline Extraction

Canny edge detection⁶ is one of the most widely used edge detection algorithms because of its sensitivity and high signal-to-noise ratio. We use the Canny edge detection to detect the waterline. However, the straightforward application also detects noisy edges, such as scars and projections on the hull. Noticing the property of the noisy edges, the edges on the hull remain stationary relative to draft marks. We extract the stationary edges by taking the logical conjunction of neighbor frames as follows:

Figure 9 shows the result of Canny edge detection applied to the image of Fig. 2(a), where the thresholds of Canny edge detection are 15 and 30. Figure 10 shows the result of waterline extraction obtained by removing noisy edges. We shall call white pixels corresponding to a waterline as waterline pixels. Figure 10 contains waterline pixels, but it still contains noisy pixels, such as edges in the sea surface. We thus use the least median of squares (LMedS) method¹⁰ robust against noisy points to estimate the waterline.

Fig. 9

Result of Canny edge detection.

Fig. 10

Result of waterline extraction.

3.4.

Waterline Estimation

We set a search region in the resulting image of the waterline extraction, and then we use the LMedS method to fit a straight line to a set of waterline pixels in the search region. Figure 11 shows the search region. We search for white pixels downward for getting waterline pixels in the search region. The region below the lowest draft mark is not searched so as not to misrecognize the draft mark’s edge as waterline pixels.

Fig. 11

Search region for waterline pixels.

We have applied the waterline estimation to the resulting image of the waterline extraction, as shown in Fig. 10. Figure 12 shows the detected waterline pixels, and Fig. 13 shows the estimated waterline superimposed on a photograph. We see that the estimated waterline agrees with the true one.

Fig. 12

Waterline pixels.

Fig. 13

Estimated waterline.

3.5.

Draft Calculation

We estimate the waterline for every frame and calculate the draft from the estimated waterlines by the following steps:

The steps 3 and 4 are employed to emulate the professional surveyors’ draft reading process¹¹ for getting the understanding of a shipper and a receiver, although the resulting draft may be almost equal to the mean of draft readings.

4.1.

Quality of Draft Mark Segmentation

We have used two images of Figs. 14(a) and 15(a) to test the performance of the draft mark segmentation. Figure 14(a) is a wide angle image, and Fig. 15(a) is a draft mark image of type 2, as shown in Fig. 1. Figures 14(b) and 14(c) show the results of Otsu’s binarization method and the proposed method, respectively. Few draft marks are segmented by the conventional method, while all draft marks are well segmented by the proposed method. Figures 15(b) and 15(c) show the corresponding results for Fig. 15(a). No draft mark is segmented by the conventional method, while all draft marks are well segmented by the proposed method.

Fig. 14

Wide angle: (a) original image, (b) Otsu’s binarization method, and (c) proposed method.

Fig. 15

Other type draft mark image: (a) original image, (b) Otsu’s binarization method, and (c) proposed method.

4.2.

Waterline Extraction in the Rain

We test the performance of the proposed waterline extraction in the rain. Figure 16(a) shows a photograph taken in the heavy rain of about 10 mm/h, and Fig. 16(b) shows the result of Canny edge detection, where the thresholds are 15 and 30. We see that raindrops are also detected in addition to the waterline. Figure 16(c) shows the result of removing very small edges in the waterline extraction. We see that raindrops are removed, thus the proposed waterline extraction is effective even in the rain.

Fig. 16

Rainy condition: (a) observed image, (b) Canny edge, and (c) noise reduction.

4.3.

Reading in a Towing Tank

4.3.2.

Experimental results

Figure 17(a) shows the draft readings of the first 450 frames estimated by the proposed method. Figure 17(b) shows the draft readings after one-dimensional median filtering of window size 9, where “square“ represents local maxima and “triangle“ represents local minima. We call a frame in which draft reading is maximum/minimum within four frames from the frame as the local maximum/minimum frame.

Fig. 17

Draft reading of the proposed method in a towing tank: (a) draft readings and (b) after median filtering.

The drafts estimated by the conventional and proposed methods were 3.59 and 3.55 m, respectively. The true draft is 3.55 m. The draft reading error of the proposed method is $<1\mathrm{cm}$, and it is smaller than that of the conventional method, because the estimated waterline of the conventional method tends to be the highest during the observation time. The simple mean of draft readings is also 3.55 m, which is equal to the result of the proposed method. However, we can get the understanding of a shipper and a receiver by emulating surveyors’ reading process. The total processing time of the proposed method was about 670 s on a Core i5 clocked at 3.20 GHz. The computation of the draft mark segmentation and recognition was dominant, and it took about 540 s.

4.4.

Reading in a Real-World Scene

We have applied the proposed method to two video sequences captured in a real-world scene, as shown in Figs. 2(a) and 18. Both sequences are in $640\times 360$ pixels resolution at 29.97 fps. We captured the sequences from a wharf using a handheld camera for adding camera shake effect, because surveyors often read drafts on a pitching and rolling boat. We have calculated the draft for 1800 frames (about 60 s). We set the binarization threshold to 50 in the conventional method. The other parameters are the same as those of the experiment in Sec. 4.3.

Fig. 18

Real-world scene.

Figures 19(a) and 20(a) show the draft readings of the first 450 frames estimated by the proposed method. Figures 19(b) and 20(b) show the draft readings after median filtering. We see from Figs. 19 and 20 that, although some outliers are caused by the failure of draft mark segmentation and waterline estimation, the outliers are removed by median filtering.

Fig. 19

Proposed method applied to Fig. 2(a): (a) draft readings and (b) after median filtering.

Fig. 20

Proposed method applied to Fig. 18: (a) draft readings and (b) after median filtering.

The drafts of Fig. 2(a) estimated by the conventional and proposed methods were 1.90 and 1.89 m, respectively. The true waterline in the video sequence was moving around the top of “8,” i.e., 1.9 m. Meanwhile, the drafts of Fig. 18 estimated by the conventional and proposed methods were 5.77 and 5.60 m, respectively. Although the waterline depicted in Fig. 18 is around 5.4 m, it represents only one scene of the moving waterline. The waterline in the video sequence was moving around the top of “5,” i.e., 5.6 m. We thus see that the true draft is about 5.6 m, and the proposed method is also effective in the real-world scene. The total processing time of the proposed method applied to Fig. 2(a) was about 1970 s. The draft mark segmentation and recognition took about 1390 s. Meanwhile, the total processing time of the proposed method applied to Fig. 18 was about 1310 s. The draft mark segmentation and recognition took about 1010 s. The processing time for Fig. 2(a) is longer than that for Fig. 18, because Fig. 2(a) includes larger draft marks in pixels and more noisy edges than Fig. 18.