2.1

Remote sensing image of UAV

The stitching of UAV remote sensing images refers to the process that UAV will collect multiple images with overlapping parts according to aerial remote sensing technology, and form a large-angle panorama through certain transformation. Remote sensing technology is a non-contact long-distance detection, because any object has different electromagnetic wave reflection or radiation characteristics, remote sensing is a technology to receive and identify the electromagnetic wave information reflected or radiated by the target object from different remote sensing platforms by using various sensors installed on the remote sensing platform ⁹. However, because the UAV is far away from the target and the performance of imaging equipment is limited, the acquired images often cannot cover the target, so it is necessary to splice the acquired remote sensing image sequences to expand the image perspective. Image stitching is an essential part of the application of remote sensing images. A single image can’t meet the actual needs when unmanned aerial vehicles collect ground object information by remote sensing. Using image stitching technology, two or more remote sensing images are stitched together to form a stitched image covering the target area, so that researchers can make correct analysis and judgment on the target scene ¹⁰.

Artificially spliced images may be dislocated, and the splicing effect is not ideal. Computer automatic stitching is an image stitching realized by program. The stitching efficiency is improved and the quality is guaranteed. Computer automatic stitching is the main way to splice images now. In the process of automatic stitching, how to improve the accuracy and efficiency of image stitching is the difficult point of current research. Image mosaic is one of the important branches in the field of computer vision. Remote sensing image mosaic is to splice the collected images with overlapping parts into one image, which mainly consists of three parts: remote sensing image preprocessing, remote sensing image matching and remote sensing image fusion. The process of image stitching is shown in Figure 1.

Figure 1.

Flow chart of remote sensing image mosaic

Select the reference image from the obtained ordered image, and the adjacent image is taken as the image to be spliced. Due to the unstable performance of UAV in the process of collecting remote sensing images and the influence of ambient illumination, some images will have noise and color difference, so it is necessary to carry out preprocessing to remove noise and balance color difference, and then register the processed images. Registration is to find the similar areas of two images, adjust the relative position under the same coordinate axis, and complete the image matching.

Because remote sensing images contain a variety of targets, and the gray level and color changes of targets are greatly influenced by the acquisition conditions of images, it is difficult to ensure the accuracy and robustness of river detection by using only one feature information. It is possible to extract targets more accurately by effectively using various features and assisting each other and giving full play to their respective advantages. Therefore, it is necessary to fuse various feature information to improve the detection rate. Table 1 shows the comparison of several common features between river area and background area.

Table 1.

Comparison of characteristics between river area and background area

Because texture features are not based on pixel points, they are statistically calculated in a region containing multiple pixel points, so texture features are widely used in image analysis and recognition. Therefore, it can be considered that the background region has different texture information. Based on the above analysis, it can be seen that texture features can be used as an important standard to distinguish river regions and background regions.

2.2

Benchmark target detection model framework

Accurate river detection results are not only the basis for the success of subsequent bridge detection algorithms, but also the important position of accurate river areas in water resources survey, water conservancy planning, and flood disaster assessment. Affected by the sensor shooting range, the actual large range remote sensing images are mostly made up of several small range remote sensing images. There will inevitably be splicing gaps between these images, and sometimes the remote sensing images used for splicing are obtained at different times or by different types of sensors. When UAV collects images, due to the existence of weather conditions and sensor instability, the collected image information may contain noise. Before image registration, the first step is to de-noise. In addition, when the sensor obtains the target information, it may be affected by the change of light intensity, which may make the acquired remote sensing image have obvious color difference, resulting in different brightness between images. The image enhancement processing can improve the image mosaic accuracy. Therefore, probability density and probability distribution function are used to describe the noise distribution. In the process of collecting remote sensing images, unmanned aerial vehicles are disturbed by different factors, and the nature of noise is also very different. Common image noises include salt and pepper noise, Gaussian noise, Ruili noise and exponential noise. The existence of noise will affect the mosaic quality of remote sensing images, so it is essential to denoise the images. According to the different transform domains, image denoising mainly includes spatial filtering and frequency filtering. Spatial filtering is to process the pixel gray value according to the intensity change of the pixel gray value on the original image. In this paper, FasterR-CNN, a two-stage target detection model based on deep learning, is used as the benchmark target detection framework, and according to the characteristics of ground objects in remote sensing images, an optimization scheme of the benchmark detection model framework is put forward to adapt it to the target detection task in optical remote sensing scenes. The basic structure of FasterR-CNN is shown in Figure 2.

Figure 2.

Basic structure of FAST R-CNN

In the RoIPooling module, the candidate regions corresponding to the original image size will be mapped back to the feature image size. Because of the existence of the fully connected layer, these candidate regions with different sizes will be divided into grids with corresponding sizes, and the maximum or average pooling operation will be carried out in each grid to meet the requirements of the fully connected layer for fixed size input. Finally, these candidate regions will be classified more accurately and the regression calculation of the corresponding bounding box will be carried out.

3.1

Image fusion base on non-multi-resolution

In the research and application of images, people are often interested in some parts of the image, which are often called targets or other parts of the foreground as the background. They generally correspond to specific areas with unique properties in the image. In order to identify and analyze the targets, they need to be separated and extracted, and then it is possible to make further use of the targets. Data set is a basic and vital part of a target detection model based on deep learning, so how to process the data set before training becomes an important problem. Conventional natural scene image data set processing methods mainly focus on external data enhancement, such as image translation, rotation, flip and so on. The main basis for the initial detection of the bridge is that the bridge target has strong edge and directionality. Marginality and directionality can clearly show the difference between bridge target and river area, which can guide the algorithm to find the bridge target suspected to exist in the river area in the image. Therefore, as long as we can find the target with strong edge and direction in the river area, we can also determine the approximate position of the bridge target. Compared with non-multi-resolution fusion, the multi-resolution fusion method has a large amount of computation, but the fused image has a good effect, can retain the details of the source image, and avoids the pixel distortion of the two images in the fusion process. Image registration can associate two images to be spliced. However, due to the physical properties of the sensor and the external conditions, when the image fusion algorithm is directly used for splicing, the color difference will inevitably occur at the splicing point of the fused image, resulting in stitching traces. The best way to eliminate the stitching traces is to find an optimal seam, and realize the smooth transition of the two images without changing the details of the image to be fused.

In theory, two conditions should be met to find the seam. The image to be fused has the smallest color similarity in the overlapping area and the most similar geometric structure. Let two adjacent images to be stitched be I₁ and I₂, and the search criterion of color and geometric difference based on weight is shown in expression (1):

In the formula, E_color is the color difference of the pixels in the overlapping area, and E_geometry is the difference of the gradient of the pixels, which indicates the geometric structure difference of the image. And represent corresponding weights, respectively. The calculation formula of is:

Among them, g_x corresponds to the gradient information in the direction of x, and g_y corresponds to the gradient information in the direction of y. The values of the weights a, b are as follows:

Where K is the exposure, when the brightness ratio of two images is 1, the values of a and b are both 0.5.

As analyzed above, in order to output fixed-size feature candidate regions to the full connection layer, the pooled layer of the region of interest in the original model framework introduces two quantization errors while achieving the goal, which makes it difficult to detect small targets. When sampling 49 small grids, the sampling point is assumed to be 4, that is, the small grid is divided into four parts evenly, and the pixel value of each midpoint is taken as the pixel of this part, and then the largest pixel of the four is the pixel point of this grid, where the pixel value of each midpoint is calculated by bilinear interpolation.

3.2

Experimental setup and result analysis

We have carried out a large number of bridge target detection experiments using the self-developed bridge target detection software based on Google Earth platform. Since there is no universal experimental data set for bridge target detection of visible light remote sensing images at present, in order to verify the effectiveness of the algorithm in this paper, we select optical remote sensing images of different regions on Google Earth as the experimental data set of this paper. In addition, after the observation and analysis of a large number of images containing bridge targets, it is found that all the target bridges to be detected are not only located on the water, but more importantly, the bridge is part of the road. The existence of the road does not necessarily exist, but the existence of the bridge must exist, and the presence of the road is a necessary condition for the appearance of the bridge. Generally, the road under the optical remote sensing image has parallel edge features, and the road gray The texture is obviously different from the surrounding objects. In order to verify the performance of the fusion algorithm proposed in this paper, this section will use the representative weighted average fusion method in the non-multi-resolution image fusion method, the Laplace image fusion method in the multi-resolution image fusion method, and the Laplace fusion method based on searching the best seam proposed in this paper to carry out comparative analysis through experiments.

The comparative experiments of YOLOv4 and MD-YOLO with other target detection algorithms are designed, and four algorithms, SSD, RetinaNet, YOLOv3 and CenterNet, are selected to train and test the performance of the algorithms on high-score bridge data sets and DOTA bridge data sets respectively. Provide enough positive samples for model training, and improve the generalization ability of the model and the detection performance of small targets. The following experiments on bridge target detection in optical remote sensing images are compared with the mAP of Task 1 and Task 2, and the PR curves are shown in Figures 3 and 4.

Figure 3.

Optimization Model 1

Figure 4.

Optimization Model 2

The optimized Faster R-CNN target detection model is used to calculate the m AP of the directional boundary box and the horizontal boundary box on the optical remote sensing image bridge data set, and the detection accuracy can be 72.08% and 72.53% respectively. On this basis, the introduction of new_anchor can improve the accuracy by 1.5% and 0.77% respectively while keeping the detection time unchanged. This paper introduces the existing bridge detection algorithms, and on this basis, according to the strong edge and directionality of the bridge target in the remote sensing image, proposes to combine the direction information measure of the image with the fuzzy mean clustering segmentation algorithm to locate the approximate position of the bridge target in the image. Then, the common characteristics of bridge targets in remote sensing images are analyzed, and the bridge knowledge base is established, and used to match bridge targets to achieve the purpose of bridge recognition. The accuracy of the bridges detected in the experiment is very high, and the bridges that are affected by river center islands, slight waves, large buildings, ponds, bridge soil, bridges or ships, as well as non-linear bridges can be well identified. At the same time, the quantitative description of the bridge target is also obtained: length, width, direction and other information.