To overcome the staircase effect and texture loss of total variation (TV), a nonblind image deblurring method that combines the multidirectional fractional-order TV and the traditional TV is proposed. First, a four-directional fractional-order TV (FOTV4) model is put forward to explore more directional information of an image. Then the FOTV4 model and the traditional TV model are related in a unified regularization framework. Finally, the split Bregman algorithm and fast Fourier transform theory are utilized to optimize the proposed hybrid regularization model. Numerical experiments demonstrate that the proposed method can protect the texture and edge while eliminating the staircase effect effectively.

We introduce a simple but effective deep fully connected neural network (FNN) to solve the edit propagation as a multiclass pixel-level classification task. We construct the feature space using three- (or one-) dimensional normalized RGB (or grayscale) vectors and spatial coordinates. Our deep FNN-based model consists of color feature extraction, spatial feature extraction, feature combination, and classifier estimation. We train our model only using the feature within the region labeled by the user with strokes in a single image. Then our method directly outputs the edit propagation results after the forward pass without any refinement process. Our method automatically determines the importance of each image feature across the whole image jointly considering feature vectors from user strokes. Extensive experiments demonstrate that the proposed algorithm achieves superior performance over state-of-the-art methods, yet it remains simple and efficient.

Among image halftoning algorithms, direct binary search (DBS) minimizes the total squared perceived error instead of using heuristic approaches. Therefore, it can generate halftone output with the best quality. The limitation of DBS is that its halftone pattern is controlled by the autocorrelation filter, which corresponds to both the human visual system and the printer model. Because of the low-pass characteristics of the filter, DBS can only generate blue-noise-like halftone pattern and the only variation is the clustered-dot algorithm proposed in 2013. We use the results of theoretical bounds to show that halftone patterns in DBS can be controlled in both frequency and spatial domains through an approach called “noise shaping.” This approach is achieved through an added term to the filtered error in DBS. We also show that DBS clustered-dot halftoning is a special case of this noise-shaping approach. Experimental results show that both the spatial-domain and frequency-domain characteristics of the generated halftone pattern can be manipulated freely using this approach.

A reliable method for the detection and tracking of facial landmarks in image sequences is presented. Given a set of prespecified facial landmarks in a reference face image, a bank of composite matched filters is constructed for reliable detection and accurate location of the landmarks in an input image sequence. The filter bank is dynamically adapted to each captured frame by learning from current and past landmark detections and considering geometrical modifications of the landmarks. The detected landmarks are accurately tracked using a kinematic motion model that predicts their coordinates in future frames. The performance of facial landmark detection and tracking obtained with the proposed method is tested by processing real-life image sequences. The obtained results are analyzed and discussed in terms of objective measures.

Video super-resolution (VSR) aims to restore a high-resolution frame from its corresponding reference frame and a series of neighboring frames in low resolution. Because of the displacement between the reference frame and each neighboring frame, which is caused by the motion of the camera or observed objects, VSR methods usually initially align the neighboring frames with the reference frame. However, the commonly used motion estimation and compensation methods highly depend on the predicted optical flow and they are affected by the lighting change. We propose a phase-aided deformable alignment network (PDAN) to alleviate the above problems. In PDAN, a phase-based method is introduced in the motion estimation subnetwork along with the traditional U-Net structure, which improves the restoration robustness in scenarios of lighting change. A deformable convolutional network is further adopted in the alignment subnetwork to enhance the alignment for objects with an irregular shape and motion distortion, which also avoids the dependence on explicit motion representation accuracy. Moreover, the reconstruction module is optimized for improved restoration. Extensive experiments conducted demonstrate that PDAN achieves state-of-the-art quantitative and qualitative performances.

Blurred images caused by rain streaks can degrade the performance of many computer vision algorithms. Therefore, the single-image rain removal problem has attracted tremendous interest. Although deep learning-based deraining methods have made significant progress, there are still many issues to be addressed in terms of improving the performance. We propose a recursive modified dense network for single-image deraining. As rain streaks have different sizes and shapes, contextual information is very important for rain removal. We use a dense network to extract image features and modify the network by removing all batch normalization layers. A simple deep network cannot completely remove rain streaks from the image, while increasing the network depth will make the computing more complicated. We take a dense block with loops to remove rain streaks stage by stage. Extensive experiments on both synthetic and real-world datasets show that the proposed method can achieve competitive results in comparison with the state-of-the-art methods for single-image rain removal.

With the complexity of the video environment and the problem of possible noise during data transmission, traditional robust principal component analysis (RPCA) failed to obtain the lowest rank representation from corrupted data. A method of video denoising and an object detection algorithm based on the RPCA model with total variation and rank-1 constraint (TVR1-RPCA) is proposed; it employs the more refined prior representations for the static and dynamic components of the video sequences. The proposed method is based on RPCA under the framework of low-rank sparse decomposition; the rank-1 constraint is exploited to describe the strong low-rank property of the background layer, TV regularization is combined with l₁ regularization to constrain the sparsity and spatial continuity of the foreground component, and l₂ norm regularization is combined to constrain the noise to make up for the deficiencies of the existing RPCA model. In addition, an efficient algorithm based on the alternating direction method of multipliers is designed to solve the proposed video denoising and moving object detection issues. Our experiments on static and moving camera videos demonstrate that the proposed method is superior to the state-of-the-art methods in terms of denoising capability and detection accuracy.

We propose a new dual-level convolutional neural network model based on Inception modules and residual connections. First, Inception has filters with different kernel sizes, and its output feature maps contain different scales of receptive fields. The feature map with the wide receptive field receives the global information, while the feature map with the small receptive field contains some local information. The multiscale feature provides more comprehensive information. Second, with the help of residual connections, the training process is simple and can avoid overfitting. Third, the proposed network adopts two levels, i.e., a low level and a high level, and uses the feature fusion operation to take full advantage of the complementary and correlated information of the two levels. Fourth, we combine the spatial features and the spectral features of hyperspectral image (HSI). The pixels to be classified with their neighborhood information serve as the input of the neural network to realize spectral–spatial classification for HSI. Experimental results show that our model performs better than other state-of-the-art methods.

A traditional image pyramid is an effective way to extract multiscale features. However, for image restoration, downsampling and upsampling lose details in the original resolution and result in an over smooth result. To overcome this defect, we propose a receptive pyramid (RP) that replaces downsampling by dilated convolution to extract multiscale features on the original resolution and make a full resolution correction. We present an RP-based convolution neural network named receptive pyramid convolutional network (RPCN) for efficient color image compression artifact reduction (CAR). Specifically, we propose residual connected convolution blocks as the baseline of RPCN. RPs work in convolution blocks to extract the hierarchical multiscale features for local feature correction. Moreover, the global feature fusion and vector correction are also introduced to further exploit the hierarchical features from the baseline. Benefiting from the RP, our RPCN achieves state-of-the-art performance with a much smaller model and much faster running speed for the CAR task.

Deep learning has enabled substantial progress in crowd counting, but existing methods still face difficulties due to significant scale variations, severe occlusion, and overcrowding. To explicitly address these problems, we propose a crowd counting method based on the inverse attention residual network. Initially, the first 10 layers of VGG-16 with a strong transfer learning ability and a feature extraction ability are used as the front-end network for preliminary extraction of head features. Afterward, we use the inverse attention residual module to divide the background information and crowd information and retain the accurate pixel position information to reduce the impact of various types of noise in the input image. Finally, the mini multiscale dilated convolution is used as the backbone of the back-end network to perform the aggregation of feature maps, so the network can learn the details of heads at different scales and use the 1  ×  1 convolutional layer to regress the feature maps to the density map. As a result, the above problems are resolved to a certain extent. Experimental results show that the mean absolute error and the mean square error of the proposed algorithm are significantly lower than those of the comparison algorithms.

Changes in surface water are normally detected by extracting the water features from individual multispectral (MS) images. We present an approach based on pixel fusion of MS and panchromatic (PAN) bands of multitemporal Landsat images to detect the changes in surface water. Spatial frequency and the maximum amplitude selection-based fusion rule on multiresolution undecimated wavelet transform (UDWT) coefficients for extracting and injecting the spatial and spectral content from MS and PAN images is introduced. Landsat Thematic Mapper (TM) and Landsat Enhanced Thematic Mapper (ETM+) of 2010 and 2000 images acquired from the US Geological Survey were fused to investigate the potential of the proposed spatial frequency-based UDWT in highlighting the changes. Various component substitution and multiresolution analysis-based fusion algorithms were also applied. It was observed that UDWT with spatial frequency effectively extracts the spectral information from MS images and the spatial information from PAN images. The performance was evaluated based on visual interpretation and spatial and spectral quality assessment metrics. A high correlation coefficient (0.9992), high contrast similarity (0.99937), high structural similarity index measure (0.9884), less relative average spectral error (23.6426), less root mean squared error (2.2256), relatively less blind referenceless image spatial quality evaluator, natural image quality evaluator, and perception-based image quality evaluator significantly support the proposed fusion technique in yielding the pansharpened image, making it more suitable for detecting the multitemporal changes in remote sensing applications.

Color correction is one of the most essential camera imaging operations that transforms a camera-specific RGB color space to a standard color space, typically the XYZ or the sRGB color space. Linear color correction (LCC) and polynomial color correction (PCC) are two widely used methods; they perform the color space transformation using a color correction matrix. Owing to the use of high-order terms, PCC generally achieves lower colorimetric errors than LCC. However, PCC amplifies noise more severely than LCC. Consequently, for noisy images, there exists a trade-off between LCC and PCC regarding color fidelity and noise amplification. We propose a color correction framework called tunable color correction (TCC) that enables us to tune the color correction matrix between the LCC and the PCC models. We also derive a mean squared error calculation model of PCC that enables us to select the best trade-off balance in the TCC framework. We experimentally demonstrate that TCC effectively balances the trade-off for noisy images and outperforms LCC and PCC. We also generalize TCC to multispectral cases and demonstrate its effectiveness by taking the color correction for an RGB-near-infrared sensor as an example.

Deepfake (a bag of “deep learning” and “fake”) is a technique for human image synthesis based on artificial intelligence, i.e., to superimpose the existing (source) images or videos onto destination images or videos using neural networks (NNs). Deepfake enthusiasts have been using NNs to produce convincing face swaps. Deepfakes are a type of video or image forgery developed to spread misinformation, invade privacy, and mask the truth using advanced technologies such as trained algorithms, deep learning applications, and artificial intelligence. They have become a nuisance to social media users by publishing fake videos created by fusing a celebrity’s face over an explicit video. The impact of deepfakes is alarming, with politicians, senior corporate officers, and world leaders being targeted by nefarious actors. An approach to detect deepfake videos of politicians using temporal sequential frames is proposed. The proposed approach uses the forged video to extract the frames at the first level followed by a deep depth-based convolutional long short-term memory model to identify the fake frames at the second level. Also the proposed model is evaluated on our newly collected ground truth dataset of forged videos using source and destination video frames of famous politicians. Experimental results demonstrate the effectiveness of our method.

J. of Electronic Imaging, 29(3), 033014 (2020). https://doi.org/10.1117/1.JEI.29.3.033014 Online Publication Date: 19 June 2020 Withdrawn from Publication: 03 September 2020. This paper was retracted from the SPIE Digital Library on September 3, 2020, at the request of the authors, who found several serious errors in the experimental results due to incorrect previous experimental setup.

Reliable multitype and orientation vessel detection is of vital importance for maritime surveillance. We develop three separate convolutional neural network (CNN) models for high-performance single-class vessel detection and then multiclass vessel-type/orientation detection. We also propose a modular combined network, which enhances the multiclass operation. The initial three models provide reliable F₁ scores of 85%, 82%, and 76%, respectively. In addition, the modular combined approach improves the F₁ scores for the multitype and orientation vessel detection by 2% and 3%, respectively. The training and testing were done on a dataset, including the multitype/orientation annotations, covering 31,078 vessel labels (10 vessel types and 5 orientations), which is offered to public access.

The acquisition of sharp, full-focus images and the restoration of microscopic scenes require complex instrumentation and algorithms. To conveniently obtain the three-dimensional (3-D) structural information of full-focus images and high-quality microscopic scenes, we construct a zoomable 3-D microscope imaging system and propose new image fusion and depth reconstruction algorithms based on this imaging system. The image acquisition environment of the system is analyzed using interference factors such as light transmission variation and jitter, and the corresponding image preprocessing methods are discussed. We combined the new sum-modified-Laplacian (SML) and local band-limited contrast methods, which contain multiple image features but measure the image definition from different angles, then proposed a mixed-contrast factor, and combined it with the SML method to propose an image fusion method. We then proposed a depth reconstruction method based on the structural similarity of multifocus images. For cases where depth reconstruction results in large distortion with high noise, we proposed a depth value restoration method based on anisotropic diffusion to improve the 3-D reconstruction results. Experimental results show that the proposed image fusion and depth reconstruction methods exhibit excellent performance.

Cross-modality person re-identification (Re-ID) between RGB and infrared domains is a hot and challenging problem, which aims to retrieve pedestrian images cross-modality and cross-camera views. Since there is a huge gap between two modalities, the difficulty of solving the problem is how to bridge the cross-modality gap with images. However, most approaches solve this issue mainly by increasing interclass discrepancy between features, and few research studies focus on decreasing intraclass cross-modality discrepancy, which is crucial for cross-modality Re-ID. Moreover, we find that despite the huge gap, the attribute representations of the pedestrian are generally unchanged. We provide a different view of the cross-modality person Re-ID problem, which uses additional attribute labels as auxiliary information to increase intraclass cross-modality similarity. First, we manually annotate attribute labels for a large-scale cross-modality Re-ID dataset. Second, we propose an end-to-end network to learn modality-invariant and identity-specific local features with the joint supervision of attribute classification loss and identity classification loss. The experimental results on a large-scale cross-modality Re-ID benchmarks show that our model achieves competitive Re-ID performance compared with the state-of-the-art methods. To demonstrate the versatility of the model, we report the results of our model on the Market-1501 dataset.

Recently, trackers composed of a target estimation module and a target classification module have presented excellent accuracy with high efficiency. However, they underperform when encountering background semantic interference, large-scale variation, and long-term tracking. To address these problems, we propose a two-stage tracking framework. First, we propose a more applicable objective function for tracking tasks named metrizable intersection over union by considering the alignment mode and the center distance between two bounding boxes. Second, multilevel features are used to eliminate the semantic ambiguity by exploring diverse semantic information. Third, a meta-synthetic decision strategy is proposed to determine the optimum location of the target. In comprehensive experiments on OTB100, Lasot, TrackingNet, TColor-128, UAV123, and UAV20L, our method performs favorably against state-of-the-art trackers.

Recent years have witnessed the success of dictionary learning (DL)-based approaches in the domain of pattern classification. We present an efficient structured dictionary learning (ESDL) method that takes both the diversity and label information of training samples into account. Specifically, ESDL introduces alternative training samples into the process of DL. To increase the discriminative capability of representation coefficients for classification, an ideal regularization term is incorporated into the objective function of ESDL. Moreover, in contrast with conventional DL approaches, which impose a computationally expensive ℓ₁-norm constraint on the coefficient matrix, ESDL employs an ℓ₂-norm regularization term. Experimental results on benchmark databases (including four face databases and one scene dataset) demonstrate that ESDL outperforms previous DL approaches. More importantly, ESDL can be applied in a wide range of pattern classification tasks.

Underwater images are degraded due to the complexity of the underwater environment and medium scattering and absorption. The image degeneration includes low contrast, color distortion, and blur, which makes the task of underwater vision difficult. A perceptually optimized cycle consistency generative adversarial network (CycleGAN-VGG) is proposed to restore distorted underwater images. We adopt the framework of CycleGAN so that our method does not require pairs of distorted underwater images and corresponding ground truth images for training. Inspired by perceptual loss, we use a combination of perceptual loss, cycle-consistent loss, and adversarial loss. The multiterm loss functions guarantee that the output has the same content and structure as the input while color looks like ground truth images. In an objective assessment, our method is significantly higher than other methods in terms of colorfulness metrics and contrast metrics. It confirms that our method effectively restores the color of underwater scenes and enhances some image details and shows a superior performance when compared to other state-of-the-art methods.

Erratum corrects an error in the affiliation of two authors.