This PDF file contains the front matter associated with SPIE Proceedings Volume 12592, including the Title Page, Copyright information, Table of Contents, and Conference Committee information.

We propose a model-driven deep learning approach to underwater image enhancement that can take advantage of both model- and learning-based approaches. We first formulate a joint optimization problem with physical priors to estimate the transmission map and latent clear image. Then, we solve the optimization problem iteratively. At each iteration, the optimization variables and image priors are updated by closed-form solutions and learned deep neural networks, respectively. Experimental results show that the proposed algorithm outperforms state-of- the-art underwater image enhancement algorithms.

There are many approaches to paint line drawings. However, they are limited in dynamic color update while drawing editing. This curtails the editability of drawings and also hampers the auto-painting of animated drawings. We propose a stroke-based painting representation that associates colors with strokes. A painted drawing is represented purely with strokes, based on which region colors are resolved. With this representation, users can freely modify painted drawings with colors being updated automatically. Color stroke is further extended to auto-paint 2D animated drawings. Experimental results demonstrate the advantages of our method in painting drawings and managing colors.

Exploring the indoor environment and finding unknown objects that appeared in a scene are important for research of scene understanding by a robot. However, background subtraction is traditionally used for segmenting unknown object regions, and it cannot be directly used for a moving camera on the robot. In this paper, we propose a task called view-independent panoptic scene change detection, which is the task of segmenting unknown object regions by comparing two images from different viewpoints before and after the objects appear. In this paper, we propose a method for segmenting unknown object regions by modeling a segmented known instance region as background. For the background modeling, we introduce two methods: histogram-based and deep metric-learning-based methods. In addition, we create a new panoptic scene change detection dataset consisting of images taken from different camera views. Through experiments, we confirm that the proposed method can segment regions of unknown class instances; the deep metric-learning-based method performs more accurately than the histogram-based method, achieving good performance on the change detection dataset.

Light field (LF) image can improve perspective effect and immersive experience. It also provides new capabilities such as depth estimation, post-capture refocusing, and 3D modelling among which refocusing is potentially very useful in many applications such as smart phone. This paper addresses a novel depth-guided enhancement of light field images to improve depth contrast. The proposed method is formulated as an optimization problem that can consider not only the desired inter-depth contrast constraints of depth map, but also intra-depth luminance intensity contrast and color saturation contrast constraints in each depth layer. Experiment result shows that it outperforms the state-of-the-art methods in its depth contrast and detail preservation in each depth layer. It is observed also that the proposed method can enlarge luminance range by 7~15% and improve color contrast by 3~8%.

Over the years, many methods have emerged to solve the super-resolution problem of light field images, and among them, those methods based on deep learning are noted quite attractive recently. Although the features extracted from epipolar domain for the super-resolution of light field images are actively investigated due to their potential capability of well capturing the relationship between spatial and angular domains, we note that spatial features are still the most important foundation in feature extraction. In this paper, we design a network, named as LFSelectSR, employing multiple convolutional kernels to fully extract spatial features and introduce a dynamic selection mechanism that can extract the most valuable spatial features. By training and testing the network using well-known datasets, we demonstrate its excellent performance of achieving the level of state-of-the-arts under certain conditions.

The real-time simulation of high-resolution cloth is challenging. Speed, stability, and robustness are crucial for creating realistic physical animations at interactive rates. Hierarchical position-based dynamics (HPBD) is a cloth simulation method that focuses on real-time simulation. This article describes a new method that leverages the mesh hierarchy of HPBD to support level of detail (LoD) simulation and rendering. During the rendering of only low-resolution meshes, high-resolution simulation is paused until high-resolution rendering is required. Thus, cloth simulation can be performed effectively using LoD algorithms. The new method is demonstrated using various experimental animations. Both HPBD and LoD are widely used in real-time applications. Combining these two techniques may improve the performance and quality of real-time applications.

At resource mining sites, drilling into the ground and bedrock often occurs in geological surveys and blasting explosives filling. However, extracting core samples from underground boreholes is time-consuming, labor-intensive, and difficult to evaluate quantitatively. Visualization of boreholes, which is realized by computer vision technology, allows engineers to evaluate the geological characteristics of underground rocks to determine trajectories and overall budgets. With the help of VR (Virtual Reality) simulation, this research develops a multi-view fiberscope camera system, which obtains videos of a borehole to generate a high-quality 3D borehole model by using one of the 3D photogrammetric techniques, Structure from Motion (SfM).

This paper presents a novel Transformer-based customer interest estimation method using videos from a security camera in a real store. Expectations for the application of Artificial Intelligence (AI) technology in various industrial fields have increased. In this study, we focus on the retail industry field rather than on the manufacturing industry field where AI technology has already been widely introduced. In the retail industry, understanding customer interest in products is one of the significant issues, and Point-of-Sales (POS) data have been used for the analysis of customer data. However, the information of customers before the purchase cannot be obtained from the POS data, which was a limitation due to the characteristics of the data. To provide a solution to the problem, we propose a new customer interest estimation method using visual information obtained from a security camera. The proposed method consists of three phases: Re-identification phase, 3D posture estimation phase, and interest estimation phase. The advantage of our architecture is that the Reidentification phase enables individual identification of multiple persons, and the 3D posture estimation phase obtains highly expressive posture information. Then interest estimation phase is used to enable individual interest estimation. Finally, we achieve a high-level customer interest estimation performance using the data obtained from a real store.

A head-mounted display (HMD) based on computer-generated holograms (CGH) is attracting attention as an ideal augmented reality (AR) technology. However, an HMD using CGH has the issues use of long calculation times and the need to design small optical systems with sensors, and no studies have been reported that perform calculations based on user’s attitude angles. Therefore, we propose an HMD system with a compact optical system with sensors that calculates CGHs fast based on the user’s attitude angle. In the proposed system, the object light is calculated in advance and the light wave is generated by phase compensation with the angle determined from the sensor values. As a result, each object light of the CGHs can be calculated in about 42ms.

In this study, we proposed a method for detecting a player’s movement trajectory from video images to evaluate the defensive range of an outfielder in baseball, quantitatively. Using this method, we succeeded in accurately estimating and visualizing the movement trajectory of a player by identifying only specific players and estimating the homography of changes in the angle of the view by matching the feature points using SIFT.

Owing to the impact of COVID-19, the venues for dancers to perform have shifted from the stage to the media. In this study, we focus on the creation of dance videos that allow audiences to feel a sense of excitement without disturbing their awareness of the dance subject and propose a video generation method that links the dance and the scene by utilizing a sound detection method and an object detection algorithm. The generated video was evaluated using the Semantic Differential method, and it was confirmed that the proposed method could transform the original video into an uplifting video without any sense of discomfort.

The ability to make the viewer perceive the distance from the speaker by voice is called “vocal distance perception”, which is one of the essential skills for a voice actor. “Rehearsal style practice” is one of the most effective practice methods for acquiring and mastering vocal distance perception. In this paper, we propose “VR rehearsal-style practice” that performs rehearsal style practice in a VR space. We also show that VR rehearsal-style practice is effective.

Abdominal ultrasound examination is considered to be highly challenging because of its need to diagnose from moving images taken while handling devices. Previous method consisted of a two-stage inference step where tumors in the input ultrasound image was detected and then the cropped area was classified. However, this previous method may be inaccurate because the tumour detection model is not suitable due to the inability to use global features for classification against the cropped diagnostic image. Therefore, we propose a method that uses SimSiam to pretrain CenterNet and infer using only a single model. The proposed method improves classification accuracy by 3%, and improves memory usage and inference speed by 50% and 33% respectively.

This paper introduces a method to generate 4D portrait of a person that can be played over a long period. 4D portrait is free-viewpoint video of a person with temporal changes in facial expression. In our proposed method, the parameters that represent person’s facial expressions and head poses are obtained from the video captured by a monocular RGB camera with a continuously moving viewpoint. A neural radiance field (NeRF) is trained from the captured video and estimated parameters. Using the radiance field, 4D portrait is generated based on the similarity of the person's facial expressions.

Traffic surveillance cameras recognize a large number of objects. In this configuration, since vehicles appear in the same lanes, past footage is useful information. Context R-CNN has been proposed to store past videos in a memory bank and use them for recognition. In this paper, we propose a method to improve recognition performance by selecting objects to be stored in memory banks. We found that the proposed method can improve mAP by 0.37 points on average compared to the conventional method.

Due to the recent development of machine learning, quality automatic football game commentary is getting possible. By now most of the effort of researchers has spent on key action spotting. This paper is on our strategy for filling in the gap between key actions in order to form a complete automatic football game commentary system. Novel deep learning and signal processing techniques have been used/created in this research work, and attractive demonstrations are available.

With the growing demand for better compression efficiency, the JVET is studying on potential future video coding technologies that can significantly exceed the compression capability of the latest standard, Versatile Video Coding (VVC). In JVET, various methods improving the existing inter prediction modes in VVC have been being discussed and the extension of Geometry Partitioning Modes (GPM) is one of them. In this paper, we propose efficient signaling methods for the extended GPM modes in the Enhanced Compression Model (ECM), which is a reference software for the exploration activity beyond VVC, based on the experimental observation of the mode occurrence frequency. The experimental results show that the proposed methods give meaningful BD-rate gain compared to ECM.

The versatile video coding (VVC) [1] standard has doubled the number of intra prediction modes and MPM modes in the picture compared to the previous standard, High Efficiency Video Coding (HEVC) [2]. The most probable mode (MPM) is used to efficiently encode the intra prediction mode based on the neighboring intra-coded blocks. The VVC improves the compression performance by increasing the number of intra prediction mode and MPM candidates as the resolution of the video increases, but the texture map may be inefficient because the characteristics of the texture map are different from the general image. In this paper, we propose the efficient MPM candidate derivation on the Truncated Signed Distance Field (TSDF) [3] volume-based mesh property (texture map) for multi-view images. The proposed method shows 0.92% BD-rate performance gain for luma component in the random-access configuration [4].

Versatile Video Coding (VVC) procured an impressive coding gain with the help of various new coding tools. In the case of intra chroma coding, chroma separate tree (CST) and cross-component linear model (CCLM) show remarkable coding gain. However, there are only 8 prediction mode candidates in intra chroma coding which is much fewer than the 67 prediction mode candidates and up to 32 matrix-weighted intra prediction (MIP) mode candidates in intra luma coding. Excluding the three CCLM modes, the number of intra prediction mode candidates for chroma is just five, which is less than one tenth of luma channel which uses 67 intra prediction mode candidates except the MIP modes. In this paper, we investigate adding one more intra chroma prediction mode whose prediction direction is derived from the neighboring reconstructed area. The proposed method achieves a BDBR gain of 0.36% and 0.26% in the Cb and Cr channels without loss in Y channel under the all intra configuration.

This paper discusses 3D reconstruction of rice plant community from multiple viewpoint images taken by a small UAV, especially, using spectral reflectance images instead of the usual RGB images. In addition, the paper examines the correlation between ”the number of reconstructed 3D points” and ”indices related to rice plant breeding” such as the number of matured grains. It is known that plants such as rice have higher spectral reflectance intensity in the red to red-edge wavelength range (640-770 nm). This fact expects the correlation between the number of reconstructed points and the evaluation indices will be stronger when reconstruction is done using spectral reflectance images in these specific wavelength ranges, rather than based on ordinary RGB images. Therefore this paper focuses on the Red channel image and attempts 3D reconstruction after applying threshold processing. Also the paper demonstrates through experimental results that the correlation coefficient between the number of reconstructed points and the measured evaluation indices is relative high, in other words, 3D reconstruction using specific wavelength images will be potentially applied to crop phenotyping.

This paper presents double-direction plane sweep stereo, an accurate 3D reconstruction method of the road surface using a fisheye stereo camera, where the normal vectors of the planes are rotated according to the camera pose. As a result, the proposed method can obtain a dense, accurate, and wide FoV reconstruction even in a situation where the road surface has a slope or steps. Moreover, a simple and fast approximation of the LR consistency check is presented. The consistency check efficiently removes outliers, especially the occluded region due to proximal obstacles. The experimental result of real image shows the effectiveness of the proposed method, where the reconstruction error of surface height is less than 2.5cm.

Stone tools are valuable archaeological findings dating to prehistoric age. The ancient people hit a rock with a stone hammer to make stone tools, it also produces flakes. To analyze human activities, reassembling stone tools is effective. When these stone tools are assembled using a computer, the flake surfaces of each stone must be extracted. The segmentation process is necessary for reassembling. This study aims for the segmentation in order to extract flake surfaces, and it requires identifying the orientation of the point normal vector. To do this, the surface of a flake must be considered. If the external space of the object can be correctly defined, directing normal vector is sufficient. While the surface is defined by adopting the voxelization technique, a surface is determined by a voxel that contains points. In addition, external and internal voxels are identified using the 3D flood filling algorithm. We applied normal vector estimation to region-growing segmentation and confirmed its effectiveness experimentally.

Walking through real space and moving through virtual space is one of the methods to improve the immersive experience of VR content. However, the space available for movement in a virtual environment is limited by the size of the real space. As a result, the experience of VR content may be compromised. One solution to this problem is Redirected Walking (RDW). RDW is a method that allows walking on a large virtual space rather than a limited real space. In this method, the virtual space is rotated to create the illusion of walking direction. Previous research has shown that the presentation of tactile sensation and traction increases the degree of illusion. In this study we propose a Redirected Walking technique combined with tilt presentation to the sole of the foot. An experiment was conducted with four subjects to investigate the effect on RDW of the presence or absence of tilt presentation to the sole. The results of the experiment showed the possibility of shifting the range over which the illusion is induced by the presentation of the tilt to the sole.

Hand gestures play an important role in human-computer interaction (HCI) system. Areas of applications such as sign language recognition, virtual environment, video games and even surgery require people to interact with devices directly with hands. In this paper, we propose a vision-based system to deal with dynamic hand gesture recognition. We focus on 6 types of gestures: 4 directions for swipe and 2 orientations for rotation. The hand gesture can be identified based on the moving trace of the hand in different video frames. However, different from existing algorithms, the proposed method applies the techniques of ellipse and line approximation to well identify the moving trace and reduce the effect of noise. Moreover, the confidence of each frame is determined by the velocity and the continuity of the motion vector. The frame with less confidence will not be adopted in hand gesture approximation. We also develop a face-based skin region detection to adaptively choose a proper color model. The proposed system is hardware-efficient and can achieve a good accuracy.

Because safety is important in everyday life, according to the National Retail Federation, organized retail theft accounts for an estimated 30 billion dollars each year in the U.S. Therefore, this paper proposes the detection system of theft walking patterns using motion-based Artificial Intelligence from a stationary camera. This method uses object detection by the Yolov5 model for localization of the objects, which is less affect to a background problem to detect the moving objects from the RGB video. After that, the coordinates of bounding boxes from Yolov5 are used to calculate the centroid of the box; then, it is used to make the motion of walking pattern behaviour. Finally, the sequences of the motion patterns are used with a bi-directional long-short term memory network, which is used for the time-independent pattern using 10-fold cross validation at training size 50% of total dataset. To evaluate the performance of the method, the datasets consist of normal walking people, which have 30 videos, and theft walking people, which have 30 videos. The accuracy of the system is approximately 97%.

This paper discusses an approach to training data augmentation for identifying and detecting tactile paving (braille) blocks in image with machine learning. Image recognition system that assists the visually impaired is necessary to efficiently identify the guiding blocks in the image, and machine learning, represented by CNNs, etc., is considered effective for this purpose. However, it is labor intensive to collect a sufficient number of training images from the pedestrian’s perspective in various environments, and furthermore, if the number of training data is insufficient, the final identification performance of the entire system will be degraded. To solve these problems, this paper attempts to generate new training data from a small number of tactile paving images by GAN(Generative Adversarial Network), and demonstrates through evaluation experiments that adding generated images to training data can stabilize learning and improve the rate of correct answers.

The human eye has three types of cone cells called L-, M-, and S-cones. These cones have sensitivity peaks at long, medium, and short wavelengths, respectively. Color vision deficiency (CVD) is the inability to distinguish between certain shades of color due to the limited function of the cones. It is important to understand how CVD patients perceive color. Since color reproduction in most of the conventional methods is based on a conversion from RGB images, it is necessary to reproduce colors from spectra in order to represent colors more accurately.

In this paper, we propose a method for reproducing the color vision of CVD patients based on the stage theory, which combines trichromatic theory and opponent-color theory. The proposed method focuses on the diversity of CVD patients and can reproduce the color vision according to the severity of CVD. Furthermore, by taking into account the spectrum, the proposed method can investigate the problem of metamerism, which cannot be realized by conventional methods using RGB images.

Spatial projection of holographic reconstructed images by blue-violet laser light is stated. Studied results are reported. Firstly, construction of mist screen is introduced, and the conditions under which optical characteristics of the screen will be improved are reported. Evaluation of improved situation is carried out by objective and subjective evaluating technics. Reconstructed images obtained by projecting blue-violet laser light are shown, and a relation between the usage of two kinds of fans and the quality of the projected images (brightness and stability) is studied. It is carried out by combining optical characteristics (objective evaluation), subjective evaluation and automatic decision making based on image analyzing process. By this, a certain relation is seen to exist between the result derived from each of the evaluating processes and the projected image judged as "good". Using this result, it is found that a clear projection of the images by blue-violet laser light is really possible.

A camera-based depth estimation method is essential for self-driving. Depth estimation methods based on stereo cameras capture objects and triangulate the depth between the objects and the camera using two cameras. However, they are vulnerable to parallax distortion owing to vibrations and degradation. On the contrary, depth estimation methods based on a monocular camera, such as depth-from-defocus (DFD), use a focusing mechanism or spectroscopic optics. However, because the sharpness does not change sufficiently in the depth direction, it is not suitable for wide-range estimation. Therefore, we are developing depth-estimation methods using a monocular camera by focusing with tilted optics. When tilted optics is used, the depth of field (DOF) increases toward the depth direction, and the plane of sharp focus (POF) appears tilted. In this study, we propose to improve the accuracy of the depth estimation method using tilted optics with a color filter aperture by performing sharpness calculations corresponding to the edge direction and sharpness model fitting using the weighted least squares method.

It is inevitable that an image is suffered from the noise. There are several existing image denoising algorithms. Some image denoising methods, including the Wiener filter, the maximum likelihood method, and their adaptive forms, can be implemented efficiently by the linear convolution and the two-dimensional discrete Fourier transform (2D DFT). However, their performance is highly affected by the distribution of the noise. For different values of the peak signal-to-noise ratio (PSNR) and different style of the noise, the optimal parameters for image denoising are also different. Therefore, noise estimation plays a critical role in image denoising. In this work, an effective way for noise estimation is proposed. The proposed algorithm is based on the following two priors. We think that, for an image without noise, there should be some patch that has the mean gradient near to zero. We call it the smooth patch prior. Moreover, in the frequency domain, there should be some bands that has the average energy near to zero. We call it the zero-band prior. Therefore, the noise can be estimated from the patch with a relatively smaller average gradient and its sparse part in the 2D DFT domain. With the proposed algorithm, the noise distribution can be well predicted and the optimal parameter of the image denoising filter can be determined automatically.

Semantic segmentation is a pixel-level classification problem in computer vision, in which pixels of the same class are grouped into a single category in order to interpret pictures at the pixel level. In this field, semantic segmentation of street fashion images is a challenging task since the clothing items would appear with wide variations in fabrics, layering, occlusion and viewpoint. To help better understanding the street fashion images, we propose a lightweight Semantic Context Aware Transformer (SCAT) to be applied to the semantic segmentation task for street fashion images, which integrates semantic context into the encoding, and models the relationship between multi-level outputs from transformer layers. Extensive experiments and comparisons show that the proposal achieves the state-of-the-art results on ModaNet dataset with relatively small model size, with over 1.1 point improvement compared to Shunted Transformer, and even surpasses other CNNs and Transformers with a large margin of over 2 point in mIoU.

Sign language is a visual language that uses hand signs, arm movements and facial expressions to convey information. However, learning sign language is challenging task, due to variety of movements that should be executed precisely to express the correct information. Many studies have been made to recognize and translate the sign languages. Traditional methods face problems when used in a real-world situation due to the background and lighting variations. Recently, the result obtained from research using multi-modal data, including human skeleton data to Sign Language Recognition task (SLR), have archived a remarkable success.¹ This paper aims to make a model that uses only skeleton data to recognize sign language using Graph Convolutional Network (GCN) and make a Japanese Sign Language dataset to help future SLR research.

Biometrics has been introduced in many facilities, and face authentication is one of the authentication methods that has attracted attention. However, face authentication is influenced easily by changes in lighting conditions. This study proposes a face authentication method using only a thermal camera, which is robust to lighting changes. The proposed method applies FaceNet[1], an authentication method for visible light images, to each of the four parts of the face (face, eyes, nose, and mouth) and tries to improve the accuracy by using a majority voting method.

For accurate color restoration of underwater images, it is necessary to acquire the spectral scattering properties of seawater. In this study, we propose a method for acquiring spectral scattering properties from an RGB image. In the proposed method, we first obtain scattering properties (scattering and extinction coefficients) in RGB space. Then, we estimate spectral scattering properties from these RGB coefficients. We conduct simulation experiments using seawater datasets with various spectral scattering properties, and evaluate the accuracy of the proposed method.

Video prediction is a task in computer vision that predicts future frames from the past few frames of video. In video prediction, a simple CNN-based approach called SimVP has marked remarkable performance without using RNN or vison transformer (ViT). In this paper, we propose a model structure to improve performance of video prediction based on FitVid. FitVid is a regression-based method of predicting future videos using not only video but also motion. We focus on video prediction only conditioned on videos. To this goal, we introduce network modules used in SimVP to FitVid. Experimental results show that the proposed structure shows better prediction accuracy compared to SimVP.

Intravesical prostatic protrusion (IPP) measurement plays an important role in urology. It can be determined by the morphology of the bladder and the prostate gland and is helpful for the diagnosis of benign prostatic hyperplasia (BPH). In this work, an adaptive ultrasonic bladder image analysis algorithm for IPP measurement is proposed. First, to accurately extract the bladder part and the prostate gland part from an ultrasonic image, the smooth filters with adaptive thresholds are adopted. Moreover, we apply the multiscale filters with adaptive thresholding to extract the contours that are possible to be the borders of the bladder and the prostate gland. The confidence, curvature, and turning points of each contour are also determined. Then, since the contour may not be continuous, several techniques, including the curvature different, the weighted distance measurement, the confidence path, and arc extrapolation are adopted to connect the contour segments into two ellipse-like contours. These two contours are treated as the borders of the bladder part and the prostate gland part. With the proposed algorithm, the accuracy of IPP measurement can be much improved and the measuring result is more robust to the noise in the ultrasonic image.

Statistic of an individual player, tactical analysis in soccer-team, and offside event in a soccer game impacts to match results. An important step to analyze information of the individual player is a soccer team classification. In this paper, we proposed a method for classifying the soccer player-team in a match by histogram features and distance classification technique. This method consists of three steps: 1) player detection 2) player segmentation 3) feature extraction and team classification. Firstly, to segment foreground objects, the soccer ground field is removed by comparing the green color level with a threshold. Morphological technique is used to remove noise in foreground image. Region of objects which is smaller than criterial is also removed. Therefore, remain object is defined as players. Player can be detected. For soccer-player-team classification, histogram in RGB layers are used as feature vectors. Using a distance classification and feature vectors in database, team is classified finally. To test the performance of methods, three videos of soccer match are used. 393 player images are selected. These player images are used for classification. The linear support vector machine are used as classifier. Accuracy rate is 98.47%. Sensitivity rate is 100%. Specificity rate is 96.03%. Based on the results, the proposed method shows excellent performance.

In this paper, we propose a face image super-resolution method, aiming to reconstruct high quality face images from a low resolution input. The proposed method introduces an attentional multi-scale feature fusion block, which aims to improve the representation power of the neural network by emphasizing the important feature maps and suppressing the unimportant ones. In addition, the facial prior information is utilized by adding a separate prior branch, an hourglass structure is used. Experiments show that the face images reconstructed by the proposed method exhibit noticeable quality improvement compared to the low-resolution images and other SR approaches.

In this study, we investigated a deep-learning method for visualizing Cutibacterium acnes (C. acnes) in facial photographs taken with visible-light irradiation. We used a dermatological device to capture photos with UV-light irradiation wherein C. acnes on the faces of people and paired these photos with visible-light images to train a deep learning model. Then, two methods were devised to increase the learning efficiency, namely a method to crop and reduce the size of the image used for learning and a method for highlighting the points where the C. acnes emitted light. These results suggested the potential application of these methods in visualizing C. acnes using only smartphones.

Human hands, an essential component of the human body, play a vital role in interacting with and sensing real-world objects and are a reliable medium in modern technology for developing human-computer-interaction (HCI). Human Hand Pose Estimation (HPE) is challenging for numerous Artificial Intelligence (AI) applications due to the strong self-occlusion of the hands, depth ambiguity, and agile movement. Implementation of vision-based hand pose estimation algorithms can give a breath of innovation of these AI applications to overcome the challenges. We proposed a framework called Cascaded Deep Graphical Convolutional Neural Network (DCGCN, where Deep Convolutional Neural Network (DCnet) is used for computing unary and pairwise potential functions. A graphical model inference module is used for cascading unary and pairwise potentials. Evaluating the generated results via subjective and objective analysis, our DCDCN outperforms the state-of-the-art models in terms of accuracy and computational cost.

In this paper, we experimentally verified that deep learning based on Convolutional Neural Network (CNN) is an effective method to judge the freshness of seafood fillets from its images. Currently, freshness of seafood is generally estimated by human’s eyes (connoisseurs) in Japanese fishing industry. However, it requires many years of experience and is a difficult task for new workers due to lack of experience. Furthermore, the Japanese fishing industry is faced with a worker shortage that makes it difficult to pass on their skills to next generation due to an aging and a lack of new workers.

Meanwhile, CNN has been achieved a certain success in image recognition areas. If this deep learning can be applied to predict the freshness of fishes, i.e., if the freshness of a subject can be inferred from an image of seafood, it would be a solution for the problems of aging and labor shortage in Japan’s fisheries industry. Therefore, in this paper, we verify that CNN-based deep learning models are effective in estimating/predicting seafood freshness using tuna and squid, which are commonly caught fish in Iwate Prefecture (in Japan) and show that representative CNN models such as ResNet-50 achieved nearly 100% of predicting accuracy from experiments with over 12,000 images.

In this research, we aim to realize the improvement of the accuracy of radiometric compensation for moving non-rigid bodies. By predicting the movement of objects using deep learning, the influence of delays in measurement and projection depending on cameras and projectors on correction accuracy is reduced.

JPEG is a widely used image compression standard that shows a reasonable image quality for a wide range of compression rates. However, when compressed with a low compression quality factor to increase the compression rate, it brings a large loss in the frequency domain, which turns into visible artifacts in the image domain. Accordingly, removing artifacts in JPEG-compressed images has been an essential image restoration task. While most previous methods use the information on compression quality factors available in the header of the JPEG file, we note this approach is not practical in the real-world scenario because many compressed images’ metadata are not exist. To deal with this issue, we propose a new method based on a Deformable Offset Gating Network (DOGNet) and a Variational Autoencoder (VAE). We train the overall network in an end-to-end manner, where the role of the VAE is to guide the offset of the deformable convolution to flexibly deal with images compressed with diverse and unknown quality factors. Extensive experiments validate that our method achieves better or comparable results to the state-of-the-art methods in JPEG Artifact Removal.

Advanced, safe, and secure walking support could be possible if the next step of a visually impaired can be estimated before the foot actually touches the ground. We define the next step as one step to touch the ground now. We aim to predict the behavior of this next steps. In this study, we investigate a method to realize next-step landing position prediction using an IMU on a smartphone and discuss the possibility of next-step prediction. Since it is difficult for a person to walk while holding a smartphone steadily against the body, next-step prediction using only an IMU is considered an ill-posed problem. In this study, we assume that there is a pattern in the sway of a person's gait. To learn the pattern, we created a training dataset consisting of pairs of IMU outputs, positions of next steps, and the smartphone position to realize the next-step landing position prediction. We used the angle and distance errors to validate the next-step landing position prediction as the evaluation indices.

This paper provides insight into the 3D scene compression represented by a neural implicit function. The goal of this paper is to introduce the aspects of implicit neural representation for 3D scenes such as NeRF (Neural Radiance Field) and propose a novel compression method for Neural Implicit representation for 3D scenes. We also provide the analysis of compression performance of 3D scene representation by using Neural implicit function.

In the perspective of alleviating the inherent trade-off between the spatial and angular resolutions of light field (LF) images, much research has been carried out to increase the angular resolution of LFs by synthesizing intermediate views. Since the height of each EPI is equal to the angular resolution of LF, we tackle the view synthesis problem as doubling the height of each EPI in LF. To efficiently stretch the EPI while not consuming too much computing time, we propose to first segment the EPI into superpixels and then adaptively interpolate each superpixel separately. The test results on the synthetic and real-scene LF datasets show that our scheme can achieve average Peak signal-to-noise ratio (PSNR) / structural similarity index measure (SSIM) around 30.58dB / 0.9131 and 32.28dB / 0.9510, by taking computing time of 5.80 minute and 1.83 minute for HCI and EPFL dataset, respectively.

This paper handles with detection of leaf diseases using deep learning networks which have learned the color and shape parameters of leaf diseases. This paper considers the color distribution and shape information of leaf diseases, and exploits two deep leaning networks in training the normal leaves and diseases. The input color image is partitioned into small segments using color clustering, and the color information of each segment is inspected by the Color Network. When a segment is determined as abnormal (that is, disease segment), the shape parameters of the segment are inspected by Shape Network to classify the disease types. This paper uses HSV color space for Color Network and proposes 24 parameters for Shape Network such as boundary length ratio, densities of subregions, correlation coefficients of x-y coordinates in the disease segments. According to the experiments with three types of diseases (type A, B, C) for images of iceberg, strawberry, coffee, sunflower, chinar, blackgram, citrus, and apple leaves images, leaf diseases are detected with 97.9% recall for a segment unit and 99.3% recall for an input image where there are more than two disease segments.

This paper proposes a method of embedding and extracting watermark in end-to-end (E2E) image compression that is an emerging image compression framework. Although end-to-end image compression algorithms are recently developed based on deep neural networks to beat the conventional image compression methods, there are no research to support watermarking embedment and extraction for contents protections. This paper proposes a NN-based watermarking algorithm for end-to-end image coding. At the end-to-end image coder and decoder, the GDN-based multi-layer modules for embedment and extraction of watermark are equipped to support watermarking for E2E image compressor. The proposed method shows an average BD-rate gain of 8.07%, over the method of directly connecting the existing watermarking network and end-to-end image compression network. In addition, bit error rate (BER) of watermark is also improved by more than 2.7% in terms of robustness.

We aim to evaluate the depth map quality by the plenoptic 2.0 cameras for DIBR applications, by simulating a multi-plenoptic camera array, and synthesizing virtual images based on the obtained color and depth maps. The real-time generated depth maps by plenoptic cameras are compared with the offline and high-quality depth maps estimated by the MPEG-I Depth Estimation Reference Software (DERS). Results show that synthesized virtual image quality obtained by RGBD plenoptic cameras has acceptable quality for real-time DIBR applications, but lower than using the depth estimated by DERS, due to a loss of inter-view depth consistency.

In this paper, we propose to speed up bilateral filtering by principal component analysis (PCA)-based dimensionality compression method with constant-time bilateral filtering. Constant-time bilateral filtering speeds up the filtering by representing it as a summation of the multiple Gaussian filters. However, a simple implementation is of the order of O(K³) for color and suffers from the curse of dimensionality. A clustering-based approximation speedup solves this problem with an order of O(K) or O(K²). PCA can provide a more informative signal relative to the transformation matrix. We have accelerated the process by converting the color information of the input image from 3-channel to 1-channel by PCA, considering the constant-time bilateral filter as joint bilateral filtering, and using the transformed image as a guide image. This dimensional reduction allowed us to filter images with sufficient accuracy at a higher speed.

This paper presents methods of Neural Network (NN) training reflecting block partitioning for Matrix-based Intra Prediction (MIP)-based networks. A training method using a dataset considering coding block partitioning leads to a NN-based predictor that is more suitable for a legacy block-based video codec compared to a training method that does not consider block partitioning. In addition, training using block partitioning of actual video encoding allows better intra prediction than a training method considering block partitioning in the training process. The MIP-based intra-prediction networks are implemented in VVC by replacing the MIP to evaluate the proposed training methods. The experimental results show that the proposed training method considering block partitioning of actual encoding gives the coding gain of 0.19% Bjøntegaard Delta (BD)-rate on average compared to training without considering block partitioning.

In this paper, we propose a novel feature map compression method for Video Coding for Machines (VCM). The proposed method performs a principal component analysis (PCA)-based transform on feature pyramid network (FPN) feature maps using predefined basis and mean vectors. In addition, the proposed method reduces redundancy between different resolution levels within FPN feature maps based on redundancy between FPN layers. The fixed predefined basis and mean are employed through PCA with a set of training data set. For any input videos, transform coefficients are obtained by performing transform with the fixed basis and compressed using Versatile Video Coding (VVC). Experimental results show that the proposed method achieves 89.22% and 86.57% BD-rate gain compared to the VCM feature anchor in instance segmentation, and object detection, respectively.

RISE is one of the methods used for visualizing the basis of neural network decisions in image recognition. RISE creates a heat map showing the importance of various parts of an image by observing the response of the network while partially obscuring the input image with a random mask. However, this method requires many mask images to obtain stationarity, resulting in a huge amount of computation time. In this study, we use a non-random patch mask that passes through only one limited region in addition to an improved random mask to reduce the number of masks needed, thereby speeding up the RISE process.

In a previous study, we applied patch classification, a fiber estimation method, on macro-images of paper obtained using a digital camera; these macro-images were obtained from a limited number of materials. The method efficiently classified these images by dividing them into image patches. However, this method did not analyze the entire macro-image. Therefore, to extend the application of this patch-based paper fiber classification method to entire macro-images, we propose a method wherein EfficientNet is applied for fiber estimation in macro-images; the type of fiber of each image patch was estimated in two stages. The first stage was patch fiber classification 1 (PFC 1), wherein image patches were classified into three fibers, yielding PFC results for the patches comprising a macro-image. If more than 80% of the patches were determined to be of the same fiber by a majority vote, these results would be used as the paper fiber estimation (PFE) results for the macro-image. If less than 80% were classified to be of the same fiber under PFC 1, then patch fiber classification 2 (PFC 2), wherein the patches between two fibers were classified, was performed. Then, by majority, the PFC 2 results would be used as the estimation results. We targeted three fibers, namely kõzo, mitsumata, and gampi; and we used 1179 macro-images (393 for each fiber). The fiber estimation accuracy for the macro-images was evaluated using three-fold cross-validation. We achieved an accuracy of 87.6% in estimating fibers in the macro-images of paper using the proposed method.

For effective super-resolution processing of video images, video images should not be processed as frame-by-frame two-dimensional information, but as spatio-temporal information, including information in the time axis direction. Most of the proposed video super-resolution processing based on deep learning uses 2D convolutional neural networks (CNNs). Therefore, the system is based on the 2D CNNs with the additional processing related to change from frame to frame. Instead of processing video images separately in space and time, comprehensive processing as spatio-temporal information can be expected to be more flexible and effective. In this research, we propose a video super-resolution process that processes spatio-temporal information by 3D-CNNs and GAN (Generative adversarial networks). By using 3D-CNNs, the configuration does not require motion alignment as preprocessing. Since the essential purpose of the super-resolution process is to predict missing high-frequency components, we added a process that directly predicts the difference between the high-resolution image and the corresponding bicubic interpolated low-resolution image.

In sports, martial arts and other interpersonal sport games, skilled players often predict the next move and other actions based on the actions of their opponents. In badminton, players predict the return of a shuttlecock based on information, such as opponent location, the impact point or the racket surface direction. The more experienced a player plays badminton, the more intuitive he or she becomes in predicting the return strokes. In addition, the rally speed of badminton has been increasing year by year in recent years due to improvements in athletic ability resulting from experience, improvements in the performance of rackets and shoes, and physical modification through efficient science-based practice. Badminton has become the fastest ball game in the world.

In badminton games, in which shuttlecocks fly around at such high speeds, it becomes extremely difficult to accurately predict the next shuttlecock destination. If the return position of the shuttlecock can be predicted, it will not only make it easier to formulate strategies to beat opponents, but also make it possible to visualize information on how players can overcome their weaknesses. In this paper, we propose a method of predicting where a player will hit a smash depending on his opponent location by applying machine learning to the positional information of the player and the shuttlecock. We also propose a method to predict where the opponent will hit the shuttlecock next by using the machine learning model obtained by this method and to predict whether the opponent can return the smash or not.

In recent years, tourism services and workshops that allow visitors to go back in time and relive history have become popular, and AR systems that recognize landmarks and superimpose old information for the viewer have been developed AR. In this study, we apply an old-fashioned style transformation to the entire image using GAN and segmentation techniques to photos taken on the spot. Then, we propose a method to merge the current and old scenes while maintaining continuity by mapping current items to old items on a domain basis. Specifically, we created a system that makes users feel as if they have been transported back in time to the Edo period by utilizing Ukiyo-e content using CycleGAN and Instance Segmentation on real images, and evaluated viewer impressions. As a result, it was confirmed that the system generally allowed visitors to feel the atmosphere of the Edo period, although the three-dimensionality and coloring of the playhouse and teahouse remained unnatural. In the future, we plan to improve the quality of the system in order to offer it to actual tourists.

DNN-based video frame prediction can be a powerful tool to improve performance of motion-compensated prediction in video coding. In this paper, we propose a method applying multiple convolution kernels with different sizes to PredNet, which is one of the DNN-based video prediction schemes, to enhance the prediction accuracy by incorporating context adaptivity in its convolutional LSTM layers. We analyze the prediction performance of the proposal, and the results show that applying multiple-size kernels is effective than applying a single-size kernel in terms of prediction error reduction.

Privacy-preserving deep neural networks (DNNs) have been proposed for protecting data privacy in the cloud server. Although several encryption schemes for visually protection have been proposed for privacy-preserving DNNs, several attacks enable to restore visual information from encrypted images. On the other hand, it has been confirmed that the block-wise image encryption scheme which utilizes block and pixel shuffling is robust against several attacks. In this paper, we propose a jigsaw puzzle solver-based attack to restore visual information from encrypted images including block and pixel shuffling. In experiments, images encrypted by using the block-wise image encryption are mostly restored by using the proposed attack.

In recent years, with the spread of devices with various screen sizes, the demand for resizing methods is increasing. Seam carving (SC) is one of the resizing methods. This method removes visually unimportant areas (hereinafter referred to as "seam") and resizes the image without losing its impression. SC for video images has the problem that the most impressive areas are shifted because the seams are different from frame to frame. There are previous methods to solve this problem, but they require a huge amount of processing time. The proposed method enables SC with reduced misalignment and processing time by area segmentation and seam reuse.

3D point cloud data consist of a large number of points with attribute information such as color in addition to geometry information of 3D positions. Since their data size tends to be large, efficient compression methods not only for the geometry but also for the attribute information are desired. In general, attribute information has a spatial correlation depending on local texture of the object surface. However, the conventional adaptive prediction technique, which is popular in 2D image coding, cannot be applied as-is, since distribution of the already encoded samples used for the prediction is usually sparse and irregular. In this paper, we propose a method for designing adaptive predictors on a point-by-point basis using a 3D directional autocorrelation model of the attribute information. The obtained predictors are utilized in the probability model optimization technique for efficient lossless coding of RGB color attribution.

This research proposes a method based on image recognition technology to guide the visually impaired to the shortest Braille block in situations where they cannot find the existence of Braille blocks. In this paper, we make four proposals. First, we propose a method for detecting Braille blocks using SSD, a deep learning network with our own image dataset. Various kinds of Braille blocks and out-of-specification Braille blocks look different by weathering indoors and outdoors. Second, we propose a detection method that can handle differences in camera height. By constructing a training dataset with images of Braille blocks taken at a person's height or a general robot, we can achieve detection with cameras of different heights. Third, we propose a standalone method, real-time recognition of Braille blocks using the cameras of mobile devices. We incorporate MobileNet, a lightweight deep-learning network, into the SSD network. As a result, we achieve standalone and real-time Braille block recognition by optimizing the network for mobile devices. Finally, we propose a method of estimating the orientation and shortest distance of Braille blocks. It considers the continuous and linear arrangement pattern of guiding blocks that indicate the direction of travel.

This paper proposes a method of detecting entry and exit of vehicles in each parking slot using a surveillance camera placed outdoors. To specify the respective parking slot areas in a surveillance camera image, polygonal windows are set by projecting rectangular boxes, each of which surrounds the typical body of a vehicle parked in each slot, onto the image based on the perspective projection matrix associated with the surveillance camera. The projection matrix is semi-automatically calculated by providing 3D coordinates to a small number of points picked up from the surveillance camera image. Then, a joint intensity histogram between a pair of images taken by the same camera with a certain time interval is calculated in each window. By analyzing the distribution of the histogram, entry and exit of vehicles in the slot can be robustly detected without being affected by lighting change during the time interval.

Most studies on high-dimensional data preprocessing for data mining and pattern recognition, attention have focused on unsupervised feature selection, which aims to reduce data redundancy and select the most representative features from massive unlabelled data. Previous work on convex nonnegative matrix factorization with adaptive graph constraint has indicated that selecting a few essential features helps replace the original features for clustering. However, the problem exists in how to decide the quantity of these essential features without prior label information. This study proposes a customized evaluation criterion to evaluate each essential feature set. A significant advantages of this method is that the optimal number of features can be determined without knowing their true labels. Experiments were verified by the consistency between the results of our method without label information and that of the comparison method, which required label information.

In this study, we investigate the weight illusion produced by changing the impression of an arm by displaying visual effects on an avatar's arm superimposed on its own arm. We have confirmed that visual changes using virtual arms produce an illusion of weight in comparisons between different weight conditions. It was pointed out that differences in the impression of the weight of the arm itself due to differences in the appearance of the virtual arms may affect the perception of weight. In this paper, we examine the effect of changes in the impression of arm strength on the perception of weight by changing the impression of the arm by visual effects that do not affect the impression of the arm's weight.

Haniwa were created for ritual use during the Kofun period and were buried with the dead as funerary offerings. Archaeologists visually observe haniwa and classify them based on archaeological findings, such as the characteristics they possess. Since observation is a subjective evaluation, an objective evaluation method is necessary for authenticity. For objective evaluation, Lu .et al.¹ presented a previous research by using 3D point clouds of Haniwa. Unfortunately, Lu .et al.¹ includes manual operation to extract target point sets.

Therefore, this paper describes a method to automate the process of extracting only the Haniwa’s face from point cloud data of the whole Haniwa body, as shown in Figure 1, which was used in the previous study.¹

We believe that many people are interested in the field of music. Due in part to Corona's influence, there has been an increase in the number of people who are home-based. Therefore, the number of people who are interested in music has probably increased even more due to the idea of increasing the number of hobbies that can be done at home. However, we believe that it is difficult for people to actually play musical instruments, produce music, or otherwise engage in music on their own initiative. In addition, playing a musical instrument requires fine manipulation for people with disabilities, and some people are not able to play a musical instrument to begin with. Therefore, we began researching simple instruments that can be played with a nod by people who have given up playing musical instruments or who are unable to play due to hand disabilities. The purpose of this research was to lower the barrier to playing a musical instrument by making the operation of playing a musical instrument simpler. In the process of developing the instrument, we thought that it could also be used for stretching the neck, so we added a variety of stretching exercises, from nodding (forward bending) to back bending and side bending, to correct straight necks, which plague many people in modern society, while performing on the instrument. Exercises used for stretching were added to the playing method. As the first stage of the research, we used OpenCV's cascade classifier, an object detection feature, to detect faces and nod to enable simple playing of the instrument. In the second stage, we used FaceOSC to improve the accuracy of face recognition, making the research more interesting and usable for stretching. Finally, measurements were made on the range of motion for forward bending, back bending, and side bending, and experiments were conducted on the actual use of what was produced in the research. The range-of-motion data from the measurement side could not be used well, and problems arose, such as the fact that the recognition of faces was off, and if the face was covered with hair, the recognition would be off. In order to improve these issues, various improvement measures were taken, including reviewing the programming in the first place to take into account the range of motion of the face and neck, and reviewing the position of the camera.

With the recent development of 3D computer graphics technology, the required hardware processing speed has been increasing. As a solution to this problem, there are methods to reduce the drawing load by omitting some drawing processes. Level-of-Detail and Occlusion Culling are examples of conventional methods, but both have the problem that they cannot reduce the rendering load for near objects. In this method, the drawing of near objects that exist in the shadow and are not illuminated by light is omitted, thereby reducing the drawing load while suppressing the loss of impression.

Japan is rich in cultural properties of great historical and artistic value, of which the most important are protected as nationally designated cultural properties. Intangible cultural properties are techniques that have been handed down from generation to generation, such as theater, music, and craft techniques. In Japan, where the population is aging rapidly, digital archiving is essential for the transmission of intangible cultural assets. In this study, we focused on the musical accompaniment of the Kanuma Imamiya Shrine Festival (Kanuma Autumn Festival), which is designated as a UNESCO Intangible Cultural Heritage and a nationally designated Important Intangible Folk Cultural Property. Although the Kanuma Autumn Festival had to be canceled last year due to the emergency declaration requested by the spread of coronavirus infection, the festival is still being carried on. In this study, we have developed multi-viewpoint viewing system for ohayashi content with the cooperation of the Kamifukatsu Music Preservation Society and evaluated the system through experiments with participants. As a result, we obtained positive feedback from the participants, while those who had experience with musical performances pointed out points that needed to be improved. The controller was used as a method of interaction with the contents, but other methods have not yet been compared. Therefore, we aim to support the inheritance of tradition and examine the most user-friendly interaction method in terms of operability.

This paper describes the results of a business administration study on what type of corporate governance structure is appropriate for the sustainable development of small and medium-sized companies in Japan. Three research methods were employed: historical research based on endogenous theory, examination of the contents of the current system, and organization of empirical knowledge gained from the author's practical experience. The research findings were that the "KANSEI" of owner-managers is essential to the governance of small and medium-sized companies, and that the current corporate governance code for large corporations needs to be screened considerably to build this governance. Based on the results of this study, we try to draft a "Corporate Governance Code for small and medium-sized companies," suggesting that its proper introduction and implementation will lead to increased corporate value.

It is difficult to design a visually appealing layout for common users, which takes time even for professional designers. In this paper, we present an interactive layout design system with shadow guidance and layout retrieval to help users obtain satisfactory design results. This study focuses in particular on the design of academic presentation slides. The user may refer to the shadow guidance as a heat map, which is the layout distribution of our gathered dataset, using the suggested shadow guidance. The suggested system is data-driven, allowing users to analyze the design data naturally. The layout may then be edited by the user to finalize the layout design. We validated the suggested interface in our user study by comparing it with common design interfaces. The findings show that the suggested interface may achieve high retrieval accuracy while simultaneously offering a pleasant user experience.

In our previous study, we have studying a generation method of a voice that fits animation character, have estimated existing voices from image features of animation characters, but we did not get good results. Therefore, in this research, in order to associate the voice features and image features that match the character, we considered analyzing the tendency of the voice that does not feel strange to the character. As the method, multiple voices are presented for one animation character illustration and evaluated by the participants. In this way, we analyze the tendency of human perception to perceive the characteristics of the animation character's voice.

This paper aims to reduce the animation director's workload when redrawing an original illustration. The job of an animation director is to unify the style of various animators' illustrations. In Japanese TV animation have problems such as a short delivery time and large numbers of corrections; therefore, the animation production team is unable to make corrections in time, and the TV company has to respond by postponing the broadcast. This paper proposes to reduce the animation director's workload using character style transformations using a Globally and Locally Consistent Image Completion (GLCIC) algorithm. This paper proposes character style transformations using a Globally and Locally Consistent Image Completion (GLCIC) algorithm. As a result, this paper was able to translate the illustrations into the directory's illustration style, this method may become a beneficial tool.

Tracking damage to current wind turbine blades over time based on human visual confirmations has issues regarding human cost and hazards during inspections. In this paper, we propose a method to generate 3D alignment frontal views of each part of a wind turbine blade from observed images using multiple 2D barcodes to inspect the damage of the blade and its change over time. Our method consists following processes: (1) 2D barcodes are extracted from the blade surface images, and their normal vectors are measured. (2) 3D surface of the blade is estimated from the barcode normals by ellipsoid model fitting. (3) The observed images are projected and mapped onto the estimated surface overlappingly. Experimental results confirm that the accuracy of the frontal view generation results of this method is sufficient.

In recent years, research on augmented reality (AR) has been active, and eye gaze input has attracted attention as one of the input methods in AR. One problem with gaze input is that it is difficult to input when the target is small or crowded. In response to this problem, the bubble cursor facilitates eye gaze input by changing the size of the circular cursor according to the distance to the nearest target. In addition, by using the distance from the eyeball to the point where the lines of sight of both eyes intersect, called the convergence distance, it is believed that the user's gazing point can be accurately determined without relying on collision determination with virtual objects. In this study, we use the convergence distance to determine the gazing point in an AR environment and consider the bubble cursor centered on the gazing point.

One of the sports training methods is VR training, where users watch a video image to recognize the situation, estimate the timing to move, and move their bodies by the situation. In this paper, we propose a real-time visual feedback method for sports motion information on the VR experience of Alpine skiing. We prepare a carefully designed visual feedback panel that presents the user's center of gravity and head height as sports motion information. An HMD presents the skiing situation on the slopes in VR180 style. The load sensor of our preliminary system is placed under the user's feet, and it acquires the center of gravity position. The tracking function of the HMD estimates head height. In the evaluation experiment, we investigated the appropriate parameters to realize the good visibility of the visual feedback panel during VR training.

The face of a subject who had been using cosmeceuticals for four years and eight months was photographed in ultraviolet light and RGB color. These images, including those taken before the use of the cosmetics, were subjected to deep learning using a neural network model referred to as pix2pix. It was demonstrated that it is possible to predict the effect of the cosmeceutical on the skin and generate images for new subjects using a trained neural network.

One of the most dangerous diseases is an aneurysm because if it ruptures the person will die unless urgent and appropriate treatment is performed. There are mainly two types of aneurysms: cerebral aneurysm (CA) and abdominal aortic aneurysm (AAA). CA needs surgical operations that are selected depending on the symptoms, while medical treatments can be applied to a small AAA to prevent it from expanding, although surgeries are operated for a hypertrophied AAA to prevent it from rupturing. The medical treatments employ micelle which is a colloidal sphere-shaped medicine, which size is too small and about 4 [nm]. The micelle is conveyed to AAA and invades from the vascular lumen to the vascular wall, where little blood flow exists and random force such as Brownian movement is dominant. On the slow flow in the small area, a general governing equation called the Navier-Stokes equation is not used and dissipative particle dynamics (DPD) is employed to solve the flow behavior. Then we performed a micelle behavior simulation in the vascular lumen using DPD. As a result, the micelle was not able to move only with random force and moved by some external force.

In this study, we develop a prototype 2/3-type 4K camera with a prism spectrograph that can capture images at a frame rate of 240 fps. Herein, the simplest configuration uses three 4K 240 fps image sensors corresponding to the three primary colors of light: red, green, and blue. Currently, the fastest 2/3-type 4K image sensor has a frame rate of 120 fps. By using the 2×2 binning function of the image sensor, the imaging speed is accelerated by a factor of four, and the screen resolution is reduced to 1/4. Particularly, we develop the 4K 240 fps camera by adopting the dual-green method, which can restore the 4K resolution by using two sensors for the green channel.

Meniscus height measurement for dry-eye syndrome detection using a smartphone and an LED ring light that is small, inexpensive, and portable. A circular image of the ring light is reflected at the center of the eye, and a distorted crescent-shaped image appears on the tear meniscus where the tears collect. The diameter of the reflected circle on the cornea and height of the crescent-shaped circle reflected on the meniscus are measured through image processing, which includes binarization, circle detection, and removal of unnecessary connected components. The variation in meniscus height caused by the difference in distance between the smartphone and the eye is divided by the diameter and normalized. We further present experimental results that demonstrate that our method is as effective as a system using medical equipment.

Finding the optimal implementation of calculations is one of the most critical challenges in image processing programming. Halide is a domain-specific language for high-performance image processing. Its auto-scheduler is a helpful tool for solving this problem; however, its scheduling is not a panacea for complex flows. In this paper, we evaluate the performance of the auto-scheduler by comparing it to hand-manually implemented C++ codes. The algorithm used for comparison is Directional cubic convolution interpolation (DCCI), whose computation schedule is challenging to optimize. We evaluate three auto-schedulers: Adams et al.’s, Li et al.’s, and Mullapudi et al.s. Experimental results show that the performance of the schedule generated by Adams’ method is comparable to that of the hand-implemented C++ code.

A physical 3D figure model of human jaw was made using a 3D printer. The model is used for a research of human speech production mechanisms and for educational purposes of speech science. To design 3D geometrical data of jaws, 3D computer graphics software is used. The modeling data are stored in STL (Stereolithography) format and then, they are transferred to a 3D printer and figure models are made. The models are used to teach speech production mechanisms in a university class.

We propose a "switchable dual dial" method for Japanese text input by flick input using two VR controllers in a VR environment. The proposed method is to input consonants by flicking the touch pad of one VR controller with one finger. Next, a vowel is selected by flicking the touchpad of the other VR controller with the other finger to input the letter. Since the above two operations can be performed almost simultaneously, the proposed method is expected to be accurate and quick. This paper reports on an additional 7-day experiment that verified the improvement of the input performance of the proposed method as the user becomes proficient in the operation.

This study proposes a method for detecting micro-vibrations along the optical axis of a target object using video image analysis. The proposed method detects vibrations as time variances in the scaling factor of the object by checking the shift in the peaks in the spatial power spectrum of the images. We conducted an experiment wherein a checkerboard placed 2 m below the video camera was vibrated using an actuator. The results show that the proposed method could measure the vibration frequency even when the amplitude is 0.25 permille of the shooting distance.

In this paper, we investigate a simple movable projection mapping system which can reduce distortion and fluctuation of a projected image by assisting a gimbal device by geometric image transformation. Since a conventional gimbal device can compensate the movement of a mounted projector only, we design a new gimbal device which can compensate the relative geometrical relationship between the projector and the screen as well. The proposed method stabilizes a projected image using the newly designed gimbal which tracks the screen with a visual marker placed at each vertex and transforms the shape of image. Through an experiment, we evaluate the performance of the proposed projected image stabilization technique.

This study proposes a motor design and presentation method for presenting virtual force sensation in the three-dimensional direction. Various grasping methods and device performances need to be considered in order to present virtual force perception in three-dimensional directions. However, commercially available motors do not allow changing parameters such as the surface magnetic flux density of the magnets inside and the number of turns of the coils, and the shape of the motor affects the design of the device. Therefore, we created motors with the same amplitude and shape for each device. By grasping that motor with the side of the fingertip, the directional presentation of virtual force sensation in the vertical direction was possible, showing its potential for 3D directional presentation.

Writing image processing in WebAssembly enables the development of highly portable libraries. However, since WebAssembly is a new technology, there are not enough libraries available, and it is only supported by OpenCV. However, OpenCV has not been optimized specifically for WebAssembly. Therefore, in this paper, we develop native code using WebAssembly and compare it with OpenCV’s Gaussian filter using the separable Gaussian filter, a classical acceleration method for Gaussian filters. Experimental results showed that the Separable Gaussian filter was faster than OpenCV by performing vector operations in WebAssembly.

Costumes are one of the major elements of the excitement in a live music performance. Although it is ideal to change the costumes to match the music theme each time, it is not practical because of the time and effort required to change the costume. In this paper, we propose a dynamic projection mapping method onto the costumes instead of “real” costume changes. Our method generates mapping images in advance, which are deformed and/or occluded in accordance with the dance choreography. Then, the projection positions are slightly adjusted in real time based on the difference between expected and actual standing position. This technique enables dynamic projection mapping onto a non-rigid object. Several experimental results verified the feasibility of the dynamic projection mapping.

The Cross Component Linear Model (CCLM) is an intra prediction tool for chroma that was adopted in Versatile Video Coding (VVC) standard. It generates an intra-predictor by utilizing the linear correlation existing between the current chroma block and its co-located reconstructed luma region. However, the predictor generated by CCLM can potentially have discontinuity with its adjacent regions since it does not consider the adjacent samples during its predictor generation. So, we propose a new method called CCLM Combined with Decoder-side Intra Mode Derivation (CCDIMD) to overcome this problem. The key idea of CCDIMD is a weighted combination of the predictor generated by CCLM and the predictor generated by an intra-prediction mode inferred using the intra mode derivation process of DIMD. Our proposed method shows BDBR gains respectively of -0.02%, -1.05%, and -0.96% in Y, Cb, and Cr for 4:2:0 color format videos; -0.03%, - 0.34%, and -0.29% in Y, Cb, and Cr for 4:2:2 color format videos; and -0.10%, -0.40%, and -0.53% in Y, Cb, and Cr for 4:4:4 color format videos.

Multiple reference line (MRL) intra prediction allows to use reference lines not necessarily immediately adjacent neighboring line or column in generating intra predictor. Compared to MRL already existing in the versatile video coding (VVC) standard which can use a total of three reference lines including the adjacent one, the MRL under discussion now in enhanced compression beyond VVC is extended to use a total of six reference lines. However, its MRL candidates are listed in a fixed order for all blocks without considering non-identical effectiveness of each reference line in intra prediction. In this paper, we propose a reordering scheme of the MRL candidate lines in the MRL list considering similarities between predictors generated by available reference lines. According to our experimental results, the proposed method is shown to achieve an overall coding gain of -0.02% in luma channel over the enhanced compression model (ECM) 5.0 under the all intra (AI) configuration of the common test condition (CTC).

Rate-distortion optimized quantization for transform skip (RDOQ-TS) is shown to provide positive coding gain in VVC. This paper reduces encoding time-complexity of the RDOQ-TS scheme in VVC by adaptively skipping its level estimation (LE) process without sacrificing coding efficiency much. At first, the characteristics of each quantized level by RDOQ-TS are analyzed statistically. Subsequently, based on the statistical analysis, the LE process in the RDOQ-TS process is simplified by removing the case where the SQ (Scalar Quantization) value and the finally determined value are highly expected to be the same. For 4:2:0 screen content videos and 4:4:4 screen content videos, our proposed simplified rate-distortion optimization for transform skip reduces total encoding time respectively by 0.77% and 1.83% with no BDBR change under all intra configuration. In the case of the random access configuration, it achieves bit rate saving by -0.01 % and -0.03 % with total encoding time reduced 0.94% and 0.52% respectively.

The purpose of this study was to develop a computerized classification method for molecular subtypes of low-grade gliomas (LGGs) in brain MRI (magnetic resonance imaging) images using multi-scale three-dimensional attention branch networks (MS3D-ABNs) with an additive angular margin penalty. Our database consisted of brain T1-weighted, T2-weighted, and FLAIR MRI images for 217 patients (58 IDH-mutant astrocytomas, 49 IDH-wildtype astrocytomas, and 110 oligodendrogliomas). The proposed network was constructed from a feature extractor, an attention branch, and a perception branch with an additive angular margin penalty. The feature extractor first extracted the feature maps of different resolutions from brain T1-weighted, T2-weighted, and FLAIR MRI images, respectively. The attention branch generated attention maps focusing on a tumor region. The feature maps were then multiplied by the attention maps to weight features on the tumor region in the feature maps. The perception branch finally evaluated the molecular subtypes of LGGs by determining the cosine similarity between the feature vector obtained from applying a global average pooling to the feature map and the representative vector of each molecular subtype class. In training the proposed network, an angular margin penalty was added to the angle between the feature vector of input image and the representative vector of the same class as the input image to make those vectors to be closer each other. The classification accuracy for the proposed network was 66.4%, showing an improvement when compared to the MS3D-ABNs without the additive angular margin penalty (60.4%).

Intracerebral hematoma (ICH) is a blood clot that forms when a blood vessel in the brain ruptures for some reason and the spilled blood coagulates. ICH has a high morbidity and mortality rate, accounting for approximately 10% of all strokes. Manual segmentation of ICH in head CT images is very complicated, time consuming, and troublesome. When ICH perforates the ventricular wall and blood flows into the ventricle, there is little difference in CT value between ICH and intraventricular hemorrhage (IVH), and the boundary between them is unclear. Convolutional neural network (CNN) has proven to be a reliable method in the field of image recognition. In addition, quantification of ICH may aid in decision making in ICH treatment. In this study, we introduce CNN in a stepwise manner to differentiate ICH and IVH in the process and extract ICH regions. The results in 18 stroke patients show that our method is promising in the extraction of ICH regions with an accuracy of 75.2%.

A typical unsupervised domain adaptation (UDA) task assumes that labeled source data is available for model adaptation. However, this assumption is often infeasible due to confidentiality issues or memory constraints on mobile devices. To address these problems, we propose a simple yet effective source-free UDA method. Motivated by self-training methods, the proposed method constrains multiple views of the same image to have similar features and prediction outputs. The feature generator is encouraged to learn consistent visual features that are away from the decision boundaries of the head classifier. On popular benchmarks, our approach demonstrates comparable or even superior performance to vanilla UDA methods without using source images.

3D holistic human mesh reconstruction has gained attention in the computer vision field. However, the absence of an in-the-wild training dataset with 3D annotations, makes this field more challenging. There is a large-scale synthetic dataset called AGORA to address this problem. Since this dataset is still synthetic, there is still a gap to deal with the real scenario. In this paper, we introduce a semi-supervised approach to bridge the gap between synthetic and real data. Our experiment shows that our approach could improve in terms of the quantitative and qualitative results.