13 June 2023 Densely sampled light field reconstruction with transformers
Xiyao Hua, Minghui Wang, Boni Su, Zhenjiang Liu, Peng Fan
Author Affiliations +
Abstract

Densely sampled light fields (LFs) are critical for their further applications, such as digital refocus and depth estimation. However, it is costly and time-consuming to capture them. LF reconstruction, which aims at reconstructing a densely sampled LF from a sparsely sampled one, has attracted extensive attention of researchers. Although existing methods have achieved significant progress, these methods synthesize novel views, either through depth estimation and image warping, which depend heavily on the accuracy of the depth maps and are prone to cause artifacts at occluded regions, or by stacking multi-layer convolutions to learn the inherent structure of the LF, which will result in blurring results due to limited receptive fields when processing scenes with large disparities. We propose a transformer-based neural network for LF reconstruction (termed as LFRTR). Specifically, two novel transformers are introduced, namely angular transformer and spatial transformer. The former can fully explore angular information and correlations among different views, whereas the latter can capture local and non-local spatial texture information within each view. Moreover, dense skip connections are employed to enhance information flow between different layers. Thanks to the inherent global modeling ability of self-attention, the proposed LFRTR can reconstruct high-quality densely sampled LF in complex scenarios, such as large disparity, occlusion, and reflection. Experimental results on both synthetic and real-world LF datasets show that the proposed LFRTR outperforms other state-of-the-art methods in terms of both visual and numerical evaluations.

© 2023 SPIE and IS&T
Xiyao Hua, Minghui Wang, Boni Su, Zhenjiang Liu, and Peng Fan "Densely sampled light field reconstruction with transformers," Journal of Electronic Imaging 32(3), 033025 (13 June 2023). https://doi.org/10.1117/1.JEI.32.3.033025
Received: 29 December 2022; Accepted: 30 May 2023; Published: 13 June 2023
Advertisement
Advertisement
RIGHTS & PERMISSIONS
Get copyright permission  Get copyright permission on Copyright Marketplace
KEYWORDS
Transformers

Image restoration

Convolution

Feature extraction

Education and training

Super resolution

Windows

RELATED CONTENT


Back to Top