LET: Vision Transformer based Refinement Network for Light Field Editing

The LF editing through propagation enables temporal change of the photographed virtual space. Existing LF propagation schemes are largely divided into two types. One is based on image warping. It moves the pixels of the updated area to other images in the LF. Although there is little change in the pixel value itself, artifacts such as speckling and distortion often occur. The other approach synthesizes images based on a convolutional neural network (CNN). However, this method can only partially observe the characteristics of the image due to the local receptive field of CNN, and the output result is easily blurred while creating an image with down-sampled features. To overcome the limitations of conventional techniques, this paper proposes a vision transformer based LET model which consists of two steps. First, an initial edited LF with minimal change in pixel values is generated by propagating the updated region to other images using the traditional forward warping technique. Second, the visual quality is consistently improved through the refinement network which is based on the dense prediction transformer (DPT). In the warping process of the first step, approximate propagation is performed minimizing the loss of pixel values. Then, the angular consistency of the LF is maintained based on the global information in the refinement network of the second step. Experimental results show that the proposed LF editing scheme achieves significant improvement both quantitatively and subjectively.