Ray Deformation Networks for Novel View Synthesis of Refractive Objects
Weijian Deng1,
Dylan Campbell1,
Chunyi Sun1,

Shubham Kanitkar2,
Matthew E. Shaffer2,
Stephen Gould1,
1Australian National University, 2RIOS Intelligent Machines

- NeRF methods learn the density field based on light transports along straight path
- When light paths intersect refractive objects, they may curve (dashed line), depending on the angle of incidence
- We propose to bend the light rays by predicting position and direction offsets for sample points along the rays


Neural Radiance Fields (NeRF) have demonstrated exceptional capabilities in creating photorealistic novel views using volume rendering on a radiance field. However, the intrinsic assumption of straight light rays within NeRF becomes a limitation when dealing with transparent or translucent objects that exhibit refraction, and therefore have curved light paths. This hampers the ability of these approaches to accurately model the appearance of refractive objects, resulting in suboptimal novel view synthesis and geometry estimates. To address this issue, we propose an innovative solution using deformable networks to learn a tailored deformation field for refractive objects. Our approach predicts position and direction offsets, allowing NeRF to model the curved light paths caused by refraction and therefore the complex and highly view-dependent appearances of refractive objects. We also introduce a regularization strategy that encourages piece-wise linear light paths, since most physical systems can be approximated with a piece-wise constant index of refraction. By seamlessly integrating our deformation networks into the NeRF framework, our method achieves significant improvements in rendering refractive objects from novel views.



Ray Deformation Networks


Improved Novel View Synthesis

Paper and Supplementary Material

Deng W, Campbell D, Sun C, Kanitkar S, Shaffer M, Gould S.
Ray Deformation Networks for Novel View Synthesis of Refractive Objects.
In WACV, 2024.
(hosted on [Paper])


Correspondence to {firstname.lastname}@anu.edu.au


This template was originally made by Phillip Isola and Richard Zhang for a colorful ECCV project; the code can be found here.