Skip to yearly menu bar Skip to main content


Poster

MVSplat360: Benchmarking 360$\textdegree$ Generalizable 3D Novel View Synthesis from Sparse Views

Yuedong Chen · Chuanxia Zheng · Haofei Xu · Bohan Zhuang · Andrea Vedaldi · Tat-Jen Cham · Jianfei Cai

[ ]
Fri 13 Dec 11 a.m. PST — 2 p.m. PST

Abstract: We introduce MVSplat360, a feed-forward approach for 360$\textdegree$ novel view synthesis (NVS) in wild scene scenarios given only sparse observations. This task of generalizable 360$\textdegree$ scene reconstruction from sparse views is challenging and ill-posed. Existing methods fail to achieve plausible 360$\textdegree$ scene reconstruction from such sparse observations due to insufficient information to recover the entire scene and the minimal overlap between given views. Therefore, our MVSplat360 takes an initial step toward addressing these challenges, which first matches and fuses view information through a cross-view transformer encoder, then constructs a coarse 3D geometry using the latest 3D Gaussian Splatting, and finally refines invisible and inconsistent appearances with a pre-trained stable video diffusion model. We construct a new benchmark using the challenging DL3DV dataset, where MVSplat360 significantly outperforms prior works in wide-sweeping and even 360$\textdegree$ NVS from sparse image observations. We also conduct extensive comparisons on the existing RealEstate10K benchmark, further demonstrating the efficacy of our method.

Live content is unavailable. Log in and register to view live content