NerfingMVS

NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo

Authors: Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, Jie Zhou

2 September, 2021

https://weiyithu.github.io/NerfingMVS/

https://arxiv.org/abs/2109.01129

ICCV 2021 (Oral)

Abstract: In this work, we present a new multi-view depth estimation method that utilizes both conventional reconstruction and learning-based priors over the recently proposed neural radiance fields (NeRF). Unlike existing neural network based optimization method that relies on estimated correspondences, our method directly optimizes over implicit volumes, eliminating the challenging step of matching pixels in indoor scenes. The key to our approach is to utilize the learning-based priors to guide the optimization process of NeRF. Our system firstly adapts a monocular depth network over the target scene by finetuning on its sparse SfM+MVS reconstruction from COLMAP. Then, we show that the shape-radiance ambiguity of NeRF still exists in indoor environments and propose to address the issue by employing the adapted depth priors to monitor the sampling process of volume rendering. Finally, a per-pixel confidence map acquired by error computation on the rendered image can be used to further improve the depth quality. Experiments show that our proposed framework significantly outperforms state-of-the-art methods on indoor scenes, with surprising findings presented on the effectiveness of correspondence-based optimization and NeRF-based optimization over the adapted depth priors. In addition, we show that the guided optimization scheme does not sacrifice the original synthesis capability of neural radiance fields, improving the rendering quality on both seen and novel views. Code is available at https://github.com/weiyithu/NerfingMVS. △ Less

文摘：在这项工作中，我们提出了一种新的多视点深度估计方法，该方法利用了最近提出的神经辐射场（NeRF）的传统重建和基于学习的先验知识。与现有基于神经网络的优化方法依赖于估计的对应关系不同，我们的方法直接对隐式体积进行优化，消除了室内场景中像素匹配的挑战性步骤。该方法的关键是利用基于学习的先验知识来指导神经网络的优化过程。我们的系统首先通过对COLMAP稀疏SfM+MVS重建进行微调，在目标场景上采用单目深度网络。然后，我们证明了NeRF的形状辐射模糊性在室内环境中仍然存在，并建议通过采用自适应深度先验来监控体绘制的采样过程来解决这个问题。最后，通过对渲染图像进行误差计算获得的每像素置信度图可用于进一步改善深度质量。实验表明，我们提出的框架在室内场景中的性能明显优于现有的方法，基于通信的优化和基于NeRF的优化在自适应深度先验上的有效性令人惊讶。此外，我们还表明，引导优化方案不会牺牲神经辐射场的原始合成能力，从而提高了可见视图和新视图的渲染质量。代码位于https://github.com/weiyithu/NerfingMVS