DReg-NeRF: Deep Registration for Neural Radiance Fields

National University of Singapore
Interpolation end reference image.

DReg-NeRF registers multiple NeRFs into a global coordinate frame.

Abstract

Although Neural Radiance Fields (NeRF) is popular in the computer vision community recently, registering multiple NeRFs has yet to gain much attention. Unlike the existing work, NeRF2NeRF, which is based on traditional optimization methods and needs human annotated keypoints, we propose DReg-NeRF to solve the NeRF registration problem on object-centric scenes without human intervention. After training NeRF models, our DReg-NeRF first extracts features from the occupancy grid in NeRF. Subsequently, our DReg-NeRF utilizes a transformer architecture with self-attention and cross-attention layers to learn the relations between pairwise NeRF blocks. In contrast to state-of-the-art (SOTA) point cloud registration methods, the decoupled correspondences are supervised by surface fields without any ground truth overlapping labels. We construct a novel view synthesis dataset with 1,700+ 3D objects obtained from Objaverse to train our network. When evaluated on the test set, our proposed method beats the SOTA point cloud registration methods by a large margin.

Interpolation end reference image.

Dataset Overview

Interpolation end reference image.

Training images are rendered from 3D objects (~30 categories, 1700+ objects) of the Objaverse dataset (800K+ 3D models with descriptive captions, tags).

Results Animation

Top: Images rendered after registration with ground-truth transformation;
Middle: Images rendered after registration with our estimated transformation;
Bottom: Images rendered without registration.


Odeon Lightable

Re-rendering the images after registration

rendered image rendered depth rendered image rendered depth



Camera poses before/after registration

Before Registration After Registration

Banana for Sale

Re-rendering the images after registration

rendered image rendered depth rendered image rendered depth



Camera poses before/after registration

Before Registration After Registration

Figura San Juan

Re-rendering the images after registration

rendered image rendered depth rendered image rendered depth



Camera poses before/after registration

Before Registration After Registration