DONeRF: Towards Real‐Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks

DoNeRF was quite an interesting journey for me. This entire field was completely new to me, but fortunately, NeRFs were new to everyone at the time we started with this project.

NeRFs were exciting at that time. It’s the first time one could create photorealistic and view-consistent renderings of real scenes at this quality. But NeRFs were still slow, requiring hundreds of expensive MLP evaluations per pixel and many hours of training per scene.

Our team did what we always do - make things go faster. While faster training would be nice too, we focused on making rendering as fast as possible. One obvious flaw of the original NeRF render was that their ray marching implementation sampled at evenly spaced positions along the rays in view direction - at least in the coarse model. And then they sampled many more positions around dense areas. Evaluating the coarse MLP >100x per pixel takes a lot of time. So, instead, we tried to estimate the distance from camera to the surface - one MLP evaluation per pixel instead of >100x. Our initial experiments using a single depth per pixel were promising, but prone to errors on semi-transparent surfaces as well as object borders. The renderings looked aliased, because we could only sample a single depth per pixel.

To solve this, we instead went for a multi-depth prediction, essentially using a classifier for the depth ranges along the ray. The ranges with the highest score would be evaluated, while the rest is discarded. The number of ranges to be evaluated can be increased for higher quality or decreased for higher frame rate. With only 2-4 samples per ray, DONeRF achieves higher quality than original NeRF while rendering 15-78x faster.

This is (at the time of writing) by far the most cited paper I was part of and also the only paper during the course of my PhD which included collaboration across team members. Looks like team work really does lead to better results - especially in such a busy area as NeRFs were back then. That said, I did not stay until the end of this project, because I started working on DeltaCNN a few months before submission of DONeRF.