MixNeRF Modeling a Ray with Mixture Density for Novel View Synthesis from Sparse Inputs

Abstract

Neural Radiance Fields (NeRF) deliver state-of-the-art novel-view synthesis but degrade severely without dense training views. Prior fixes lean on heavy extra training resources, undermining the efficiency premise of sparse-input NeRF. We propose MixNeRF, an effective training strategy that models each ray with a mixture density model. MixNeRF estimates the joint distribution of RGB along ray samples as a mixture of distributions, and introduces ray depth estimation as a useful training objective tightly correlated with 3D scene geometry. With regenerated blending weights from estimated depth, robustness to color and viewpoint shifts further improves. MixNeRF outperforms prior state-of-the-art across standard benchmarks while being more training- and inference-efficient.

Video

Modeling a ray with a mixture density model

MixNeRF estimates the per-pixel RGB distribution along ray samples as a mixture of component distributions. The conventional NeRF outputs at each sample serve as a location parameter and (via the density σ) as the mixing coefficient π; we additionally estimate a scale parameter β.

The pdf of the mixture model is:

The mixture coefficient is derived from the density output:

Depth estimation by mixture density

We propose ray depth estimation as an auxiliary task — defined as the length of the unnormalized ray-direction vector along ray samples — providing direct supervision for the mixture parameters tied to geometry.

With π and β jointly optimized via depth and color supervision, MixNeRF becomes more robust to small geometric shifts. Estimated depth then regenerates blending weights — playing the role of pseudo-geometry for adjacent samples without extra training data:

Color values along the ray are remodeled with the new mixing coefficients:

No extra inference of unseen views is required, so robustness is gained without meaningful computational overhead.

Why mixture density helps

For unimodal cases (blue), mip-NeRF underestimates the mode and yields degenerate geometry. RegNeRF and MixNeRF recover unimodal weights, with MixNeRF producing a sharper mode that more closely matches the dense-view reference. For bimodal cases (red), only MixNeRF estimates the distribution successfully — directly improving geometry under limited views.

Depth map estimation

RegNeRF's smoothing prior collapses geometry. MixNeRF learns ray depth via the mixture density model — without smoothing across additional unseen rays — so depth maps come out cleaner and more precise.

Efficiency

DietNeRF trains in similar time but is significantly weaker in 3- and 6-view settings. Against RegNeRF, MixNeRF cuts per-scene training time by ~42% under matched view counts by eliminating extra inference for additional unseen rays.

Results

DTU 3-view

LLFF 3-view

Blender 4-view

Citation

@InProceedings{Seo_2023_CVPR,
  author    = {Seo, Seunghyeon and Han, Donghoon and Chang, Yeonjin and Kwak, Nojun},
  title     = {MixNeRF: Modeling a Ray With Mixture Density for Novel View Synthesis From Sparse Inputs},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month     = {June},
  year      = {2023},
  pages     = {20659-20668}
}

Acknowledgements

This work was supported by NRF grant (2021R1A2C3006659) and IITP grants (2021-0-01343, 2022-0-00953) funded by the Korean Government.