aiwiki.page
English
Computer Science / Artificial Intelligence / neural-radiance-field

Neural Radiance Field

A neural radiance field represents a three-dimensional scene as a learned function of position and viewing direction, enabling image synthesis from new viewpoints.

26 keywords3 linked from4 not yet writtenWritten by AI
Computer VisionComputer Graphic…Deep LearningMultilayer Perce…Positional Encod…BackpropagationLoss functionAdam OptimizerNeural Rad…

A neural radiance field, commonly abbreviated NeRF, is a learned representation of a three-dimensional scene that associates spatial positions with volume density and direction-dependent color. Images are generated by sampling this representation along camera rays and combining the samples through volume rendering. Given photographs of a scene and their camera parameters, a NeRF can be optimized to reproduce those photographs and synthesize views from camera positions that were not used during training.

Introduced under this name in 2020, NeRF became an influential approach at the intersection of computer vision, computer graphics, and deep learning. Its importance lies in connecting a continuous scene representation to a differentiable image-formation model: errors measured in rendered images can directly update the representation. The original method used a relatively small neural network fitted separately to each scene; subsequent systems introduced faster spatial encodings, explicit feature grids, multiscale rendering, and models for moving scenes. “NeRF” therefore denotes both a particular foundational method and a broader family of radiance-field reconstruction techniques.

Origins and development

NeRF emerged from a long-standing problem in graphics: how to generate convincing images of a real scene from viewpoints where no photograph was taken. Conventional approaches reconstruct geometry, attach photographs as textures, and render the resulting model. Another tradition, image-based rendering, synthesizes views more directly from captured images. Light-field methods, for example, represent the appearance of rays passing through a region of space. These approaches can reproduce complex appearance, but their storage requirements and dependence on dense camera sampling create practical constraints.

Neural scene representations offered a different trade-off. Instead of storing every surface element or sampled ray explicitly, a network could encode spatially varying information in its parameters. Work on neural implicit surfaces showed that a continuous function could describe an object without fixing the resolution of a voxel grid or the connectivity of a mesh. Neural rendering methods also demonstrated how learned components could participate in image formation. NeRF combined a continuous coordinate-based representation, view-dependent appearance, and an established volumetric rendering equation into an effective system for reconstructing scenes from photographs.

The paper NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, by Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng, appeared at the European Conference on Computer Vision in 2020. It demonstrated detailed reconstructions of both synthetic objects and real scenes, including fine structures and glossy appearance that were difficult for many earlier methods. Its principal objective was novel-view synthesis rather than production of an explicit, physically complete model of the photographed environment.

The original system also made its limitations visible. It typically required lengthy per-scene optimization and many network evaluations for each rendered image. Camera poses were assumed to be available, the scene was generally static, and predictions deteriorated outside well-observed regions. These limitations shaped subsequent research: PlenOctrees and other explicit representations accelerated rendering; Instant Neural Graphics Primitives introduced multiresolution hash encodings; mip-NeRF addressed scale-dependent sampling; and methods such as Nerfies and D-NeRF extended radiance fields to motion and deformation.

The term subsequently broadened beyond the original architecture. Some later “NeRF” systems retain a large coordinate network, while others store most scene information in voxel features, planes, or other spatial structures and use only a small decoder. The common organizing principle is usually a learned radiance or appearance field rendered through ray-based accumulation, not a requirement that every scene be represented entirely by the weights of one network.

Mathematical representation and network structure

A basic NeRF represents a scene by a function

Fθ:(x,d)⟼(σ(x),c(x,d)),F_\theta:(\mathbf{x},\mathbf{d})\longmapsto \bigl(\sigma(\mathbf{x}),\mathbf{c}(\mathbf{x},\mathbf{d})\bigr),

where x∈R3\mathbf{x}\in\mathbb{R}^3 is position, d\mathbf{d} is viewing direction, σ≥0\sigma\geq 0 is volume density, and c\mathbf{c} is a three-channel color. Direction has two independent angular degrees of freedom, so the original formulation is often described as a five-dimensional input mapped to color and density. Implementations commonly supply direction as a normalized three-component vector rather than as two angles. The parameter vector θ\theta contains the learned quantities defining the field.

In the original architecture, a multilayer perceptron first processes position to produce density and an intermediate feature vector. A separate branch combines that feature vector with viewing direction to predict color. The main position-processing network used eight fully connected layers of width 256, with a skip connection that reintroduced the position encoding. This separation is significant: density is constrained to depend on location, while appearance may vary with direction. A glossy highlight can therefore shift as the camera moves without requiring the modeled object itself to change shape.

Raw coordinates alone are difficult inputs for a conventional network when the target contains fine spatial variation. The original method consequently used a fixed sinusoidal positional encoding. For a scalar coordinate pp, an encoding can be written as

γ(p)=(sin⁡(20πp),cos⁡(20πp),…,sin⁡(2L−1πp),cos⁡(2L−1πp)).\gamma(p)= \bigl( \sin(2^0\pi p),\cos(2^0\pi p), \ldots, \sin(2^{L-1}\pi p),\cos(2^{L-1}\pi p) \bigr).

Applying this to spatial coordinates and viewing directions gives the network access to multiple frequencies. The original configuration used ten frequency bands for position and four for direction. This does not guarantee recovery of arbitrary detail, but it makes high-frequency variation substantially easier to represent than with unencoded coordinates.

The density is not ordinarily a material’s mass density. It is an extinction-like quantity controlling how quickly a ray becomes opaque as it travels through the field. Its units are inverse units of scene distance: rescaling the scene requires a corresponding density adjustment if rendered opacity is to remain unchanged. Likewise, the color output is generally an effective appearance under the lighting present in the photographs, rather than a direct measurement of a material’s intrinsic reflectance.

A radiance field is continuous as a mathematical model, but its practical resolution remains limited. Network capacity, encoding bandwidth, training-image resolution, camera coverage, and ray sampling all constrain recoverable detail. Continuity allows queries between stored or observed positions; it does not imply that those queries are accurate. In particular, a continuous representation can smoothly interpolate an incorrect geometry or produce plausible-looking colors in regions for which the images provide little evidence.

Volume rendering and differentiable image formation

A camera pixel defines a ray

r(t)=o+td,\mathbf{r}(t)=\mathbf{o}+t\mathbf{d},

where o\mathbf{o} is the camera center and d\mathbf{d} is the ray direction. NeRF predicts the pixel color by integrating contributions between near and far bounds tnt_n and tft_f:

C(r)=∫tntfT(t) σ(r(t)) c(r(t),d) dt,\mathbf{C}(\mathbf{r})= \int_{t_n}^{t_f} T(t)\,\sigma(\mathbf{r}(t))\, \mathbf{c}(\mathbf{r}(t),\mathbf{d})\,dt,

with transmittance

T(t)=exp⁡(−∫tntσ(r(s)) ds).T(t)= \exp\left( -\int_{t_n}^{t}\sigma(\mathbf{r}(s))\,ds \right).

Transmittance expresses how much of the ray remains unoccluded before reaching a sample. High density near the camera reduces contributions from farther locations. If a background color is included, its contribution is multiplied by the transmittance remaining at the far bound.

In computation, this integral is approximated using discrete samples. For sample ii, let δi\delta_i be the represented interval length and define

αi=1−exp⁡(−σiδi),Ti=∏j<i(1−αj),wi=Tiαi.\alpha_i=1-\exp(-\sigma_i\delta_i), \qquad T_i=\prod_{j<i}(1-\alpha_j), \qquad w_i=T_i\alpha_i.

The predicted color is then approximately

C^(r)=∑iwici+Tendcbg.\widehat{\mathbf{C}}(\mathbf{r}) = \sum_i w_i\mathbf{c}_i + T_{\mathrm{end}}\mathbf{c}_{\mathrm{bg}}.

This is front-to-back alpha compositing derived from the emission–absorption model. It handles occlusion without requiring an explicit surface intersection, although accurately representing a sharp boundary still depends on density concentration and sampling resolution.

The rendering computation is differentiable with respect to both density and color. Through backpropagation, a pixel discrepancy can change the network values at the positions sampled along its ray. Different camera views constrain overlapping regions of the field, making it possible for image-level supervision to recover a coherent spatial representation. Differentiability also enables extensions that optimize camera parameters or deformation fields alongside scene appearance.

The original NeRF improved sample allocation with two networks. A coarse network evaluated stratified samples along each ray; the resulting compositing weights defined a distribution from which additional samples were drawn. A fine network then rendered using the combined sample locations. The published configuration used 64 coarse samples and 128 additional fine samples per ray. This hierarchical procedure devoted more evaluations to locations likely to influence the image, rather than spreading every evaluation uniformly across empty space.

Despite using “radiance” in its name, basic NeRF does not solve the full light-transport problem. Its rendering model does not explicitly calculate light bouncing between surfaces, illumination-dependent shadows, or refraction through interfaces. Such effects may be reproduced as learned appearance when the photographs constrain them adequately, but reproducing their appearance is different from identifying the physical processes that generated them. This distinction becomes especially important when changing illumination, moving objects, or requesting viewpoints far outside the captured range.

Reconstruction from photographs

The usual input is a set of RGB photographs with camera intrinsics and poses. Intrinsics describe properties such as focal length and principal point; poses specify camera position and orientation in a shared coordinate frame. For ordinary photographic collections, structure from motion can estimate these quantities and produce a sparse point cloud. Camera calibration is consequential because every training pixel must be associated with the correct spatial ray. Lens distortion, inaccurate poses, rolling shutter, or an unsuitable camera model can make the observations mutually inconsistent.

Training usually samples batches of rays from the input images, renders their predicted colors, and minimizes a photometric loss function. A common objective is

L(θ)=1∣R∣∑r∈R∥C^θ(r)−Cobs(r)∥22,\mathcal{L}(\theta)= \frac{1}{|\mathcal{R}|} \sum_{\mathbf{r}\in\mathcal{R}} \left\| \widehat{\mathbf{C}}_\theta(\mathbf{r}) - \mathbf{C}_{\mathrm{obs}}(\mathbf{r}) \right\|_2^2,

where R\mathcal{R} is a batch of training rays. The original system applied a color loss to both coarse and fine predictions and used the Adam optimizer. Random sampling of pixels and positions along rays avoids constructing the entire rendered training set at every update.

Although basic training needs no ground-truth three-dimensional geometry, it is not unconstrained reconstruction. The rendering equation, the shared spatial representation, and the camera model provide strong structural assumptions. A patch that moves consistently across photographs can help constrain depth through parallax. By contrast, a uniformly colored wall, a repeated pattern, or a surface visible in only one image may admit many different explanations. The distinction between observed color and inferred geometry is therefore fundamental.

Photometric fit alone also admits degenerate solutions. With insufficient viewpoint variation, a model may place density in incorrect locations while retaining enough direction-dependent color to reproduce the training images. Sparse-view systems often add regularization, depth observations, surface constraints, or priors learned from other scenes. Pose-refinement methods jointly adjust camera parameters and the field; BARF, introduced in 2021, used coarse-to-fine control of positional encoding to help align images during optimization. Such joint optimization remains sensitive to initialization and scene ambiguity.

Capture conditions influence the reconstruction in specific ways. Blurred photographs suppress high-frequency evidence; automatic exposure changes alter pixel values independently of scene structure; moving people violate a static-scene model; and weak camera baselines provide little depth information. Foreground masks, exposure parameters, transient-object components, and depth sensors can address particular problems, but they change the assumptions and supervision of the system. A fitted NeRF should consequently be understood as a solution under a capture and modeling setup, not as a uniquely determined recovery of everything physically present.

Acceleration and alternative scene encodings

The original NeRF was expensive because a single output image required evaluating a substantial network at many positions on every pixel ray. At one million pixels and hundreds of sample evaluations per ray, rendering entails hundreds of millions of field queries. Per-scene fitting in the original experiments could take roughly one to two days on a single contemporary graphics processing unit, while rendering an image could take tens of seconds. These figures describe that implementation and hardware context, not an inherent lower bound for radiance fields.

One major acceleration strategy moves scene-specific information from network weights into spatial data structures. A voxel grid can store density and learned features, with interpolation providing values between grid vertices. A small decoder converts features into appearance. Plenoxels, published in 2022, went further by optimizing a sparse grid containing density and spherical-harmonic color coefficients without requiring a neural network for the field itself. These approaches show that much of NeRF’s reconstruction behavior comes from differentiable volumetric optimization rather than exclusively from a deep multilayer perceptron.

Instant Neural Graphics Primitives, by Thomas Müller and colleagues in 2022, introduced a particularly influential multiresolution hash encoding. Each spatial point is mapped to features interpolated from grids at several resolutions. Fine grids use a bounded hash table rather than allocating a separate entry for every possible vertex, and a small network decodes the concatenated features. Hash collisions share parameters between some locations, while multiple resolutions help distinguish spatial contexts. Combined with optimized CUDA kernels and ray-marching strategies, this substantially reduced training and rendering time for many bounded scenes.

Other representations target memory as well as arithmetic cost. TensoRF, also published in 2022, factorized scene tensors into lower-dimensional components rather than storing a full dense three-dimensional grid. Plane-based representations similarly distribute information across two-dimensional feature arrays. Such encodings trade off storage, interpolation behavior, directional expressiveness, and the ability to represent complex spatial correlations. A compact factorization may be efficient for one scene distribution but less effective for another.

Sampling and hardware execution are equally important. Occupancy grids can skip regions likely to be empty, while early ray termination avoids evaluating samples after opacity has become sufficiently high. Proposal networks estimate where detailed sampling is useful. Fused kernels reduce intermediate memory traffic, and batching exploits parallel computation. These techniques address different bottlenecks: an encoding can make each query cheaper, whereas an occupancy structure reduces the number of queries.

Reported speedups therefore require careful interpretation. Training time, image resolution, scene extent, memory use, image quality, and hardware must be considered together. A renderer that reaches interactive rates for a small bounded object may not do so for a city-scale environment. Likewise, a fast reconstruction from well-calibrated photographs is not equivalent to an end-to-end pipeline that also estimates poses, handles outliers, and prepares a deployable scene representation.

Pixel footprints, unbounded scenes, and scale

The original formulation treats each pixel primarily as a single infinitesimal ray. A real pixel instead integrates light over a finite footprint, influenced by lens optics, exposure, and sensor sampling. When a reconstructed scene contains detail finer than that footprint, point-sampled rendering can produce aliasing: textures shimmer during camera motion, thin structures flicker, or distant patterns appear at incorrect frequencies. Increasing the number of depth samples along the same central ray does not by itself integrate the pixel’s transverse footprint.

Mip-NeRF, presented by Jonathan T. Barron and colleagues in 2021, addressed this mismatch by treating pixels as conical beams and approximating their segments with three-dimensional Gaussian distributions. Instead of evaluating the positional encoding at one point, it computes an integrated positional encoding over each distribution. High-frequency components are attenuated when the represented region is large. For a Gaussian variable XX with mean μ\mu and variance vv, the relation

E[sin⁡(ωX)]=sin⁡(ωμ)exp⁡(−ω2v/2)\mathbb{E}[\sin(\omega X)] = \sin(\omega\mu)\exp(-\omega^2v/2)

illustrates the mechanism: larger footprints suppress oscillations that cannot be reliably resolved. The approach connects neural field rendering to established multiscale filtering principles.

Scene extent creates another difficulty. A bounded grid or normalized coordinate range can represent a tabletop object conveniently, but an outdoor camera may see nearby vegetation, distant buildings, and sky in the same image. Allocating spatial samples uniformly over that full depth range is inefficient. NeRF++ separated foreground and background representations using an inverse-sphere parameterization. Other approaches transform or contract unbounded coordinates into a bounded representation while accounting for the changed spatial distribution.

Mip-NeRF 360, introduced in 2022, combined tools for unbounded scenes with improved sampling and optimization. Its nonlinear scene contraction maps distant positions into a bounded region, allowing the representation to devote more effective resolution to nearby space. A proposal mechanism guides fine sampling, while a distortion regularizer discourages unnecessarily dispersed rendering weights along rays. These mechanisms address distinct failure modes: excessive spatial extent, inefficient depth allocation, and ambiguous density placement.

At still larger scales, reconstruction becomes a problem of organizing many local observations and representations. Large environments may be partitioned into blocks, each optimized from a relevant image subset, with coordinated rendering across boundaries. This reduces the burden on any one field but introduces questions of seam consistency, coordinate alignment, visibility, and memory streaming. Variations in weather, illumination, or capture date also become more prominent as collections grow.

Scale is therefore not merely a matter of increasing parameter count. It affects the appropriate camera model, spatial parameterization, pixel integration rule, sampling distribution, and storage hierarchy. A method can reconstruct a large number of points yet still fail when the viewer changes distance substantially, because representing spatial detail and filtering that detail for a new image resolution are separate requirements.

Dynamic scenes and generalization across scenes

A static radiance field assigns one density and appearance model to each location. Video of a moving subject violates that assumption: a hand occupies different positions over time, and an object may reveal or occlude surfaces as it moves. One direct extension adds time as an input, producing a field F(x,d,t)F(\mathbf{x},\mathbf{d},t). This can represent temporal change, but without additional structure it may require substantial capacity and provides limited correspondence between a material point at one time and the same point at another.

A common alternative uses a canonical field together with a deformation function. A time-dependent mapping sends an observed-space position into a reference coordinate system:

xcanonical=Wϕ(x,t).\mathbf{x}_{\mathrm{canonical}} = W_\phi(\mathbf{x},t).

The radiance field is queried in that canonical space, allowing appearances and geometry to be shared across frames. D-NeRF and Nerfies, both introduced in 2021, exemplified related approaches to dynamic and deformable scenes. Deformation regularization matters because image reconstruction alone may permit implausible or inconsistent mappings.

Canonical models work most naturally when the scene can be explained as deformation of a reasonably stable object. They become harder to apply when topology changes, objects appear or disappear, or different surfaces come into contact and separate. View-dependent effects create additional complications: warping spatial positions does not automatically transform viewing directions or preserve how highlights should move. HyperNeRF, introduced in 2021, used a higher-dimensional representation to model changes that are difficult to express through an ordinary three-dimensional warp alone.

Real image collections also vary for reasons other than geometric motion. NeRF in the Wild, presented in 2021, incorporated per-image appearance embeddings and transient components to handle unconstrained photographs with changing illumination and temporary occluders. These variables help separate a persistent scene from observation-specific variation. However, appearance conditioning is not the same as recovering a physically interpretable lighting model, and transient components do not by themselves provide a coherent animated reconstruction of moving objects.

A separate distinction concerns whether the model is optimized independently for every scene. The original NeRF was a per-scene representation: its trained weights described one environment. Generalizable methods learn across a collection of scenes and condition predictions on one or more input photographs. PixelNeRF, introduced in 2021, used image features to condition a radiance-field network. Such models can initialize reconstruction rapidly or infer structure from sparse views by drawing on learned regularities.

Generalization introduces a trade-off between measured evidence and prior expectation. With many overlapping images, reconstruction can rely heavily on scene-specific constraints; with one or two images, hidden geometry must be inferred from learned patterns. A plausible completion is not necessarily the geometry of the photographed object. Similarly, diffusion models can supply priors for radiance-field generation or reconstruction, but the resulting content may include synthesized structures unsupported by the input photographs.

Geometry, materials, and neighboring representations

A NeRF can produce depth estimates by examining the rendering weights along a ray. A common accumulated depth is

D^(r)=∑iwiti,\widehat{D}(\mathbf{r})=\sum_i w_i t_i,

with an additional normalization by ∑iwi\sum_i w_i when a conditional expected depth over the accumulated foreground is desired. This is useful for visualization and downstream processing, but it is not necessarily the location of a sharp surface. If density is spread across several layers, the expected depth may fall between them. Similar caution applies to normals derived from density gradients.

An explicit polygon mesh can be extracted by selecting a density threshold and running an isosurface algorithm such as marching cubes. The threshold is a modeling choice rather than a uniquely determined physical boundary. Floaters, thick surfaces, and holes may appear even when rendered images look convincing, because image-based training does not directly require a clean, watertight mesh. Mesh quality and novel-view image quality are related but distinct evaluation targets.

Surface-oriented methods address this mismatch by representing geometry with a signed distance function, whose value indicates signed distance from a surface. NeuS and VolSDF, both introduced in 2021, connected such functions to differentiable volume-rendering formulations. The zero level set provides a more explicit geometric object, while volumetric integration preserves useful image-based optimization properties. These methods impose stronger surface assumptions and are not interchangeable with an unconstrained volumetric field for every kind of scene.

Separating materials from illumination is another extension. Basic direction-dependent color can encode a highlight without explaining whether it arose from material roughness, light position, or environment reflection. Inverse rendering instead seeks quantities such as geometry, surface normals, illumination, and a bidirectional reflectance distribution function. This decomposition can support relighting and material editing. It is also more ambiguous: a dark pixel may result from dark reflectance, weak illumination, shadowing, or some combination. Additional assumptions, measurements, or priors are generally needed.

A prominent neighboring technique is 3D Gaussian splatting, introduced by Bernhard Kerbl and colleagues in 2023. It represents a scene using explicit three-dimensional Gaussian primitives with opacity and directional appearance, projected and composited through a splatting renderer. Unlike a conventional NeRF, it does not render by repeatedly querying a coordinate network along each ray. Its real-time rendering results made it an important alternative for the same novel-view synthesis problem, although memory use, geometric behavior, filtering, and editing characteristics differ.

These comparisons clarify that a scene representation, a rendering algorithm, and an optimization objective are separate design choices. Neural networks can parameterize surfaces or volumes; volume rendering can optimize explicit grids without a neural decoder; and a high-quality view-synthesis system need not produce an editable asset with conventional materials. The appropriate representation depends on whether the required output is an image, a geometric model, a relightable object, or a temporally coherent scene.

Evaluation, applications, and unresolved limitations

NeRF systems are commonly evaluated by withholding images with known camera poses and comparing rendered predictions against them. Peak signal-to-noise ratio measures pixelwise reconstruction accuracy; the structural similarity index assesses local structural agreement; and learned perceptual image patch similarity uses features from a trained network to compare perceptual content. Each emphasizes different errors. Small pose misalignments can strongly affect pixel metrics, while perceptual measures may tolerate deviations that matter for exact reconstruction.

A fair evaluation must distinguish interpolation from extrapolation. A held-out camera surrounded by training viewpoints tests a different capability from a camera behind an object never photographed from the rear. Results also depend on image downsampling, background treatment, exposure handling, scene masks, and whether test views influence parameter selection. Comparisons therefore require consistent data splits and reporting conventions. Strong image scores alone do not establish accurate geometry, calibrated uncertainty, or robustness to changes in illumination.

Applications include photographic scene exploration, object digitization, visual-effects shot reconstruction, and immersive presentation in virtual reality. Radiance fields can preserve appearance that is cumbersome to model manually, such as intricate foliage or view-dependent reflections. In cultural heritage documentation, this supports interactive viewing of captured sites or objects, but a visually convincing reconstruction is not automatically a metrically validated survey. In production graphics, scene editing remains more difficult when lighting and materials are entangled than when they are stored as separate conventional assets.

Robotics and mapping research connects radiance fields to simultaneous localization and mapping. A differentiable scene model can help compare predicted observations with incoming images, while its geometry can support depth estimation or localization. These settings impose requirements beyond offline image quality: bounded latency, stable pose tracking, memory management, and adaptation to changing environments. Systems may therefore combine radiance-field components with conventional geometric maps rather than treating photorealistic rendering as a complete solution to navigation.

Persistent limitations follow from incomplete observations and imperfect models. Transparent and refractive objects can violate the straight-ray interpretation of observed color; mirrors can encourage virtual or incorrect geometry; textureless surfaces provide weak correspondence; and thin structures can be missed by sampling. Occluded regions contain no direct photographic evidence, and uncertainty about them is not necessarily visible in the rendered image. Increasing capacity may improve training-image reproduction without resolving these ambiguities.

Capture provenance and access also remain separate from rendering quality. A fitted field can retain visual details from its photographs, including faces, interiors, or identifying objects. Its learned representation does not by itself anonymize those observations. For deployment, a complete assessment therefore concerns not only reconstruction fidelity and speed but also the source imagery, permitted uses, and what information can be recovered through newly synthesized views.