NVIDIA's Lyra 2.0 Builds Full 3D Worlds From a Single Photo

NVIDIA’s Spatial Intelligence Lab has released Lyra 2.0 and plans to present the system at SIGGRAPH Asia. Given one photograph, an optional text prompt, and a user-defined camera path, Lyra generates a long walkthrough and reconstructs its frames as 3D Gaussian splats or surface meshes for real-time rendering and simulation.

The team calls the approach generative reconstruction . A WAN-14B-based video diffusion model synthesizes camera-controlled views, then a feed-forward reconstruction network converts those views into explicit 3D geometry. This pipeline uses the video model’s visual prior to generate unseen space and a geometric memory to keep revisited locations consistent.

Why long routes lose the room

Long camera trajectories expose two weaknesses in video generation, especially when the route includes large viewpoint changes or returns to an earlier location. Each new segment has access to limited history, and small generation errors accumulate as the sequence grows.

• Spatial forgetting: When a location leaves the model’s context window, a later revisit may produce different walls, furniture, colors, or object positions.
• Temporal drift: Small errors in geometry and appearance pass into subsequent segments, gradually distorting the generated scene.

A 3D cache keeps old views in reach

Lyra addresses spatial forgetting with a running 3D cache that routes relevant historical frames back into the model’s attention context. Each cache entry contains a full-resolution depth map, camera parameters, and a downsampled point cloud created by projecting depth pixels into world coordinates.

When the camera reaches a new pose, Lyra projects cached point clouds into the target image plane, accounts for occlusion, and calculates how much of each earlier frame should remain visible. Frames with the highest visibility scores return to the context alongside dense correspondences between old and new viewpoints. The cache selects source evidence, and the video model synthesizes the target appearance.

Training on drift teaches correction

Self-augmented history training exposes the diffusion transformer to degraded versions of its previous outputs. The model therefore learns to remove accumulated artifacts during generation instead of passing each error into the next segment. This places imperfect autoregressive histories inside the training distribution and makes recovery part of the learned task.

From one frame to a 3D asset

Lyra’s inference loop accepts an image at 480x832 , a camera route drawn in the interactive 3D explorer, and an optional prompt describing the scene. NVIDIA recommends 81 frames per generated segment.

1. Set the route: The user chooses camera positions and viewing directions in the explorer.
2. Generate a segment: The WAN-14B-based model synthesizes the next sequence of camera-controlled frames.
3. Update memory: Estimated depth and camera parameters produce point clouds that enter the 3D cache.
4. Continue navigation: Relevant earlier views condition later segments, including revisits to previously generated locations.
5. Reconstruct the scene: The completed frames become 3D Gaussian splats or polygonal meshes.

The single-GPU bill

NVIDIA reports the following inference times on individual data-center GPUs. The measurements cover different tasks and units, so they do not provide a direct comparison between the GB200 and H100.

A four-step distribution matching distillation adapter, distributed as a LoRA, cuts diffusion sampling to four steps for faster inference. This option trades the full sampling process for a lightweight adapter that developers can load alongside the base model.

Code, weights, and license

NVIDIA publishes Lyra 2.0 as an open-source framework under the Apache 2.0 license and describes the release as available for commercial use. The project provides downloadable weights , plus a GitHub repository containing the interactive interface, training code, reconstruction pipeline, and distillation LoRA. Production deployments should also review the licenses attached to WAN-14B and other dependencies.

Where generated geometry fits

Robotics provides a concrete use case because exported scenes can enter NVIDIA Isaac Sim. Gaussian splats supply visual appearance, and meshes can be prepared as collision surfaces. A warehouse photograph could seed synthetic navigation environments in minutes, reducing the manual work required to build training scenes. Unseen layouts and object geometry come from model prediction, so these environments lack the fidelity of measured facility replicas.

• Games and VR: Block out levels and immersive scenes from reference images.
• Embodied AI: Generate varied environments for navigation and robot-policy training.
• Historical imagery: Create plausible explorable spaces when additional capture is unavailable.
• Film and architecture: Build previs scenes and editable concept models from still references.

Limits set by the source image

• Visual evidence: A single photograph constrains visible pixels; hidden rooms, surfaces, and object backs are generated from the model’s prior.
• Geometric precision: Predicted depth and synthesized views provide approximate geometry rather than survey-grade measurements.
• Physical properties: Mesh export supplies surfaces, but collision settings, mass, materials, joints, and semantic labels still require engineering work.
• Asset cleanup: Generated meshes may contain incomplete surfaces or topology that needs repair before production use.
• Compute: Reported generation times and the 80GB H100 configuration place interactive authoring beyond typical desktop hardware.

Video priors become 3D inputs

Lyra turns a pretrained video model into a source of multiview observations, then feeds those observations into explicit 3D reconstruction. The architecture reuses video pretraining, reduces dependence on scarce 3D training sets, and produces representations that existing rendering and simulation tools can consume. Its practical value depends on how much geometric error a project can tolerate and how much cleanup its exported assets require.

Architecture, training, and evaluation details appear in the paper , with the implementation available as source code .