3D Gaussian Splatting Lets Cloud Games Render Any Camera Angle Locally
Research prototype streams a live 3D Gaussian Splatting scene instead of rendered video frames. Unity server builds and continuously optimizes Gaussians from engine G-buffers, skipping offline SfM. Hybrid protocol: Draco-compressed full snapshots plus per-attribute tensor deltas over sockets.
Takeaways
- Research prototype streams a live 3D Gaussian Splatting scene instead of rendered video frames.
- Unity server builds and continuously optimizes Gaussians from engine G-buffers, skipping offline SfM.
- Hybrid protocol: Draco-compressed full snapshots plus per-attribute tensor deltas over sockets.
- Supports diffuse relighting via spherical harmonics and rigid object motion via per-Gaussian object IDs.
- Replay trace: 92.5 Mbps mean bitrate, client SSIM 0.91 vs server 0.96 with 8 reference cameras.
- Beats depth-assisted image warping on quality after convergence; no public code released.
Matti Siekkinen and Teemu Kämäräinen describe a cloud rendering prototype in an arXiv paper that continuously converts a live Unity scene into a 3D Gaussian Splatting model. The server sends model updates over a socket, and the client renders its own camera view. Once scene data arrives, head and camera movements can appear without another server round trip, although scene changes and newly explored areas still depend on network updates.
Why streamed pixels lose the camera race
Cloud gaming and XR services usually render a server-selected camera view and encode the resulting 2D frames as video. Video codecs compress those frames efficiently, but each decoded image remains tied to an earlier camera pose. Clients can warp or reproject the latest frame to compensate for latency, with quality degrading during large movements, disocclusions, depth errors, and lighting changes.
3D Gaussian Splatting, or 3DGS, represents a scene as a collection of oriented 3D ellipsoids with position, scale, rotation, opacity, and color parameters. A GPU projects and blends these Gaussians to render new viewpoints in real time. Existing 3DGS pipelines commonly reconstruct a scene offline from many photographs; this prototype learns from a running game engine and updates the representation during play.
Unity supplies the geometry
The prototype divides work between a Unity application and a native C++ plugin:
| Component | Responsibilities |
|---|---|
| Unity | Runs the simulation, manages cameras, tracks objects, and produces reference textures. |
| C++ plugin | Uses CUDA and LibTorch to store, optimize, render, and serialize Gaussian parameters. |
| Remote client | Receives snapshots and incremental updates, then renders the current model from a local viewpoint. |
Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.
Read original report
