From Videos to Pointmaps to Simulated Videos: DA3 + cameras over time, Gaussians, and video models along new VLA trajectories

Another weekend project. Code for one scene, end to end: av-rig-4d-demo.

Summary. After aligning pointmaps using calibrated point clouds and a ruler (Post 1), I align generated pointmaps from DA3 [1] across multiple timesteps (3D → 4D). We place Gaussian Splats [2] there and render from the GT trajectory and from new trajectories proposed by Alpamayo [3]. GEN3C [4] and Vista4D [5] fill the holes of the GS renderings, then DA3 is re-run on the new renderings. Both models generate plausible renderings when the trajectory stays within a reasonable range of the GT video, and 3D pointmaps reconstructed from these videos are consistent frame by frame. We can generate new videos for an ego vehicle following a newly sampled trajectory. Finally, interpolated renderings across almost 360 degrees from the ego vehicle work from the same cloud.

TL;DR

• We can stitch pointmaps over different timesteps as well as multiple viewpoints at the same timestep.
• 3D-informed VDMs can fill the holes of the Gaussian Splats up to some degree. Limitation on near-360-degree views, probably domain gap and model performance.
• Generated frames from the VDMs give pointmaps consistent with their input videos.
• Dataset: NVIDIA Physical AI AV [6]. Real-world, 7 synchronised cameras plus LiDAR, many frames per clip.
• Models:
• Feed-forward 3D recon: Depth Anything 3 (DA3-GIANT-1.1, camera-conditioned).
• Metric depth estimator: MoGe-2 [7], per frame (GEN3C's default depth, and the comparison in Section 1).
• Video diffusion: GEN3C (Cosmos 7B) and Vista4D (Wan2.1-14B). Both take point clouds as input and inpaint the holes.
• VLA: Alpamayo-R1-10B. It samples 6.4 s trajectories from four cameras' frames and 1.6 s of ego history.
• Limitations:
• We need 3D reconstruction models that handle varying intrinsics and take camera parameters as a constraint rather than a hint (DA3's metre is never a function of the cameras, so it is fitted afterwards).
• GEN3C and Vista4D work reasonably well, but we need a model that reacts to the new vehicle trajectory. In our replay every other agent keeps its recorded motion (the pedestrian below).

Section 1: DA3 + camera conditioning + calibrated point clouds

DA3 is a 3D reconstruction model, but its pointmaps can be stitched across frames. The known issue is flicker: the metre of each frame's depth wanders. We fix the metre from the calibrated point clouds of Post 1, in three ways:

• Horizontal. Cross-camera anchors at one timestep: pixels seen by two overlapping cameras, triangulated with the calibrated rig, give the scale of each camera's depth (Post 1).
• Vertical. Temporal anchors: optical-flow correspondences on static pixels between frames, triangulated over the recorded ego motion, give the scale of each camera's depth over time (13-21k anchors per camera and timestep).
• Both + radial. One log-metre per timestep and camera solved from both anchor sets with weak temporal smoothing, plus a per-camera radial term: DA3's metre on a 120-degree lens runs from 0.72 at the image centre to 1.10 at the edge, so the correction is a function of the angle from the optical axis, fitted from the same anchors.

Rendered-depth AbsRel vs LiDAR (lower is better), PSNR of the GS render at the recorded poses, and PSNR without any own view (held-out), on the three scenes of Post 1:

PSNR rewards any calibrated anchor by about 1 dB and cannot tell them apart, because a wrong metre moves a window's cameras together; the vertical and radial terms pay in depth and in flicker.

Depth maps over time: videos/<scene>/depth_flicker_front_wide.mp4 DA3 + camera conditioning takes and outputs 504x280 (the 1920x1080 frames resized to 504 px on the long side, in 14 px patches), while MoGe-2 takes and outputs the 1280x704 rectified frame (its ViT sees about 1134x630 inside, 3600 tokens). DA3 + cameras (stitched) struggles more on the night scene and on the fast-moving highway: the calibrated points are hard to track there, so the temporal anchors get sparse. Log-std of the per-frame scale against LiDAR, stitched: day 0.028, night 0.207, highway 0.139. MoGe-2's per-frame scale is more consistent (0.041, 0.092, 0.033), although its metre is too long on the wide lenses (1.30 / 1.25 / 1.04 vs LiDAR).

We place GS on the pointmaps (one Gaussian per depth pixel on the calibrated ray, metric rig world) and render the GT videos after DA3 + camera, stitched horizontally and vertically.

GT vs GS, 7 views: videos/4e40e1f9/win4_joint_space_s1/gt_vs_render.mp4 videos/02ad147d/win4_joint_space_s1/gt_vs_render.mp4 videos/21d8bd88/win4_joint_space_s1/gt_vs_render.mp4

GS vs GT on the recorded path, 15 scenes, 7 cameras, 80 frames each, each camera rendered from its own window of 4 timesteps through the f-theta lens at 504x284 (gen3c/gs_vs_gt_ftheta.py):

Example frames, real vs GS render, front_wide and cross_right: out/gs_vs_gt_examples.png

DA3 can generate (relatively) consistent pointmaps both horizontally and vertically, and we can place GS on there.

DA3 runs at 504 px on the long side (frames are 1920x1080); all renders and comparisons are at the rectified 1280x704 pinhole used by the VDMs.

Section 2: running the VDMs on new trajectories from Alpamayo

Inpainting GS holes with VDMs: As we have seen in the previous videos, GS renders have holes and artifacts where no camera saw the surface. We can run VDMs that take the renderings (and masks) from the dynamic 3D point clouds as input, and inpaint and generate better frames as the output. We chose two VDMs that inpaint GS holes. GEN3C's default depth is MoGe-2 [7], so we run GEN3C as is and GEN3C with MoGe-2 replaced by our DA3 pointmaps. For Vista4D we replace its DA3 SLAM with our DA3 + camera conditioning + calibrated point clouds. GEN3C's backbone is Cosmos-7B and Vista4D's is Wan2.1-14B. Recipe: GEN3C native 1280x704, 12 steps, guidance 0; Vista4D 672x384, 61 frames, 6 steps.

On the recorded path: panels: real | GS | GEN3C as is | GEN3C + DA3 points | Vista4D + DA3 points, with the warp buffers each model received below: videos/4e40e1f9/gen3c/gt_mosaic.mp4 videos/02ad147d/gen3c/gt_mosaic.mp4 videos/21d8bd88/gen3c/gt_mosaic.mp4

Rendering from new trajectories: Alpamayo is a VLA: a VLM plus a flow-matching action module. Given the input frames it samples trajectories for the next 6.4 s. With our VDMs we can re-render videos following Alpamayo's trajectories. We first look at what these trajectories look like. Then we run DA3 with cameras + calibrated point clouds on the new trajectories again, and compare with the LiDAR points to check whether our VDM renderings are consistent from a geometric point of view. Bird's-eye views: out/bev_samples_4e40e1f9.png out/bev_samples_02ad147d.png out/bev_samples_21d8bd88.png

Re-rendered along the samples: with these trajectories we can re-render the videos. Panels: reference (real, recorded path, same time) | GEN3C as is | GEN3C + DA3 points | Vista4D + DA3 points | BEV: videos/4e40e1f9/gen3c/g2_mosaic.mp4 videos/02ad147d/gen3c/g2_mosaic.mp4 videos/21d8bd88/gen3c/g4_mosaic.mp4

What we see: GEN3C with MoGe-2 depth or with our DA3 pointmaps, and Vista4D + DA3 pointmaps, perform similarly on these scenes and across the camera views. They look okay-ish, but the renderings tend to become blurry as the clip progresses. Three things stand out: 1) day scene, a pedestrian appears later, so the current trajectory needs to be modified; 2) the night scene has the same issue: planned at t = 0, sample g2 swings 2.9 m into the next lane while passing the truck; 3) highway scene, the car deviates from its original lane.

Re-planning when the pedestrian appears: in the day scene the pedestrian is not visible at t = 0 when Alpamayo sampled (46 m away on the kerb). She appears later, and sample g2 cannot address it (clearance −0.15 m) while the GT driver slows. So we sample again at t = 3 s, when she is visible: all samples clear her and the reasoning reads "Yield to the pedestrian crossing the road ahead". The new trajectory follows the recorded path to t = 3 s, then the re-planned sample.

Videos: videos/4e40e1f9/gen3c/g6_mosaic.mp4 The BEV with both samples is in Figure 11.

We can clearly see that the car now slows down and follows the lane properly. This implies that planning needs to be updated more frequently. It also shows one limitation of this pointmap-based VDM approach: in the real world the pedestrian would likely stop.

The night scene: it has the same issue: planned at t = 0, sample g2 swings 2.9 m into the next lane while passing the truck. Re-sampled at t = 3 s, with the truck alongside, all six samples read "keep lane" and the chosen one follows the recorded path to 6 cm on average (out/bev_samples_02ad147d.png), so its videos are the recorded-path set: videos/02ad147d/gen3c/gt_mosaic.mp4.

DA3 on the generated frames: do the generated frames give the same DA3 points? DA3 + cameras on the generated frames, calibrated via the point clouds over 4-timestep windows, vs LiDAR on the recorded path (new trajectories have no GT; there we compared against our own cache instead). AbsRel with one scale per view / one scale per clip / no rescaling:

By day the generated videos reconstruct like the real one; on the highway they lose about half of its metre stability.

Closing the loop for the ego: we can re-generate the videos along the new trajectory, assuming all other agents behave the same: after 2 s on g2, the planner shown our generated frames re-plans clear of the pedestrian in all six samples (+1.3 m GEN3C, +1.8 m Vista4D). This is replay, closed for the ego only.

Almost 360 degrees: the camera travels through the six side and rear mounts at one frozen instant (t0+3 s); GS render first, then both VDMs fill it. videos/4e40e1f9/gen3c/spinc_front_wide_gs1_fill3.mp4 videos/02ad147d/gen3c/spinc_front_wide_gs1_fill3.mp4 videos/21d8bd88/gen3c/spinc_front_wide_gs1_fill3.mp4 GEN3C as is cannot do this: its cache is one camera's MoGe-2 pointmap, which never sees the rest of the circle; both models here run on our 7-camera cloud. AbsRel vs cache, covered / invented:

Conclusion

In this post, I (and Claude Code) generate pointmaps from multi-view videos and stitch them, then put Gaussian Splats, render from trajectories, then run VDMs to fill holes from renderings. The output videos achieve similar performances with source videos when we run DA3, though renderings tend to become blurry when there is no close source frame available. We sample trajectories from VLA model which can generate plausible trajectories given the past frames, and also show examples why we need to update trajectories as we update our simulated frames. We can formulate this as given s_t, we take action a_t and now we recognize the new state s_t+1 passively, as pedestrians or other vehicles follow the pre-defined movements.

This NVIDIA dataset is a unique one as it has synchronized and multiview cameras having a distance of a rig. Also, cameras have different intrinsics, which is rare in the real world dataset, as many of them are monocular multiview captures running COLMAP. With this, I can test feed-forward 3D reconstruction models, VDMs and VLA in an autonomous vehicle setting. I thought GB10 is pretty good, but running VDM was very expensive and took a long time.

Lastly, I would like to thank all authors of Depth Anything 3, Alpamayo-R1, GEN3C, Vista4D, NVIDIA Physical AI Autonomous Vehicle dataset and MoGE-2 for releasing the pretrained weights and/or the data. This small post is built on top of these building blocks.

If you have any feedback, corrections, or ideas for follow-ups, I would love to hear them: send me an email.

References

1. Depth Anything 3. arXiv:2511.10647, code.
2. 3D Gaussian Splatting for Real-Time Radiance Field Rendering, Kerbl et al., SIGGRAPH 2023. project; rendering here uses gsplat.
3. Alpamayo-R1. arXiv:2511.00088, model.
4. GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control. code, built on Cosmos-Predict1.
5. Vista4D (Eyeline Labs). arXiv:2604.21915, project, model; built on Wan2.1.
6. NVIDIA PhysicalAI Autonomous Vehicles dataset. Hugging Face.
7. MoGe-2. code, model.