NVIDIA OmniDreams#

Blog page

Tech report

Model page

Official code

OmniDreams is a HDMap-conditioned world model for single-view and multi-view driving generation, with presets that balance visual fidelity and runtime throughput.

Teaser video source: OmniDreams project page.

Requirements#

  • Minimum VRAM: ~48 GB.

  • PyTorch: >= 2.11.

Installation#

# from the repo root
uv sync --project integrations_v2/omnidreams

Generate the default MP4 demo from bundled example data:

uv run flashdreams-run omnidreams mp4

The command writes outputs/omnidreams.mp4. Use a launch manifest to override the scenario, rollout length, frame rate, or output path.

Running the method#

To run OmniDreams, launch one of the registered runner slugs. For example:

uv run --project integrations_v2/omnidreams \
    flashdreams-run \
    omnidreams \
    --example-data True \
    --example_data_uuid "239560dc-33d1-11ef-9720-00044bcbccac" \
    --total-blocks 20

Sample example-data UUIDs for the inference script are available in the nvidia/omni-dreams-samples Hugging Face dataset.

We provide the following variants:

Method

Description

omnidreams

Default single-view 2-step HDMap-conditioned I2V demo and runner.

omnidreams-perf

Opt-in compile and CUDA-graph tuning across all pipeline stages.

For multi-GPU inference, use:

uv run --project integrations_v2/omnidreams \
    torchrun --nproc_per_node=4 --no-python flashdreams-run \
    omnidreams \
    --example-data True \
    --example_data_uuid "239560dc-33d1-11ef-9720-00044bcbccac" \
    --total-blocks 20

To inspect all supported CLI arguments and their default values, run:

uv run --project integrations_v2/omnidreams \
    flashdreams-run \
    omnidreams \
    --help

Some generated samples from the above commands:

example_data_uuid: "239560dc-33d1-11ef-9720-00044bcbccac"
example_data_uuid: "24b84744-4156-11ef-b27d-00044bf655de"

Launch the interactive demo#

OmniDreams exposes webrtc and local-window through the shared flashdreams-run <runner> <mode> command. WebRTC only requires a CUDA-capable GPU; local-window additionally requires a display and Vulkan.

The demo requires access to NVIDIA/flashdreams and an HF_TOKEN with read access to nvidia/omni-dreams-scenes (scene USDZs) and nvidia/omni-dreams-models (checkpoints).

First-time setup:

git clone https://github.com/NVIDIA/flashdreams.git
cd flashdreams
export HF_TOKEN=<your-hf-token>
uv sync --package flashdreams-omnidreams --extra interactive-drive

Optionally, pre-download scenes and checkpoints so the first launch isn’t blocked on network I/O:

uv run --package flashdreams-omnidreams omnidreams-prepare

Run the WebRTC demo:

uv run --package flashdreams-omnidreams flashdreams-run-v2 \
    interactive-drive-omnidreams --mode webrtc \
    --host 0.0.0.0 --port 8089

Then open http://<server-ip>:8089/request_session in any browser on the same network.

Note

The first launch is slow. The first time you start the demo, the world model spends several minutes in a one-time optimization pass – checkpoint loading, torch.compile / CUDA-graph capture, and Triton autotuning – before the view becomes interactive. The on-screen indicator shows Loading world model... during warmup and then Optimizing world model... while the first generated chunk is autotuned; this phase is longest on the perf configuration. Subsequent launches are much faster because the compiled kernels and CUDA graphs are cached and reused.

On a GPU with a graphics stack, launch the Vulkan window:

uv run --package flashdreams-omnidreams flashdreams-run-v2 \
    interactive-drive-omnidreams-perf --mode native-window

The local window’s HUD adds a weather-variant selector (clear, rain, snow) next to the scene picker, so the same scene can be switched between conditions.

Note

The local window requires a display server and the system OpenGL / Vulkan client libraries. On Debian/Ubuntu:

sudo apt install -y libx11-6 libxcb1 libgl1 libglx-mesa0 libvulkan1

A Failed to initialize GLFW error indicates the display or one of these libraries are missing.

Steering wheel and game controller#

A steering wheel or game controller can be used to control the local window mode. Any device that Ubuntu detects as a standard game controller or joystick is viable. We provide a configuration tool to calibrate these:

uv run --package flashdreams-omnidreams interactive-drive-configuration

The demo auto-loads your default profile on subsequent launches. When you have more than one profile, the configuration tool’s start screen lists them with Make default (plus Edit and Delete) buttons – re-run the tool to choose which profile local-window loads by default, tweak a profile (steering sensitivity, deadzone, buttons, force feedback), or remove one.

Multiple devices. A profile can bind controls across several devices – for example a wheel base plus a separately-connected or different-brand pedal set. Ctrl+click to select more than one device on the configuration tool’s device page; each control binds to whichever selected device it moves on.

Force feedback. The method is auto-detected per wheel: a driver-managed autocenter spring (Thrustmaster, Logitech) or a self-rendered constant force (Fanatec, which has no autocenter). FFB needs the vendor’s Linux driver and write access to /dev/input/* (add your user to the input group):

Vendor

Driver

Thrustmaster

Out-of-tree hid-tmff2 plus a wheel-mode init (hid-tminit, or tmdrv for TX / TS-XW), for modern wheels (T300RS, T248, TX, T-GT II, TS-PC, TS-XW, …).

Fanatec

hid-fanatecff with the base in PC mode (CSL DD, ClubSport, Podium, DD Pro).

Logitech

In-kernel hid-lg4ff or new-lg4ff (G29, G27, G923 PS); the G920 and Xbox/PC G923 use the HID++ driver (kernel 6.3+).

Native acceleration (perf configuration)#

The registered interactive-drive-omnidreams-perf configuration runs the DiT and LightVAE through the OmniDreams single-view CUDA extension (native_dit_acceleration: required), which is faster than the default PyTorch path. The extension builds against pinned checkouts of CUTLASS, SageAttention, SpargeAttn, and cudnn-frontend that are not vendored in the repo. omnidreams-prepare --perf clones them at their pinned commits into integrations_v2/omnidreams/impl/omnidreams_singleview/3rdparty/:

uv run --package flashdreams-omnidreams omnidreams-prepare --perf

This step only syncs sources; the extension itself compiles on the first launch that uses the perf configuration (one-time, a few minutes). It requires a Blackwell-class GPU (SM 12.0) or newer, a source checkout (the omnidreams_singleview sources ship only in the git tree, not the wheel), git, and a CUDA toolchain (nvcc) matching your PyTorch build. Then launch the perf application:

uv run --package flashdreams-omnidreams flashdreams-run-v2 \
    interactive-drive-omnidreams-perf --mode native-window

native_dit_acceleration="required" makes the perf config fail loudly if the extension can’t build or load, rather than silently falling back to PyTorch.

WebRTC server#

For deployments that require a richer browser frontend with WebRTC’s lower video-delivery latency and a streaming gRPC service for multi-client setups, the webrtc launch mode ships a polished HTML5 client on top of the same OmniDreams pipeline.

# from the repo root
uv run --package flashdreams-omnidreams flashdreams-run-v2 \
    interactive-drive-omnidreams --mode webrtc \
    --host 0.0.0.0 --port 8089

Sample scene UUIDs for the interactive server are available in the nvidia/omni-dreams-scenes Hugging Face dataset. Each scene ships clear, rain, and snow weather variants as sibling archives. Pass -- --variant rain or -- --variant snow to select one (the default is clear weather).

The server may take a few minutes to warm up. Once ready, it prints Connect via http://<server-ip>:8089/request_session. Here, <server-ip> is the server IP address you are connecting to (can use localhost when running locally).

Note

On a remote or cloud GPU instance (e.g. Brev), the server port is usually not reachable at the host IP directly. Forward it to your local machine first, then open http://localhost:8089/request_session:

# Brev
brev port-forward <instance> -p 8089:8089
# or plain SSH
ssh -L 8089:localhost:8089 <user>@<host>

Once successfully connected, the browser-based UI looks like this:

Note

If /request_session loads but the video never appears, the browser is likely obfuscating local IPs in WebRTC ICE candidates (replacing them with mDNS .local hostnames), which prevents the peer connection from completing. Disable the setting and reload:

  • Chrome / Edge: chrome://flags/#enable-webrtc-hide-local-ips-with-mdns → Disabled, then restart the browser.

  • Brave: brave://settings/privacy/security → WebRTC IP handling policy → Default public and private interfaces.

  • Firefox: about:config → media.peerconnection.ice.obfuscate_host_addresses → false.

Performance table#

Single-view latency on NVIDIA GB300 at 704 x 1280 resolution.

Stage

1x GPU

2x GPU

4x GPU

8x GPU

HDMap Encoder

28 ms

26 ms

26 ms

26 ms

Diffusion DiT

84 ms

71 ms

49 ms

47 ms

VAE Decoder

6 ms

5 ms

5 ms

5 ms

KV-cache Update

42 ms

34 ms

23 ms

22 ms

Total

118 ms

102 ms

80 ms

78 ms

Effective FPS

68

78

100

103

KV-cache Update is off the hot path and excluded from Total.

Further reading#

  • OmniDreams interactive latency tuning covers the supported interactive-drive latency knobs: model and backend choice, resolution, chunk-size constraints, FP8 and native acceleration, transport, and the validated GB300 reference.

Citation#

If you use OmniDreams, please cite the original work:

@misc{nvidia2026omnidreams,
  title         = {{NVIDIA} {OmniDreams}: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simulation},
  author        = {Basant, Aarti and Kar, Amlan and Paschalidou, Despoina and Wei, Fangyin and Ferroni, Francesco and Garcia Cobo, Guillermo and Turki, Haithem and Ling, Huan and Seo, Jaewoo and Lucas, James and Wu, Jay Zhangjie and Wang, Jialiang and Lorraine, Jonathan and Gao, Jun and He, Kai and Tothova, Katarina and Xie, Kevin and Tyszkiewicz, Micha{\l} and Wu, Qi and de Lutio, Riccardo and Li, Ruilong and Fidler, Sanja and Kim, Seung Wook and Shen, Tianchang and Cao, Tianshi and Pfaff, Tobias and Lew, William and Wu, Xindi and Ren, Xuanchi and Lu, Yifan and Zhang, Yuxuan and Gojcic, Zan and Wang, Zian},
  year          = {2026},
  eprint        = {2606.03159},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  doi           = {10.48550/arXiv.2606.03159},
  url           = {https://arxiv.org/abs/2606.03159},
}