# Can AI Play Geometry Dash? DashVMC Learns at 60 FPS Using World Models and Reinforcement Learning

Explore how DashVMC uses 2.33 hours of Geometry Dash gameplay, world models and PPO to train an AI agent that controls the live game at 60 FPS.

Canonical URL: https://aiidelist.com/blog/dashvmc-ai-geometry-dash-world-model-reinforcement-learning

Language: en

Published: 2026-10-17

Updated: 2026-10-17

## Key Takeaways

- **Yes, an AI can control Geometry Dash in real time.** DashVMC is a research agent that reads captured game pixels and chooses when to hold or release the jump input. Its optimized deployment loop runs at **60 Hz on an NVIDIA RTX 2060**; this is not a claim that it completes every level.
- **The central innovation is offline practice inside a learned world model.** The researchers trained on **4,264 recorded episodes (251,963 frames, approximately 2.33 hours at 30 FPS)**, then improved a behavior-cloned controller using Proximal Policy Optimization (PPO) in simulated latent rollouts, without collecting more live-game experience.
- **Policy improvement transferred to the real game.** PPO outperformed its corresponding behavior-cloning (BC) starting point in all **15 seed–level comparisons** across three official levels, a ship-mode test layout, and a held-out community layout.
- **Fast control does not require rendering imagined frames.** The live agent skips its visual decoder and next-frame generator; an additional ablation suggests the transformer can also be removed from deployment for this largely reactive task.
- **The limits matter.** The work reports improved survival, not reliable full-level completion, general mastery of Geometry Dash, or confirmed compatibility with version 2.209.

A game that appears to need just one button is a stringent AI-control benchmark. In Geometry Dash, the environment keeps moving while an agent captures an image, interprets the scene and dispatches a keypress. A late jump can end an attempt immediately. **DashVMC** investigates whether an agent can learn a useful model of those consequences, practice without further interaction with the live game, and still act within a frame budget.

## What Is DashVMC?

DashVMC stands for **Real-Time Discrete World Model Control in Geometry Dash**. Developed by **Florent Tariolle and Florian Yger**, it is described in a [research paper released on September 30, 2026](https://arxiv.org/abs/2609.40003), associated with the NeurIPS 2026 PTA workshop. The [official project site](https://tariolle.github.io/dash-vmc/) includes videos and an interactive world-model demo; [the GitHub repository](https://github.com/Tariolle/dash-vmc) provides code and reproduction instructions.

Unlike a fixed script that presses jump at predetermined timestamps, DashVMC learns from screen observations and action-labeled play. Unlike a purely online reinforcement-learning agent, its policy-improvement stage does not repeatedly restart the actual game. And unlike a video generator, its objective is **choosing actions that prolong survival**, not producing photorealistic future frames.

The distinction is important: the system learns predictive dynamics **during training**, but the reported live controller is fundamentally **reactive during deployment**. It does not need to generate a new video frame before each keypress.

## Why Geometry Dash Is a Difficult Test for AI Agents

Geometry Dash combines a small action space with narrow timing tolerances:

- **Continuous motion:** the avatar moves forward while the level scrolls, even when inference is running.
- **Irreversible mistakes:** a mistimed cube jump can make the next collision unavoidable.
- **Different movement modes:** the cube requires carefully timed jumps, while ship mode demands sustained input and release to manage altitude.
- **Partial observation:** the controller receives processed screenshots, not a perfect list of obstacle coordinates, velocities or collision boxes.
- **Real-time deadlines:** at **60 frames per second, the available interval is about 16.67 milliseconds** for capture, preprocessing, inference and input dispatch.

A 60 Hz control loop means the agent can **update its held-key state once per frame**, not that it presses jump 60 times every second. That distinction is essential for understanding both the demo and the benchmark.

## How DashVMC's World Model Works

The architecture separates perception, prediction and action selection. Its compact representation reduces the computational burden that normally makes model-based control difficult in a live game.

**1. Convert screenshots into discrete visual tokens.** Each captured frame is cropped, converted to grayscale, filtered with Sobel edge detection, and resized to **64 × 64 pixels**. A convolutional encoder produces an **8 × 8 grid** of latent vectors. Improved Finite Scalar Quantization (iFSQ) turns these into **64 discrete token IDs**, each drawn from **1,000 possible implicit codes** (8 × 5 × 5 × 5). This prioritizes collision-relevant geometry over decorative textures.

**2. Predict consequences conditioned on actions.** An **eight-layer transformer with width 384** processes four recent frame–action pairs. It predicts the next 64-token grid and an alive/death signal. Crucially, it can model different outcomes for different actions from a shared starting context: **jump now** versus **keep moving without jumping**.

**3. Learn a lightweight controller.** A **45,546-parameter actor–critic** combines the current visual grid with a temporal summary from the transformer. Its actor chooses a binary control action, while its critic estimates the usefulness of the current state. An auxiliary head also predicts an eight-action sequence to support training.

**4. Deploy without generating pictures.** During real gameplay, the optimized baseline reuses the visual encoder and temporal-context computation but **skips next-grid prediction and image decoding**. This shrinks latency because the policy only needs information useful for a timely decision, not a visible imagined future.

The most transferable idea is **separating the model needed to learn a policy from the computation needed to execute it**. A sophisticated training environment can produce a much simpler real-time controller.

## How Reinforcement Learning Improves the Agent Without More Live Gameplay

DashVMC's learning process follows a disciplined sequence.

**Collect offline experience.** The researchers gathered **4,264 action-labeled episodes**, representing **251,963 frames** or approximately **2.33 hours** at the nominal 30 FPS recording rate. The archive includes deliberate failures, official levels 1–7, community layouts and **36 expert trajectories or segments totaling less than 20 minutes**. Augmenting the data produces more training variants, but not additional independent real-world experience.

**Train the visual representation and world model.** The tokenizer learns to compress edge maps, and the action-conditioned transformer learns short-horizon transitions and death signals. Those models are then frozen for policy improvement.

**Warm-start using behavior cloning.** BC learns to imitate recorded key states. It gives the controller a useful starting policy instead of requiring random exploration to discover basic movement.

**Practice with PPO in imagined episodes.** The frozen world model generates up to **45-step** continuations from recorded four-frame contexts. Each PPO iteration uses **512 imagined episodes**. A sampled action changes the model's predicted next state; PPO adjusts the controller based on the simulated outcomes. The first four generated transitions only refresh the model context and are excluded from the scored rollout.

**Reward survival without rewarding pointless jumps.** The training reward is **+1 per survived step and −0.25 per jump action**. The penalty matters: without it, a controller could learn to jump early simply to delay a collision while airborne, instead of navigating an obstacle properly. Rollouts terminate when the model's death prediction crosses its operational threshold.

In simplified form, the training logic is:

```python
# Conceptual pseudocode, not the repository's executable training loop.
world_model = train_from_recorded_frames_and_actions(dataset)
policy = behavior_clone(recorded_actions)
freeze(world_model)

for iteration in range(ppo_iterations):
    imagined_episodes = world_model.roll_out(
        policy=policy, recorded_contexts=sample_contexts(512), max_steps=45
    )
    policy = ppo_update(policy, imagined_episodes)

# The trained policy later acts on fresh game screenshots.
```

This is **model-based reinforcement learning**, not proof that the AI discovered the complete rules of Geometry Dash. It exploits enough local predictive structure to make better decisions. The research also does **not** use a live game as an online training environment during the PPO refinement stage.

## DashVMC Results: Does the AI Actually Play Better?

Yes, under the paper's tested conditions. The central evaluation compares each PPO-refined policy with **the exact BC checkpoint used to initialize it**. Below are mean **live-game frames survived** for controller **seed 43**, based on **25 attempts per policy**. These are survival measurements, not completion rates.

| Tested layout | BC: mean frames | PPO: mean frames | Improvement |
|---|---:|---:|---:|
| Stereo Madness | 120.2 | 280.1 | +159.9 |
| Back on Track | 123.5 | 260.0 | +136.5 |
| Polargeist | 52.4 | 65.2 | +12.8 |
| Stereo INSANE Nerfed (held out) | 104.8 | 290.4 | +185.5 |

The results reported in the [paper's live-survival table](https://arxiv.org/abs/2609.40003) are not limited to this seed. Across seeds **43, 44 and 45**, PPO achieved higher mean survival in all **15** comparisons across five layouts. For the three official levels, average PPO-minus-BC improvements across controller seeds were **154.2 frames on Stereo Madness**, **103.5 on Back on Track**, and **16.3 on Polargeist**.

The headline survival comparisons were conducted through the repository's **30 FPS diagnostic evaluation path**, not the optimized 60 FPS path. A separate cadence probe on **Stereo Madness Copy** compared one frozen policy at 30 and 60 FPS and found similar observed mean wall survival (**17.79 vs 17.72 seconds**). This single-layout result cannot establish identical 60 FPS performance across all official levels.

There are important qualifications:

- **Polargeist is less decisive.** The seed-43 95% bootstrap interval for its **+12.8-frame** gain is **−0.2 to +28.6**, which includes zero. The game introduces an airborne orb interaction that is harder than earlier jump patterns.
- **Held-out does not mean universal generalization.** Stereo INSANE Nerfed was withheld from training selection, but it uses supported mechanics. There is no evidence here that the model solves arbitrary custom levels or every movement mode.
- **The protocol measures survival.** A longer-lived agent is useful evidence of improved control; it is not the same as reaching 100% completion, clearing Demon levels or outperforming expert players.
- **All controller seeds reuse one learned representation and dynamics model.** Three controllers therefore do not establish three independent end-to-end training replications.

**The core finding is transfer:** optimization conducted only in learned, imperfect simulations produced measurable improvement when the controllers were returned to the real, unpaused game.

## Can DashVMC Really Run Geometry Dash at 60 FPS?

The reported **optimized capture-to-action loop** ran on an **RTX 2060**, even while that GPU rendered the game. Across **5,000 measured frames**, the timing was:

| Metric | Reported latency |
|---|---:|
| Mean | 12.018 ms |
| Median | 12.083 ms |
| 95th percentile | 14.068 ms |
| 99th percentile | 14.913 ms |
| Maximum | 24.059 ms |
| Frames over the 16.67 ms budget | 11 of 5,000 (0.22%) |

Those figures support a 60 Hz control loop **most of the time**, not a guarantee that every frame meets the deadline. The loop skips its pacing sleep after a miss instead of replaying a stale input.

The team separately measured the **cost of an imagined transition** on the same RTX 2060:

| World-model system | Native imagined transition |
|---|---:|
| DashVMC | 11.05 ms |
| DIAMOND | 49.30 ms |
| IRIS | 177.83 ms |

That is approximately **4.5× lower latency than DIAMOND** and **16.1× lower than IRIS** in this specific experiment. **These are not comparable game-playing scores:** the published checkpoints target different games and use different prediction interfaces. Nor should the **11.05 ms imagined-transition** result be confused with the **12.018 ms live capture-to-action** result; they measure separate operations.

The design choice behind the efficiency is technical but straightforward: predicting a **64-token grid in parallel**, feeding those compact states into the next imagined step, and avoiding full-image decoding are cheaper than producing visually detailed frames at every training step.

## Why a World Model Can Help Even When Its Predictions Are Imperfect

A counterintuitive result is that the dynamics model does **not** achieve high exact-token prediction accuracy. Its reported one-step development accuracy is **29.58%**, while its death-prediction head reaches **0.794 F1** on the in-distribution development set. In autoregressive rollout diagnostics, exact-token accuracy falls to approximately **2.91% at 45 steps**.

This does not automatically invalidate the policy gains. A model can be inaccurate about the precise visual token arrangement yet retain **action-relevant distinctions**: whether a jump clears an obstacle, whether an idle action produces collision, or whether continued thrust changes a ship's altitude.

An action-intervention diagnostic provides supporting evidence: when researchers flipped the final recorded action while keeping the observation context fixed, the factual action had a lower next-frame negative log-likelihood in **69.52% of development transitions**. That suggests the model's predictions genuinely depend on control inputs rather than merely extrapolating scenery.

Nevertheless, **long imagined rollouts are unreliable**. Predicted levels can eventually repeat, become empty or invent physically impossible geometry. This helps explain the choice of short rollouts seeded from real recorded contexts. The broader lesson is not that visual accuracy is irrelevant; it is that **control usefulness and image-generation fidelity are different evaluation targets**.

## The Surprising Ablation: Does the World Model Need to Run During Play?

Not necessarily. DashVMC also tested a **spatial-only** controller, trained with the same imagined-world approach but deployed without the transformer's temporal summary. In its seed-43 ablation, this variant retained broadly similar live performance across the tested layouts, although the experiment used fewer live attempts and was not a proof of equivalence.

This distinction creates three separate questions for any world-model agent:

- **Did the world model improve learning?** DashVMC's BC-versus-PPO comparison provides evidence that imagined practice did.
- **Must the world model predict future states during inference?** No. Its primary optimized live path disables next-frame generation.
- **Must the dynamics transformer's hidden state stay in the deployed policy?** The spatial-only ablation suggests not always, at least for this largely reactive game.

For robotics, browser agents and other real-time systems, this points to a useful architectural pattern: **train with expensive predictive models; deploy the smallest controller that preserves behavior**. However, tasks requiring long-term memory or hidden-state inference may not tolerate the same simplification.

## How to Try DashVMC for Free

**Option 1: Use the browser dream player.** The simplest hands-on experience is the [official interactive DashVMC world-model player](https://tariolle.github.io/dash-vmc/player/?embed=1). It runs model inference locally in the browser using WebGPU where available, with a WASM fallback. Use **Space**, **Up Arrow**, or the on-screen jump control; the page also offers an **AI pilot**. The demo targets **30 Hz** and lets visitors interact with a generated world. It is **not** the same as giving the research agent control over an installed copy of Geometry Dash.

**Option 2: Inspect or reproduce the open-source system.** The [DashVMC GitHub repository](https://github.com/Tariolle/dash-vmc) contains training, evaluation and deployment scripts, configuration files and model artifacts. Its reported experiment is preserved in Git tag **V7** at commit **b636bcd4**. The maintained main branch documents the following local dream-player command after environment and checkpoint setup:

```bash
python scripts/play_dream.py --config configs/deepdash/v7-phase0.yaml --vae-checkpoint checkpoints_v7/fsq_best.pt --transformer-checkpoint checkpoints_v7/transformer_best.pt --controller-checkpoint checkpoints_v7/controller_ppo_best.pt --fps 30 --scale 12 --max-dream-steps 500
```

For **live-game deployment**, the repository documents an optimized inference path:

```bash
python scripts/deploy.py --config configs/deepdash/v7-phase0.yaml --vae-checkpoint checkpoints_v7/fsq_best.pt --transformer-checkpoint checkpoints_v7/transformer_best.pt --controller-checkpoint checkpoints_v7/controller_ppo_best.pt --preprocessor exact-cuda --fps 60
```

**Prerequisites and cautions:** the published environment specifies **Conda, PyTorch 2.11.0+cu126 and CUDA 12.6**. The live screenshot path uses **Windows Desktop Duplication** and an NVIDIA GPU. Commands assume compatible dependencies, downloaded Git LFS checkpoints and the correct game-window setup; they are not a one-click installer. Consult the repository's current README and platform-specific requirements before running the live controller. Avoid using automated input where it violates a game's rules.

Retraining the published pipeline is substantially more expensive than opening the demo. The reported retained run took **3.19 A100 GPU-hours** for tokenization, **4.20 A100 GPU-hours** for the dynamics model and **17.41 H200 GPU-hours** for the seed-43 controller, totaling **24.80 hours of sequential training** across different GPU types. **Two hours of source gameplay is not two hours of model training.**

## How to Evaluate or Build a Similar AI Game Agent

A meaningful reproduction should test **learning transfer, timing and failure modes** separately rather than judging a highlight video:

- **Compare against the exact imitation baseline.** Record BC and PPO performance from the same starting policy; report mean survival and variation over multiple seeds and attempts.
- **Use genuinely held-out layouts.** Split by entire level or mechanic, not merely by frames. Repeated deterministic prefixes can make apparently unseen observations deceptively familiar.
- **Measure end-to-end latency.** Include capture, preprocessing, inference, input dispatch, deadline misses and the game's own GPU workload; a fast neural network alone is not sufficient.
- **Audit action sensitivity.** From identical starting contexts, compare what the dynamics model predicts after jump versus idle and verify whether collision decisions change correctly.
- **Check model exploitation.** A controller optimized in dreamed environments may discover inaccuracies that generate unrealistic rewards. Compare simulated survival with real-game survival and audit long rollouts.
- **Test mechanics individually.** Cube jumps, orb interactions, ship thrust, gravity inversion and platformer controls can require very different state representations and action timing.

For a smaller research prototype, start with a **controlled original rhythm-platformer** whose states and collisions are instrumented. Collect screen-and-action trajectories, learn a small predictive representation, clone successful inputs, and only then add short-horizon imagined PPO. This is easier to debug than learning simultaneously from a changing commercial game and a complex renderer.

## Limitations, Game-Version Compatibility and What Comes Next

The [original paper](https://arxiv.org/abs/2609.40003) is encouraging but deliberately narrow. Its evaluation uses a deterministic binary-action game, three official levels and a single held-out community layout within the supported movement mechanics. **Gravity inversion and some other mechanics are outside its training scope.** Development data include repeated, similar level prefixes; model diagnostics are not an independent cross-game benchmark. The limited spatial-only and shorter-rollout experiments do not settle all architectural choices.

**What about Geometry Dash 2.209?** The game's [2.209 update released on October 15, 2026](https://steamdb.info/patchnotes/25909607/) adds features including custom keybinds and platformer controls. DashVMC's September paper does **not** establish that its desktop automation has been validated against this later release. Treat compatibility as **unverified** until the project's maintainers or a reproducible test confirm it, especially where capture layout and key mapping differ.

The broader research directions are promising: combine richer memory with reactive control, measure transfer to unseen physics, calibrate rare death events, and explore whether short imagined rollouts can reduce interaction costs in robotics or GUI automation. These are **future research opportunities**, not demonstrated DashVMC results.

## Frequently Asked Questions

**Can AI beat Geometry Dash automatically?** DashVMC can autonomously control tested portions of the real game and survives longer after model-based policy refinement. The published results do **not** establish reliable completion of entire levels or top-difficulty content.

**Does DashVMC use reinforcement learning?** Yes. It begins with **behavior cloning** from recorded gameplay and then uses **PPO** to improve the controller inside a frozen, action-conditioned world model.

**Does the agent need internet access or a cloud AI API to jump?** The demonstrated live controller uses local model inference on an NVIDIA GPU. The browser dream player also advertises local inference; its performance depends on the viewer's browser and hardware.

**Is this powered by ChatGPT or a general-purpose chatbot?** No. DashVMC uses a purpose-built discrete visual encoder, action-conditioned transformer and compact actor–critic. Its central mechanism is model-based reinforcement learning, not text prompting.

**Is DashVMC a Geometry Dash level generator?** No. Its purpose is to **control gameplay**, not to export playable editor levels. The interactive dream player generates approximate game-like continuations, not verified Geometry Dash level files.

**Can DashVMC run at 60 FPS on any laptop?** No. The published 60 Hz measurements used a specific optimized pipeline and **RTX 2060**. Different GPUs, drivers, capture methods and game settings may change latency.

**Is DashVMC free or open source?** The public [GitHub repository](https://github.com/Tariolle/dash-vmc) is distributed under an **MIT license**. The project also offers a free browser-based world-model demo. A separate legitimate copy of Geometry Dash may be needed for live-game deployment.

## Conclusion

DashVMC demonstrates a compelling approach to **AI agents playing Geometry Dash**: compress recorded video into discrete states, learn action-dependent consequences, initialize a policy from human examples, then improve it with **reinforcement learning inside a frozen world model**. The key evidence is not a perfect imagined game; it is **better survival in the actual game** and a **60 Hz capture-to-action pipeline on consumer hardware**.

Its limitations are equally instructive. The study does not prove universal Geometry Dash mastery, and its world model becomes unreliable over long generated trajectories. But it shows how predictive learning can **reduce the need for additional real-environment practice** while supporting tight real-time deadlines.

Explore the [interactive DashVMC demo](https://tariolle.github.io/dash-vmc/player/?embed=1), examine the [open-source code](https://github.com/Tariolle/dash-vmc), or read the [full research paper](https://arxiv.org/abs/2609.40003) to study how discrete world models and PPO turn recorded gameplay into actionable control.