HOW IT WORKS

CoasterBench asks a language model to design a working roller coaster inside OpenRCT2. The model chooses the track, but the game does all the building, testing and rating. There is no second model deciding whether a design looks plausible: the ride has to work in the actual simulation.

That means every track piece has to fit, the circuit has to close, and a train has to make it all the way around. Once it does, OpenRCT2 gives the ride the same excitement, intensity and nausea ratings it would give a player's coaster. The benchmark follows how well a model can use that feedback to improve its design over several rounds.

The implementation protocol

For each round, OpenRCT2 loads a scenario and runs headless, without a game window. Its software renderer stays active, so screenshots still come from the game itself. OpenRCT2 then serves a small tool API using MCP over HTTP. The model runs in a separate sandbox and changes the park only through that API. Its main tools are:

• new_ride starts a coaster at a chosen position and direction.
• place_piece and place_pieces add one piece or a batch of pieces. valid_next_pieces asks the game what can fit at the current end of the track.
• get_state and screenshot show the model what it has built.
• finish_and_test adds the entrance and exit, starts a test train, and runs the simulation. demolish lets the model clear the ride and try again.

The server has a separate, loopback-only control endpoint for resetting the park and saving artifacts. Those controls do not appear in the model's tool list, and its sandbox cannot reach that endpoint. Tool requests are handled one at a time on the game thread, which keeps the order of changes deterministic.

Every placement goes through the same GameActions::Execute path used for player input. If a curve cannot connect, a piece collides with the map, or the selected coaster cannot draw that piece, OpenRCT2 rejects the action and explains the failure in the tool response. The harness does not repair the track or quietly accept an invalid design.

A ride receives ratings only after its test train completes the circuit. OpenRCT2 then calculates excitement, intensity and nausea in its normal ratings engine. That code, RideRatings.cpp, is unchanged from upstream OpenRCT2. Each new round also restores the exact same starting park, including the clock, random state, guests and weather.

Scoring

The leaderboard score starts with the ride's excitement rating from OpenRCT2. The game's own rules reduce excitement when intensity rises above 10, and a crashed ride is not eligible for a score.

The harness then checks the track against OpenRCT2's stock design library. It compares the sequence of track pieces using edit distance and longest common substring, and checks mirrored versions too. A design below the published similarity threshold keeps its full excitement rating. Above that threshold, the score falls toward zero; an exact or mirrored copy scores zero. This originality rule applies in every run mode.

The model can build and test as many candidates as its round budget allows. The best successful candidate becomes the result for that round, even if the model later demolishes it. A run has several rounds—six on the current leaderboards. Each new prompt includes the previous round's result, so the model can improve that design or start again. The track profiles on the round cards show what changed from one round to the next.

Checking the results

Each round includes the complete session trace, with every tool call and reply, along with the exact track program and the reported token use and cost. It also includes a saved .park file. You can open that file in OpenRCT2 and test the coaster yourself. The file's SHA-256 checksum is recorded in the run's artifact manifest.

The site also reports a circuit audit made with OpenRCT2's own track iterator. It checks that every placed piece belongs to the circuit being scored; a green “verified circuit” label means that no disconnected pieces were found. Replays show the train running, while build montages reconstruct the recorded sequence of track pieces.

Finally, run.json records the run mode, scenario hash, budgets, model version, and the exact harness and scorer versions. Together, those files show what the model saw, what it did, and which code produced the result.

Run modes

The modes change both the game tools and the local tools available to the model. Their leaderboards are kept separate because they answer different questions.

The harness also has an explicit override that can grant extra tools such as Bash to a Claude Code or Codex run without staging the engine source. The override and the resulting shell capability are recorded in run.json; such a run does not have the standard design- or library-mode tool set.