Version. Evaluation code v1.0.0, scoring mind2act-final-scoring-20261105, budget e2e-p90x1.2-v1. If the code, budgets or scoring change, earlier results are archived and the new version restarts its count from 0 / 4,500.
Two leaderboards
End-to-end models and coding agents see the same cases and the same goal instruction, but act through different interfaces and budgets. They are ranked separately and their scores are not compared.
VLA / WAM board
The model outputs joint actions
Observations: three 640×480 RGB views (head, left and right wrist), both arms' joint angles and grippers, the time and the English instruction exactly as in training. Actions: 6 joint angles per arm plus two normalized grippers, 14 values per 30 Hz control step. The model runs as a service over XPolicyLab WebSocket RPC; simulation pauses during inference. No tools and no external memory: history frames are supplied in the model's training format. 50 cases per task × difficulty.
- observe
- act()
- 14-D joint row
Coding-agent board
The agent calls four tools
Through the RoboProbe adapter the agent sees the three RGB views, the measured 14-D end-effector poses and grippers, read-only joint angles, the last action's result and its remaining budget. It can move_eef (absolute world targets, planned by a fixed cuRobo version), wait_steps, memory_resolve (reopen up to six earlier screenshots) or give_up. A task recipe states the capability, scene, completion condition and scoring table. 5 of the 50 cases per task × difficulty; we plan to extend agents to all 50 as resources allow.
- observe
- tool call
- planned motion
Neither track sees task metadata, scene definitions, ground truth, scorers, expert trajectories or reference data.
Cases
30 tasks × 3 difficulties × 50 cases = 4,500 cases per setting, run in two settings. A case fixes the task, difficulty, seed, scene and instruction, identically for every model.
- MindE, E2, F, F2, M, M2, P, P2, S, S2
- ActFR1, FR2, DR3, DR7, PR1–PR6
- Mind2ActCP01–CP10
- CleanOriginal scenes
- Random23 table materials, 6 light types, clutter
- Agent subset5 cases per cell, 450 per setting
- Hidden set. The evaluation cases are kept by Mind2Act and have no validation split; evaluation seeds are separate from training seeds.
- Random changes appearance only. Lights keep their poses, with each colour channel at 0–2× white and intensity at 0.8–1.2×. Object poses, robot, cameras, collisions, masses, seeds, instructions and scoring match Clean, and every Random case is paired with a Clean case and looks the same on every run.
- One run per case. Each case is evaluated once, without repeated seeds.
- Official runs. Mind2Act runs every model on its own machines with the released code, assets and budgets.
Budgets and endings
Control and RGB run at 30 Hz and physics at 120 Hz: one control step is four physics steps, 1/30 s. Budgets are counted in physics steps, so model latency never consumes task time.
VLA / WAM
Physics budget per task × difficulty: 1.2 × the P90 duration of its training demonstrations, rounded to whole control steps. The same budget applies to Clean and Random.
- Calls
- Unlimited
- Wall clock
- 2 h per case (safety stop, not scored)
Coding agents
4 × the end-to-end physics budget and at most 150 decisions; whichever runs out first ends the case. A rejected or invalid action still costs a decision. Network retries (up to 12 per decision) do not.
- Calls
- 150 decisions
- Wall clock
- 6 h per case (safety stop, not scored)
Two kinds of ending. Outcome tasks (22) check the success condition after every action, end at 100 when it is met, and otherwise run to the end of the budget and score the final state. Event tasks (8: E, E2, F2, FR2, M, CP04, CP05, CP10) score mechanism events and end when the prescribed sequence is complete.
Violations do not end a case early; only the final state is judged. "Inside" means supported by the container with its centre inside, with no tolerance margin; buttons, keys and levers count once per press of the mechanism itself.
Scoring
Every case gets a score from the task's own scorer, which reads native measurements only.
Success. Anything short of success earns the task's partial credit.
Partial-credit caps for Mind and Act tasks (Easy / Medium / Hard).
Partial-credit caps for Mind2Act tasks, so harder tiers never outscore easier ones.
Success % = share of valid cases scoring 100
Suite, Overall = plain means over 10 and 30 tasks
Count-based tasks award the cap × correct units / required units; graded tasks use 1.5 / 1.1 / 0.75 steps of the cap. Only core units earn credit: preparatory steps such as opening a drawer or lid, pressing START or returning a tool earn nothing, but are required for 100. PR4, S2, F2 and CP07 give no partial credit.
Results are aggregated per task × difficulty, then by suite and overall. With 50 cases per cell and 10 tasks per suite, averaging over cases, cells or tasks gives the same value. Clean and Random, and the three suites, are reported separately. Only Score and Success are reported, without uncertainty estimates.
Validity and integrity
Only valid cases enter the averages, and every result states how many there were.
| Status | Meaning |
|---|---|
| valid | Passed native queue and seal checks. Valid zeros are always kept and never re-run. |
| infra_failed | Rendering, asset, process, controller or snapshot failure. Re-queued once; if it fails again it is excluded, not scored as 0. |
| operator_cancelled | Stopped by an operator. Data is kept but excluded from averages. |
Each result reports valid cases out of 4,500 (450 for agents), infrastructure failures and unfinished cells. Every case records the evaluation commit, asset release, budget table, protocol version and checkpoint SHA. Agents cannot reset or extend their call ledger. Before a run, the release, instructions, P90 budgets, 6 Hz recording and actual model inputs are verified, and expert replays confirm that each scene loads, completes and scores; replays are never model results.
What submitters declare
A leaderboard row compares complete systems as submitted, not equal pretraining or compute.
- Model and versionName and release
- CheckpointServable checkpoint and its SHA
- Inference settingsChunk length, execution rate, stochasticity
- PreprocessingImage resizing is the adapter's choice
- Agent settingsModel, effort, transport, call limit, retries
- StatisticsData statistics an E2E model needs
Each case stores its score, success, termination reason, score components, the three camera videos and run metadata. See Reading the leaderboard for how these appear on the boards.
From submission to leaderboard
VLA and world-action models, and coding agents, are all welcome. Mind2Act runs the evaluation.
Get in touch
Open an issue with the model, its board and what it needs to run.
Integrate
Serve the model over XPolicyLab RPC, or describe the agent's transport and limits for RoboProbe.
Freeze
Fix the checkpoint SHA and inference settings.
Evaluate
Mind2Act runs all Clean and Random cases with the released code and budgets.
Publish
The row appears with valid cases out of the total; unpublished results stay marked “—”.
Evaluate your system
Questions about a track or an adapter are welcome before you submit.