Document

Mind2Act World

A diagnostic benchmark for embodied mind, physical action, and mind-action composition in robotic manipulation.

Evaluation scale

Case counts are per model, across Clean and Random.

SuiteTasksLevelsSettingsVLA / WAM casesAgent cases
Mind10323,000300
Act10323,000300
Mind2Act10323,000300
Total30329,000900

Every task has three levels (Easy, Medium, Hard) and is run in two settings (Clean, Random). An end-to-end model gets 50 cases for each level in each setting, an agent 5.

A robot has to think and act in a loop: what it decides sets the next action, and what that action does changes the next decision. A success rate alone cannot say which part went wrong, the reasoning, the execution or the hand-off between them.

Three suites

Mind tests reasoning and Act tests execution, each on its own. Mind2Act needs both in the same task.

SuiteQuestion it asksTasks
MindWhat must the robot remember or infer before it acts?10
ActCan the robot turn the intended action into a valid physical outcome?10
Mind2ActHow does execution feedback change what happens next?10

Design principles

  • DiagnosisEach failed case is traced to the step where it went wrong.
  • Same conditionsEvery model on a track gets the same cases, instructions and budgets.
  • Hidden casesEvaluation seeds are held out from training.
  • Physical scoringScores come from where objects end up and what the mechanisms register.

01Benchmark

Tasks & difficulty

30 tasks across Mind, Act and Mind2Act, each with three difficulty levels.

How difficulty increases

A harder level asks more of the same skill, for example more items to remember or more steps to plan. Each task page shows the demo videos, the instruction, the success condition, the scoring rules and the three levels side by side.

Task categoryWhat increasesTasks
EEntity memoryMore identities and associations to retain; longer retrieval spans and more complex expressions.2
PProcess memoryMore accumulated orders or layers to reconstruct, with additional orders and distractors.2
SSpatial memoryFaster changes to remembered spatial arrangements and larger navigation grids.2
FFeedback-rule learningLarger action or permutation spaces to infer from limited feedback.2
MIntegrated memoryLonger routes with revisits and more identity–position pairs to track.2
DRDynamic responseMore moving targets, shorter response windows, multiple gates and tighter transport constraints.4
PRPrecise executionTighter reading and heading tolerances, more complex paths and folds, and longer physical chains.6
CPMind–action coordinationMore items or remembered steps, additional prerequisites, and tighter feedback-dependent execution.10

Task index

02Evaluation

Evaluation protocol

We run every model ourselves, on hidden cases with fixed time budgets, and score the physical result.

Tracks

End-to-end models (VLA/WAM) and agents run on separate tracks, with different case counts and budgets.

Counts are per model. A configuration is one task at one difficulty in one setting.

MeasureVLA / WAMAgent
Action interfaceRobot actionsActions through tools
Cases / config.505
Cases / setting4,500450
Total cases9,000900

Cases and environments

A case fixes the task, level, seed, scene and instruction. Evaluation seeds are never used for training. Each case contributes one accepted result, obtained on our machines with versioned code and assets.

Clean and Random

Clean uses the original scene. Random varies materials and lighting and adds physical clutter, while preserving the task goals and scoring rules.

E2 keypad and task objects on the original wood table with neutral light
Clean Original tabletop, lighting and scene.
The E2 keypad and task objects on a darker reflective table with a green cast and extra objects around the edges
Random Reflective tabletop, green-tinted light and extra objects around the task.

Random varies table materials (23 options), lighting (6 light types) and physical clutter. Lights keep their positions. Each colour channel ranges from 0 to 2 times white, and intensity from 0.8 to 1.2 times its original value.

Clean and Random cases come in pairs: same task objects and poses, robot, camera, physics settings, seed, instruction and scoring. The Random scene of a case is fixed, so every model sees the same one. The leaderboard shows the mean of the two settings; each model page shows both.

Scoring

A successful case scores 100. Other outcomes receive the partial credit defined for that task, or zero. The individual task pages show the success conditions and score values for each difficulty beside the videos.

Task score
The mean score over the task’s fixed case set, including zeros and partial credit.
Success rate
The percentage of evaluation cases that score 100.
Suite score
The mean over the ten tasks in Mind, Act or Mind2Act.
Overall score
0.25 × Mind + 0.25 × Act + 0.5 × Mind2Act

Leaderboard values average Clean and Random. Overall success rate is not weighted: it is the number of successful cases divided by the fixed case count, so Mind, Act and Mind2Act each count for a third.

Only what happens in the scene counts. An object has to rest inside its container; sending the command is not enough. Buttons, keys and levers count when the mechanism itself moves.

Budgets and termination

Control and RGB run at 30 Hz and physics at 120 Hz: one control step is four physics steps, 1/30 s. Budgets are counted in physics steps, so model latency never consumes task time.

LimitVLA / WAMAgent
Physics budget1.2 × demonstration P904 × VLA / WAM budget
DecisionsUnlimited150
Wall-clock limit2 h6 h

Physics budgets use the 90th-percentile duration of training demonstrations for each task and difficulty, rounded to whole control steps. Clean and Random use the same budget.

An agent case ends when either the physics budget or 150 decisions are used. Rejected and invalid actions count as decisions; network retries do not, with up to 12 retries per decision. Wall-clock limits are unscored safety stops.

Two kinds of ending. Outcome tasks (22) check the success condition after every action, end at 100 when it is met, and otherwise run to the end of the budget and score the final state. Event tasks (8) score mechanism events and end when the prescribed sequence is complete.

Scores reflect final outcomes and task-specific events during execution. Containment follows each task’s rules; buttons, keys and levers count once per press of the mechanism itself.

Validity and integrity

Scores and success rates use the fixed evaluation set. An accepted rerun replaces the result for the same case without changing the case count.

valid
Included in the reported scores and success rates, including zero-score outcomes.
infra_failed
Rendering, asset, process, controller or snapshot failure. The same case is rerun; its accepted result replaces the previous attempt.
operator_cancelled
Stopped by an operator. The attempt is retained, and the case remains in the evaluation set.

Results report the number of valid cases and infrastructure failures.

03Results

Leaderboard

VLA/WAM and agents are ranked separately.

MetricDefinition
Overall score0.25 × Mind + 0.25 × Act + 0.5 × Mind2Act
Suite scoreMean score across the suite’s ten tasks.
Success ratePercentage of evaluation cases scoring 100.
SettingsMean of Clean and Random.

View leaderboard →

04Participation

Release & submission

Code and setup guides are available on GitHub, with training data on Hugging Face. For evaluation inquiries, contact Mind2ActCommittee@outlook.com.

What comes next

Sandbox agent leaderboard. Agents on the current leaderboard run inside our in-house harness. A sandbox setting with its own leaderboard will let participants bring their own agent harness, which runs in an isolated container with access only to the cameras, the robot state and joint-level control. Submission details will follow.

Mind2Act Mini World. A smaller, fully open-source version of the benchmark will make training and evaluation easier to run on your own hardware.