A success rate shows how often a model completes a task, but not where it goes wrong. As part of
Mind2Act World, we publish model profile cards for selected models with distinctive behaviour. A card locates where a model’s episodes break down, explains why,
and points to the capabilities worth improving first.
From model profile card to a better model
Your model
End-to-endRound 1 of 2
Mind
Act
Mind2Act
1 · Evaluate
Three views
Mind
Act
Mind2Act
Levels
EasyMediumHard
Scenes
CleanRandom
30 tasks × 3 levels × 2 scenes
2 · Diagnose In this project
Model profile card
Views
Weak view
Act
Points to
Targeted data
First breaks at
Perceive
Reason
Execute
Close
Why
A-GRASPgrasp not held
3 · What helps In development
End-to-endTargeted data
AgentExperience library
4 · Improve and evaluate again In development
Illustrative example. Evaluation and model profile cards are part of this project; the dashed steps, which feed a diagnosis back into the model, are ongoing work.
Same score, different problems
Illustrative comparison: two hypothetical models both score 0% on building a bridge from blocks, for different reasons.
Score0%both models
Model A
Process
stops at step 1 of 6
First breaks at
Execute the first grasp slips
Why
A-GRASP grasp not held
Direction
Grasp and lift demonstrations
Model B
Process
reaches step 4 of 6
First breaks at
Reason places the deck before the piers
Why
M-PLAN wrong step order
Direction
Planning that checks prerequisites
How far it getsA process score counts the milestones each episode reaches.
Where it first failsThe first step it could not recover from: perceive, reason, execute or close the loop.
Why, and a directionA failure code, its view (Mind, Act or Mind2Act), and a direction for the model's architecture.
What the dimensions cover
The card groups results by suite, capability and application domain. Each task is evaluated at Easy,
Medium and Hard in Clean and Random scenes, 180 configurations in all.
Mind
Remember task information, learn rules from feedback and choose the next operation. Ten tasks, five capabilities.
Measured cards for three systems, one per paradigm: an end-to-end VLA, a world action model and an agent.
Every failed episode is traced to the first step it could not recover from.
Perceive read the scene
Reason recall and decide
Execute carry out the action
Close the loop check and update
End-to-end board
G0.5 VLA
Galaxea AI
5.9%success over 9,000 episodes
1.9%Mind13.4%Act2.3%Mind2Act
Act carries the successes; Mind is nearly empty past Easy.
Success on every task at Easy, Medium and Hard, and how it holds up as tasks get harder.
Capability profile
30 tasks, 12 capabilities
RingsEasy, Medium and Hard, outward
Success01+10+30+60+
Mind
Act
Mind2Act
Act carries the successes; Mind nearly empty past Easy
Under pressure
success (%) by difficulty, bars 0–30%
Mind
Easy5.6
Medium0.2
Hard0.0
Act
Easy21.9
Medium9.8
Hard8.6
Mind2Act
Easy6.2
Medium0.5
Hard0.2
All tasks
11.2%Easy
3.5%Medium
2.9%Hard
04
What could help
Directions drawn from the failure codes and breakpoints above.
1Grasp and hold
Grasp failures are its most frequent first failure (1,324), and 202 of 300 bridge-building episodes (CP07) never grasp a piece.
Change
a grip check before lifting, with re-grasp when the object slips.
Data to add
Contact-rich demonstrations of grasping small parts, including slips and the re-grasp that recovers them.
2Task state and the final step
924 memory failures and 866 stalls: 210 of 300 meal-order episodes (P1) forget the order and 250 of 300 repeated-item episodes (E1) stall before submitting.
Change
a task-state memory that P1 checks before hand-over, and an escape from repeated motions so that E1 reaches its submission.
Data to add
Long demonstrations that carry orders and items across steps and end with the P1 hand-over or the E1 submission.
End-to-end board
InternW0 WAM
Shanghai AI Lab
8.7%success over 9,000 episodes
5.4%Mind17.6%Act3.0%Mind2Act
Act holds up even at Hard; Mind and Mind2Act fade past Easy.
Success on every task at Easy, Medium and Hard, and how it holds up as tasks get harder.
Capability profile
30 tasks, 12 capabilities
RingsEasy, Medium and Hard, outward
Success01+10+30+60+
Mind
Act
Mind2Act
Act holds up even at Hard; Mind and M2A fade past Easy
Under pressure
success (%) by difficulty, bars 0–30%
Mind
Easy13.5
Medium2.1
Hard0.5
Act
Easy20.6
Medium19.5
Hard12.7
Mind2Act
Easy8.0
Medium0.8
Hard0.3
All tasks
14.0%Easy
7.5%Medium
4.5%Hard
04
What could help
Directions drawn from the failure codes and breakpoints above.
1Precise contact
Fine-control errors (1,132) and grasp failures (806) make up most of its Act failures.
Change
shorter action chunks and closed-loop correction near contact, where its grasps and fine alignment fail.
Data to add
Demonstrations of grasping and fine alignment, with the corrective motions near the target.
2Memory and preconditions
1,011 memory and 761 precondition failures: 271 of 300 cooking episodes (CP04) heat before all ingredients are in the pot, and 245 of 298 failed meal-order episodes (P1) forget the order.
Change
a task-state memory and a precondition check: all ingredients in the pot before HEAT in CP04, and the order checked before hand-over in P1.
Data to add
Demonstrations that check the pot before heating, and long episodes that carry orders and rules across steps.
Agent board
GPT-6 Astra Agent
OpenAI
34.6%success over 900 episodes
52.7%Mind39.0%Act12.0%Mind2Act
Mind and the Easy rings fill up; DR4 and PR6 stay empty.
Success on every task at Easy, Medium and Hard, and how it holds up as tasks get harder.
Capability profile
30 tasks, 12 capabilities
RingsEasy, Medium and Hard, outward
Success01+10+30+60+
Mind
Act
Mind2Act
Mind and Easy rings fill up; DR4 and PR6 stay empty
Under pressure
success (%) by difficulty, bars 0–70%
Mind
Easy64.0
Medium50.0
Hard44.0
Act
Easy44.0
Medium39.0
Hard34.0
Mind2Act
Easy26.0
Medium7.0
Hard3.0
All tasks
44.7%Easy
32.0%Medium
27.0%Hard
04
What could help
Directions drawn from the failure codes and breakpoints above.
1Finer grasp and pose tools
Act accounts for 55% of its failures: grasp (120), fine control (107) and placement (47); all 30 lift episodes of DR4 lose control.
Tools
grasp-quality feedback and small relative pose moves, so a grasp can be checked and corrected.
Library to add
failed grasps and lifts with the observation that showed the slip, retrieved as examples.
2Verify and retry before stopping
Typical tasks stop at one physical step: 21 of 30 cooking episodes (CP04) fail to grasp a jar and 21 of 30 domino episodes (PR6) knock pieces while placing.
Tools
an observation after the tray lift (DR4), the jar grasp (CP04) and each domino placement (PR6), and a retry before the episode is ended.
Library to add
recovered attempts (slip, check, retry) from earlier episodes of the same task.