Agent Leaderboard

GPT-6 Astra

AgentOpenAI

Results reported

Results

Score / Success

SuiteMeanCleanRandom
Overall30.24 / 34.56%31.77 / 36.67%28.71 / 32.44%
Mind53.88 / 52.67%55.90 / 54.67%51.86 / 50.67%
Act40.44 / 39.00%44.56 / 43.33%36.31 / 34.67%
Mind2Act13.32 / 12.00%13.31 / 12.00%13.33 / 12.00%

Task results

Score / Success · Averages across Easy, Medium and Hard.

Mind

TaskMeanCleanRandom
EEntity memory70.83 / 68.33%72.04 / 70.00%69.61 / 66.67%
E1Duplicate Item Identification76.92 / 73.33%81.77 / 80.00%72.07 / 66.67%
E2Object-Code Keypad Entry64.73 / 63.33%62.31 / 60.00%67.16 / 66.67%
PProcess memory23.55 / 21.67%28.15 / 26.67%18.94 / 16.67%
P1Order-Based Meal Assembly46.84 / 43.33%55.80 / 53.33%37.88 / 33.33%
P2Steamer Stack Order Reconstruction0.25 / 0.00%0.50 / 0.00%0.00 / 0.00%
SSpatial memory61.92 / 61.67%67.17 / 66.67%56.67 / 56.67%
S1Rotating Tray Layout Reconstruction23.83 / 23.33%34.33 / 33.33%13.33 / 13.33%
S2Hidden-Zone Navigation100.00 / 100.00%100.00 / 100.00%100.00 / 100.00%
FFeedback-rule learning66.12 / 65.00%68.17 / 66.67%64.07 / 63.33%
F1Unknown Button Mapping and Slider Navigation78.90 / 76.67%69.67 / 66.67%88.13 / 86.67%
F2Color-Block Trial Ordering53.33 / 53.33%66.67 / 66.67%40.00 / 40.00%
MIntegrated memory46.98 / 46.67%43.96 / 43.33%50.00 / 50.00%
M1Workpiece Tracking and Route Reproduction0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
M2Memory Card Matching93.96 / 93.33%87.92 / 86.67%100.00 / 100.00%
Results by difficulty
TaskLevelMeanCleanRandom
E1Duplicate Item IdentificationEasy91.50 / 90.00%100.00 / 100.00%83.00 / 80.00%
Medium58.25 / 50.00%83.30 / 80.00%33.20 / 20.00%
Hard81.00 / 80.00%62.00 / 60.00%100.00 / 100.00%
E2Object-Code Keypad EntryEasy61.50 / 60.00%63.00 / 60.00%60.00 / 60.00%
Medium62.20 / 60.00%62.93 / 60.00%61.47 / 60.00%
Hard70.50 / 70.00%61.00 / 60.00%80.00 / 80.00%
P1Order-Based Meal AssemblyEasy91.50 / 90.00%83.00 / 80.00%100.00 / 100.00%
Medium27.15 / 20.00%44.40 / 40.00%9.90 / 0.00%
Hard21.88 / 20.00%40.00 / 40.00%3.75 / 0.00%
P2Steamer Stack Order ReconstructionEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.75 / 0.00%1.50 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
S1Rotating Tray Layout ReconstructionEasy20.75 / 20.00%21.50 / 20.00%20.00 / 20.00%
Medium30.00 / 30.00%60.00 / 60.00%0.00 / 0.00%
Hard20.75 / 20.00%21.50 / 20.00%20.00 / 20.00%
S2Hidden-Zone NavigationEasy100.00 / 100.00%100.00 / 100.00%100.00 / 100.00%
Medium100.00 / 100.00%100.00 / 100.00%100.00 / 100.00%
Hard100.00 / 100.00%100.00 / 100.00%100.00 / 100.00%
F1Unknown Button Mapping and Slider NavigationEasy91.50 / 90.00%83.00 / 80.00%100.00 / 100.00%
Medium82.20 / 80.00%100.00 / 100.00%64.40 / 60.00%
Hard63.00 / 60.00%26.00 / 20.00%100.00 / 100.00%
F2Color-Block Trial OrderingEasy90.00 / 90.00%100.00 / 100.00%80.00 / 80.00%
Medium60.00 / 60.00%80.00 / 80.00%40.00 / 40.00%
Hard10.00 / 10.00%20.00 / 20.00%0.00 / 0.00%
M1Workpiece Tracking and Route ReproductionEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
M2Memory Card MatchingEasy100.00 / 100.00%100.00 / 100.00%100.00 / 100.00%
Medium100.00 / 100.00%100.00 / 100.00%100.00 / 100.00%
Hard81.88 / 80.00%63.75 / 60.00%100.00 / 100.00%

Act

TaskMeanCleanRandom
DRDynamic response62.79 / 61.67%71.81 / 71.67%53.77 / 51.67%
DR1Continuous Conveyor Picking79.42 / 76.67%87.24 / 86.67%71.61 / 66.67%
DR2Light-Guided Button Response91.73 / 90.00%100.00 / 100.00%83.47 / 80.00%
DR3Periodic Gate Crossing and Withdrawal80.00 / 80.00%100.00 / 100.00%60.00 / 60.00%
DR4Ball-on-Tray Transport and Placement0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
PRPrecise execution25.53 / 23.89%26.39 / 24.44%24.68 / 23.33%
PR1Linear Resistor Adjustment100.00 / 100.00%100.00 / 100.00%100.00 / 100.00%
PR2Rotary Voltage Regulator Setting52.21 / 43.33%56.36 / 46.67%48.05 / 40.00%
PR3Marker Line Tracing0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
PR4Cloth Edge Alignment and Placement0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
PR5Oriented Stamp Placement1.00 / 0.00%2.00 / 0.00%0.00 / 0.00%
PR6Domino Chain Bell Ringing0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Results by difficulty
TaskLevelMeanCleanRandom
DR1Continuous Conveyor PickingEasy100.00 / 100.00%100.00 / 100.00%100.00 / 100.00%
Medium74.84 / 70.00%100.00 / 100.00%49.68 / 40.00%
Hard63.43 / 60.00%61.71 / 60.00%65.14 / 60.00%
DR2Light-Guided Button ResponseEasy100.00 / 100.00%100.00 / 100.00%100.00 / 100.00%
Medium92.20 / 90.00%100.00 / 100.00%84.40 / 80.00%
Hard83.00 / 80.00%100.00 / 100.00%66.00 / 60.00%
DR3Periodic Gate Crossing and WithdrawalEasy60.00 / 60.00%100.00 / 100.00%20.00 / 20.00%
Medium100.00 / 100.00%100.00 / 100.00%100.00 / 100.00%
Hard80.00 / 80.00%100.00 / 100.00%60.00 / 60.00%
DR4Ball-on-Tray Transport and PlacementEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
PR1Linear Resistor AdjustmentEasy100.00 / 100.00%100.00 / 100.00%100.00 / 100.00%
Medium100.00 / 100.00%100.00 / 100.00%100.00 / 100.00%
Hard100.00 / 100.00%100.00 / 100.00%100.00 / 100.00%
PR2Rotary Voltage Regulator SettingEasy85.63 / 80.00%71.26 / 60.00%100.00 / 100.00%
Medium42.77 / 30.00%51.53 / 40.00%34.02 / 20.00%
Hard28.22 / 20.00%46.30 / 40.00%10.14 / 0.00%
PR3Marker Line TracingEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
PR4Cloth Edge Alignment and PlacementEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
PR5Oriented Stamp PlacementEasy3.00 / 0.00%6.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
PR6Domino Chain Bell RingingEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%

Mind2Act

TaskMeanCleanRandom
CPMind–action coordination13.32 / 12.00%13.31 / 12.00%13.33 / 12.00%
CP01Conveyor Meal Assembly61.24 / 60.00%61.31 / 60.00%61.16 / 60.00%
CP02Rotating Cake Decoration18.27 / 16.67%21.72 / 20.00%14.83 / 13.33%
CP03Dynamic Rack Loading0.03 / 0.00%0.07 / 0.00%0.00 / 0.00%
CP04Dual-Arm Cooking0.74 / 0.00%0.16 / 0.00%1.33 / 0.00%
CP05Keyboard Sequence Reproduction24.44 / 20.00%19.13 / 13.33%29.76 / 26.67%
CP06Produce Weighing and Balancing12.59 / 10.00%9.65 / 6.67%15.52 / 13.33%
CP07Bridge Construction0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
CP08Reference-Guided Block Construction15.13 / 13.33%21.08 / 20.00%9.17 / 6.67%
CP09Whiteboard Drawing Reconstruction0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
CP10Predictive Ball Return and Red-Ball Avoidance0.75 / 0.00%0.00 / 0.00%1.50 / 0.00%
Results by difficulty
TaskLevelMeanCleanRandom
CP01Conveyor Meal AssemblyEasy100.00 / 100.00%100.00 / 100.00%100.00 / 100.00%
Medium52.00 / 50.00%43.00 / 40.00%61.00 / 60.00%
Hard31.71 / 30.00%40.93 / 40.00%22.49 / 20.00%
CP02Rotating Cake DecorationEasy32.62 / 30.00%42.25 / 40.00%23.00 / 20.00%
Medium20.33 / 20.00%20.33 / 20.00%20.33 / 20.00%
Hard1.87 / 0.00%2.57 / 0.00%1.17 / 0.00%
CP03Dynamic Rack LoadingEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.10 / 0.00%0.20 / 0.00%0.00 / 0.00%
CP04Dual-Arm CookingEasy1.50 / 0.00%0.00 / 0.00%3.00 / 0.00%
Medium0.50 / 0.00%0.00 / 0.00%1.00 / 0.00%
Hard0.23 / 0.00%0.47 / 0.00%0.00 / 0.00%
CP05Keyboard Sequence ReproductionEasy63.56 / 60.00%44.88 / 40.00%82.25 / 80.00%
Medium5.62 / 0.00%9.25 / 0.00%2.00 / 0.00%
Hard4.14 / 0.00%3.27 / 0.00%5.02 / 0.00%
CP06Produce Weighing and BalancingEasy31.50 / 30.00%23.00 / 20.00%40.00 / 40.00%
Medium5.00 / 0.00%4.00 / 0.00%6.00 / 0.00%
Hard1.26 / 0.00%1.96 / 0.00%0.56 / 0.00%
CP07Bridge ConstructionEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
CP08Reference-Guided Block ConstructionEasy41.50 / 40.00%60.00 / 60.00%23.00 / 20.00%
Medium1.50 / 0.00%1.00 / 0.00%2.00 / 0.00%
Hard2.38 / 0.00%2.24 / 0.00%2.52 / 0.00%
CP09Whiteboard Drawing ReconstructionEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
CP10Predictive Ball Return and Red-Ball AvoidanceEasy2.00 / 0.00%0.00 / 0.00%4.00 / 0.00%
Medium0.25 / 0.00%0.00 / 0.00%0.50 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%

Efficiency

Avg. cost per attempt
≈ ¥175.88
Avg. task time
22.2 min
Avg. response time
11.0 s
Mind · Act · Mind2Act
TasksAvg. cost per attemptAvg. task timeAvg. response time
Mind≈ ¥165.2719.1 min9.9 s
Act≈ ¥115.0516.8 min9.2 s
Mind2Act≈ ¥247.3330.7 min13.0 s
Usage totals

Evaluation cost

900 selected official episodes across Clean and Random.

MetricRecorded totalCoverage
Model decisions71,623900 / 900 episodes
API request attempts73,204900 / 900 episodes
Upstream requests73,20473,204 / 73,204 requests
Input tokens2,596,018,01471,623 / 73,204 requests
Output tokens7,026,36571,623 / 73,204 requests
Cached input tokens (included)410,869,63471,623 / 73,204 requests
Reasoning output tokens (included)3,281,77271,623 / 73,204 requests
Cumulative API request time224 h 03 min 58 s73,204 / 73,204 requests
Cumulative inference time241 h 39 min 05 s900 / 900 episodes
Cumulative episode time333 h 07 min 07 s900 / 900 episodes
Cumulative run time354 h 04 min 52 s900 / 900 episodes
Estimated API costCNY 158,295.7071,623 / 73,204 requests
Cost basis

Recorded requests with complete input, cached-input and output usage.

CNY per 1M tokens: input 70; cached input 7; output 350.

Cost = uncached input × input rate + cached input × cache rate + output × output rate.

API platform billing, 1–10 Nov 2026.

Diagnosis

Where GPT-6 Astra’s failed episodes first break, why, and what could help. All 589 failed episodes are attributed by rules, an LLM judge and human review.

Agent board

GPT-6 Astra Agent

OpenAI

34.6%success over 900 episodes
Mind and the Easy rings fill up; DR4 and PR6 stay empty.
ISFJ · Dispatcher
01

Where it breaks

The first step a failed episode could not recover from, and which side failed.

Where failures break

first unrecoverable step

  • perceive5.6%
  • reason24.8%
  • execute55.3%
  • close the loop7.1%
  • episode level7.1%
Which side failed

share of failures

  • 30.4%Mind
  • 55.3%Act
  • 7.1%composition
  • 7.1%other
02

Why it fails

The most frequent failure codes and the tasks where episodes stop.

First failures

most frequent failure codes

  1. A-GRASPgrasp failed or not held120
  2. A-PRECISIONfine-control error107
  3. A-PLACEmisplaced on release47
  4. M-MEMORYforgot or misremembered45
Where tasks break

episodes stopping there, three typical tasks

  1. 30/30DR4 lose control in the lift
  2. 21/30CP04 fail to grasp a jar
  3. 21/30PR6 knock pieces while placing
03

What it can do

Success on every task at Easy, Medium and Hard, and how it holds up as tasks get harder.

Capability profile

30 tasks, 12 capabilities

MindIntegrated memoryIntegratedM1 · Integrated memory · Easy: 0% successM1 · Integrated memory · Medium: 0% successM1 · Integrated memory · Hard: 0% successM1M2 · Integrated memory · Easy: 60% or more successM2 · Integrated memory · Medium: 60% or more successM2 · Integrated memory · Hard: 60% or more successM2Spatial memorySpatialS1 · Spatial memory · Easy: 10–29% successS1 · Spatial memory · Medium: 30–59% successS1 · Spatial memory · Hard: 10–29% successS1S2 · Spatial memory · Easy: 60% or more successS2 · Spatial memory · Medium: 60% or more successS2 · Spatial memory · Hard: 60% or more successS2Process memoryProcessP1 · Process memory · Easy: 60% or more successP1 · Process memory · Medium: 10–29% successP1 · Process memory · Hard: 10–29% successP1P2 · Process memory · Easy: 0% successP2 · Process memory · Medium: 0% successP2 · Process memory · Hard: 0% successP2Entity memoryEntityE1 · Entity memory · Easy: 60% or more successE1 · Entity memory · Medium: 30–59% successE1 · Entity memory · Hard: 60% or more successE1E2 · Entity memory · Easy: 60% or more successE2 · Entity memory · Medium: 60% or more successE2 · Entity memory · Hard: 60% or more successE2Feedback-rule learningRuleF1 · Feedback-rule learning · Easy: 60% or more successF1 · Feedback-rule learning · Medium: 60% or more successF1 · Feedback-rule learning · Hard: 60% or more successF1F2 · Feedback-rule learning · Easy: 60% or more successF2 · Feedback-rule learning · Medium: 60% or more successF2 · Feedback-rule learning · Hard: 10–29% successF2ActDynamic responseDynamicDR1 · Dynamic response · Easy: 60% or more successDR1 · Dynamic response · Medium: 60% or more successDR1 · Dynamic response · Hard: 60% or more successDR1DR2 · Dynamic response · Easy: 60% or more successDR2 · Dynamic response · Medium: 60% or more successDR2 · Dynamic response · Hard: 60% or more successDR2DR3 · Dynamic response · Easy: 60% or more successDR3 · Dynamic response · Medium: 60% or more successDR3 · Dynamic response · Hard: 60% or more successDR3DR4 · Dynamic response · Easy: 0% successDR4 · Dynamic response · Medium: 0% successDR4 · Dynamic response · Hard: 0% successDR4Precise executionPrecisionPR1 · Precise execution · Easy: 60% or more successPR1 · Precise execution · Medium: 60% or more successPR1 · Precise execution · Hard: 60% or more successPR1PR2 · Precise execution · Easy: 60% or more successPR2 · Precise execution · Medium: 30–59% successPR2 · Precise execution · Hard: 10–29% successPR2PR3 · Precise execution · Easy: 0% successPR3 · Precise execution · Medium: 0% successPR3 · Precise execution · Hard: 0% successPR3PR4 · Precise execution · Easy: 0% successPR4 · Precise execution · Medium: 0% successPR4 · Precise execution · Hard: 0% successPR4PR5 · Precise execution · Easy: 0% successPR5 · Precise execution · Medium: 0% successPR5 · Precise execution · Hard: 0% successPR5PR6 · Precise execution · Easy: 0% successPR6 · Precise execution · Medium: 0% successPR6 · Precise execution · Hard: 0% successPR6Mind2ActPredictive decisionPredictCP10 · Predictive decision · Easy: 0% successCP10 · Predictive decision · Medium: 0% successCP10 · Predictive decision · Hard: 0% successCP10Sequence and planPlanCP05 · Sequence and plan · Easy: 60% or more successCP05 · Sequence and plan · Medium: 0% successCP05 · Sequence and plan · Hard: 0% successCP05CP07 · Sequence and plan · Easy: 0% successCP07 · Sequence and plan · Medium: 0% successCP07 · Sequence and plan · Hard: 0% successCP07Feedback adjustmentFeedbackCP04 · Feedback adjustment · Easy: 0% successCP04 · Feedback adjustment · Medium: 0% successCP04 · Feedback adjustment · Hard: 0% successCP04CP06 · Feedback adjustment · Easy: 30–59% successCP06 · Feedback adjustment · Medium: 0% successCP06 · Feedback adjustment · Hard: 0% successCP06Prerequisites and repairRepairCP02 · Prerequisites and repair · Easy: 30–59% successCP02 · Prerequisites and repair · Medium: 10–29% successCP02 · Prerequisites and repair · Hard: 0% successCP02CP08 · Prerequisites and repair · Easy: 30–59% successCP08 · Prerequisites and repair · Medium: 0% successCP08 · Prerequisites and repair · Hard: 0% successCP08CP09 · Prerequisites and repair · Easy: 0% successCP09 · Prerequisites and repair · Medium: 0% successCP09 · Prerequisites and repair · Hard: 0% successCP09Cumulative progressCumulativeCP01 · Cumulative progress · Easy: 60% or more successCP01 · Cumulative progress · Medium: 30–59% successCP01 · Cumulative progress · Hard: 30–59% successCP01CP03 · Cumulative progress · Easy: 0% successCP03 · Cumulative progress · Medium: 0% successCP03 · Cumulative progress · Hard: 0% successCP03

RingsEasy, Medium and Hard, outward

Success01+10+30+60+
Mind
Act
Mind2Act

Mind and Easy rings fill up; DR4 and PR6 stay empty

Under pressure

success (%) by difficulty, bars 0–70%

Mind

  • Easy64.0
  • Medium50.0
  • Hard44.0

Act

  • Easy44.0
  • Medium39.0
  • Hard34.0

Mind2Act

  • Easy26.0
  • Medium7.0
  • Hard3.0

All tasks

  • 44.7%Easy
  • 32.0%Medium
  • 27.0%Hard
04

What could help

Directions drawn from the failure codes and breakpoints above.

  1. 1Finer grasp and pose tools

    Act accounts for 55% of its failures: grasp (120), fine control (107) and placement (47); all 30 lift episodes of DR4 lose control.

    Tools
    grasp-quality feedback and small relative pose moves, so a grasp can be checked and corrected.
    Library to add
    failed grasps and lifts with the observation that showed the slip, retrieved as examples.
  2. 2Verify and retry before stopping

    Typical tasks stop at one physical step: 21 of 30 cooking episodes (CP04) fail to grasp a jar and 21 of 30 domino episodes (PR6) knock pieces while placing.

    Tools
    an observation after the tray lift (DR4), the jar grasp (CP04) and each domino placement (PR6), and a retry before the episode is ended.
    Library to add
    recovered attempts (slip, check, retry) from earlier episodes of the same task.