Agent Leaderboard

DeepSeek V4.1 Flash

AgentDeepSeek

Results reported

Results

Score / Success

SuiteMeanCleanRandom
Overall6.09 / 6.56%5.81 / 6.22%6.37 / 6.89%
Mind12.37 / 11.00%10.98 / 9.33%13.77 / 12.67%
Act11.34 / 8.67%11.80 / 9.33%10.89 / 8.00%
Mind2Act0.32 / 0.00%0.23 / 0.00%0.41 / 0.00%

Task results

Score / Success · Averages across Easy, Medium and Hard.

Mind

TaskMeanCleanRandom
EEntity memory23.59 / 20.00%20.92 / 16.67%26.26 / 23.33%
E1Duplicate Item Identification22.85 / 20.00%17.63 / 13.33%28.07 / 26.67%
E2Object-Code Keypad Entry24.33 / 20.00%24.22 / 20.00%24.44 / 20.00%
PProcess memory0.18 / 0.00%0.00 / 0.00%0.37 / 0.00%
P1Order-Based Meal Assembly0.37 / 0.00%0.00 / 0.00%0.73 / 0.00%
P2Steamer Stack Order Reconstruction0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
SSpatial memory10.00 / 10.00%6.67 / 6.67%13.33 / 13.33%
S1Rotating Tray Layout Reconstruction0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
S2Hidden-Zone Navigation20.00 / 20.00%13.33 / 13.33%26.67 / 26.67%
FFeedback-rule learning11.18 / 8.33%10.13 / 6.67%12.23 / 10.00%
F1Unknown Button Mapping and Slider Navigation22.37 / 16.67%20.27 / 13.33%24.47 / 20.00%
F2Color-Block Trial Ordering0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
MIntegrated memory16.92 / 16.67%17.17 / 16.67%16.67 / 16.67%
M1Workpiece Tracking and Route Reproduction0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
M2Memory Card Matching33.83 / 33.33%34.33 / 33.33%33.33 / 33.33%
Results by difficulty
TaskLevelMeanCleanRandom
E1Duplicate Item IdentificationEasy63.00 / 60.00%46.00 / 40.00%80.00 / 80.00%
Medium3.30 / 0.00%4.40 / 0.00%2.20 / 0.00%
Hard2.25 / 0.00%2.50 / 0.00%2.00 / 0.00%
E2Object-Code Keypad EntryEasy23.00 / 20.00%23.00 / 20.00%23.00 / 20.00%
Medium25.87 / 20.00%24.40 / 20.00%27.33 / 20.00%
Hard24.12 / 20.00%25.25 / 20.00%23.00 / 20.00%
P1Order-Based Meal AssemblyEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium1.10 / 0.00%0.00 / 0.00%2.20 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
P2Steamer Stack Order ReconstructionEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
S1Rotating Tray Layout ReconstructionEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
S2Hidden-Zone NavigationEasy50.00 / 50.00%40.00 / 40.00%60.00 / 60.00%
Medium10.00 / 10.00%0.00 / 0.00%20.00 / 20.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
F1Unknown Button Mapping and Slider NavigationEasy37.50 / 30.00%9.00 / 0.00%66.00 / 60.00%
Medium16.60 / 10.00%28.80 / 20.00%4.40 / 0.00%
Hard13.00 / 10.00%23.00 / 20.00%3.00 / 0.00%
F2Color-Block Trial OrderingEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
M1Workpiece Tracking and Route ReproductionEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
M2Memory Card MatchingEasy81.50 / 80.00%83.00 / 80.00%80.00 / 80.00%
Medium20.00 / 20.00%20.00 / 20.00%20.00 / 20.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%

Act

TaskMeanCleanRandom
DRDynamic response14.51 / 10.00%16.05 / 11.67%12.98 / 8.33%
DR1Continuous Conveyor Picking9.97 / 6.67%10.36 / 6.67%9.59 / 6.67%
DR2Light-Guided Button Response21.42 / 6.67%27.16 / 13.33%15.67 / 0.00%
DR3Periodic Gate Crossing and Withdrawal26.67 / 26.67%26.67 / 26.67%26.67 / 26.67%
DR4Ball-on-Tray Transport and Placement0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
PRPrecise execution9.23 / 7.78%8.96 / 7.78%9.50 / 7.78%
PR1Linear Resistor Adjustment42.50 / 40.00%41.66 / 40.00%43.33 / 40.00%
PR2Rotary Voltage Regulator Setting12.88 / 6.67%12.11 / 6.67%13.64 / 6.67%
PR3Marker Line Tracing0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
PR4Cloth Edge Alignment and Placement0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
PR5Oriented Stamp Placement0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
PR6Domino Chain Bell Ringing0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Results by difficulty
TaskLevelMeanCleanRandom
DR1Continuous Conveyor PickingEasy26.00 / 20.00%28.00 / 20.00%24.00 / 20.00%
Medium2.20 / 0.00%2.64 / 0.00%1.76 / 0.00%
Hard1.71 / 0.00%0.43 / 0.00%3.00 / 0.00%
DR2Light-Guided Button ResponseEasy40.10 / 20.00%52.00 / 40.00%28.20 / 0.00%
Medium13.20 / 0.00%16.28 / 0.00%10.12 / 0.00%
Hard10.95 / 0.00%13.20 / 0.00%8.70 / 0.00%
DR3Periodic Gate Crossing and WithdrawalEasy30.00 / 30.00%40.00 / 40.00%20.00 / 20.00%
Medium30.00 / 30.00%20.00 / 20.00%40.00 / 40.00%
Hard20.00 / 20.00%20.00 / 20.00%20.00 / 20.00%
DR4Ball-on-Tray Transport and PlacementEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
PR1Linear Resistor AdjustmentEasy43.27 / 40.00%60.00 / 60.00%26.54 / 20.00%
Medium43.07 / 40.00%24.99 / 20.00%61.14 / 60.00%
Hard41.16 / 40.00%40.00 / 40.00%42.31 / 40.00%
PR2Rotary Voltage Regulator SettingEasy11.86 / 0.00%13.12 / 0.00%10.59 / 0.00%
Medium15.93 / 10.00%3.20 / 0.00%28.66 / 20.00%
Hard10.84 / 10.00%20.00 / 20.00%1.68 / 0.00%
PR3Marker Line TracingEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
PR4Cloth Edge Alignment and PlacementEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
PR5Oriented Stamp PlacementEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
PR6Domino Chain Bell RingingEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%

Mind2Act

TaskMeanCleanRandom
CPMind–action coordination0.32 / 0.00%0.23 / 0.00%0.41 / 0.00%
CP01Conveyor Meal Assembly1.00 / 0.00%0.56 / 0.00%1.44 / 0.00%
CP02Rotating Cake Decoration0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
CP03Dynamic Rack Loading0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
CP04Dual-Arm Cooking0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
CP05Keyboard Sequence Reproduction1.18 / 0.00%0.49 / 0.00%1.86 / 0.00%
CP06Produce Weighing and Balancing0.64 / 0.00%1.28 / 0.00%0.00 / 0.00%
CP07Bridge Construction0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
CP08Reference-Guided Block Construction0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
CP09Whiteboard Drawing Reconstruction0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
CP10Predictive Ball Return and Red-Ball Avoidance0.41 / 0.00%0.00 / 0.00%0.82 / 0.00%
Results by difficulty
TaskLevelMeanCleanRandom
CP01Conveyor Meal AssemblyEasy1.50 / 0.00%0.00 / 0.00%3.00 / 0.00%
Medium1.50 / 0.00%1.67 / 0.00%1.33 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
CP02Rotating Cake DecorationEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
CP03Dynamic Rack LoadingEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
CP04Dual-Arm CookingEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
CP05Keyboard Sequence ReproductionEasy2.81 / 0.00%0.75 / 0.00%4.88 / 0.00%
Medium0.38 / 0.00%0.50 / 0.00%0.25 / 0.00%
Hard0.35 / 0.00%0.23 / 0.00%0.47 / 0.00%
CP06Produce Weighing and BalancingEasy1.00 / 0.00%2.00 / 0.00%0.00 / 0.00%
Medium0.50 / 0.00%1.00 / 0.00%0.00 / 0.00%
Hard0.42 / 0.00%0.84 / 0.00%0.00 / 0.00%
CP07Bridge ConstructionEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
CP08Reference-Guided Block ConstructionEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
CP09Whiteboard Drawing ReconstructionEasy0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Medium0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
Hard0.00 / 0.00%0.00 / 0.00%0.00 / 0.00%
CP10Predictive Ball Return and Red-Ball AvoidanceEasy0.50 / 0.00%0.00 / 0.00%1.00 / 0.00%
Medium0.50 / 0.00%0.00 / 0.00%1.00 / 0.00%
Hard0.23 / 0.00%0.00 / 0.00%0.47 / 0.00%

Efficiency

Avg. cost per attempt
≈ ¥8.26
Avg. task time
96.1 min
Avg. response time
35.6 s
Mind · Act · Mind2Act
TasksAvg. cost per attemptAvg. task timeAvg. response time
Mind≈ ¥8.5191.9 min38.7 s
Act≈ ¥6.9593.0 min35.4 s
Mind2Act≈ ¥9.31103.2 min33.3 s
Usage totals

Evaluation cost

900 selected official episodes across Clean and Random.

MetricRecorded totalCoverage
Model decisions100,632900 / 900 episodes
API request attempts107,979900 / 900 episodes
Upstream requests107,979107,979 / 107,979 requests
Input tokens32,352,392,211100,631 / 107,979 requests
Output tokens433,769,839100,631 / 107,979 requests
Cached input tokens (included)30,977,891,016100,570 / 107,979 requests
Reasoning output tokens (included)399,793,11896,345 / 107,979 requests
Cumulative API request time1,066 h 42 min 15 s107,979 / 107,979 requests
Cumulative inference time1,296 h 38 min 30 s899 / 900 episodes
Cumulative episode time1,439 h 10 min 57 s899 / 900 episodes
Cumulative run time1,465 h 29 min 43 s899 / 900 episodes
Estimated API costCNY 7,432.09100,570 / 107,979 requests
Cost basis

Recorded requests with complete input, cached-input and output usage.

CNY per 1M tokens: input 2; cached input 0.04; output 8.

Cost = uncached input × input rate + cached input × cache rate + output × output rate.

API platform billing, 1–10 Nov 2026.