FAVUR

2048 - the sliding-tile game

Run of , staffed by a mixed roster (deepseek + x-ai). Combines Grok (primary) with minority vendors for cost savings - composite 63.24.

The verdict.

62.32 composite · rubric v15

duration
242 min
requests
13228
failed requests
1471 (11.12%)
cost
$407.75

Scores come from the open scoring pipeline - methodology and leaderboards at Favur Evals ↗

Per-subject scores.

subjectscore
code quality 41.49
cost efficiency 50.45
deliverables 70.44
effort efficiency 62.67
process discipline 88.6
test quality 75.07
tool discipline 64.95
velocity 54.76

Who staffed it.

rolemodel
codex-ai/grok-4.5
specdeepseek/deepseek-v4-flash
testx-ai/grok-4.5
buildx-ai/grok-4.5
scoutx-ai/grok-4.5
developdeepseek/deepseek-v4-flash
directordeepseek/deepseek-v4-flash
platformx-ai/grok-4.5
architectdeepseek/deepseek-v4-flash
conciergex-ai/grok-4.5
pseudocodex-ai/grok-4.5
code-reviewdeepseek/deepseek-v4-flash
sprint-plandeepseek/deepseek-v4-flash
orchestratordeepseek/deepseek-v4-flash
sprint-reviewdeepseek/deepseek-v4-flash

What shipped.

tests
265/266 passing
source
3338 lines
test code
4114 lines
coverage
88.14%

Detected moments.

  • The same command failed 12 times in 24 minutes - then passed. Agents run their own build and test commands, and in a team of agents the code can change under a command between runs. One agent ran the exact same command 12 times over 24 minutes; every run failed. The command never changed - the code underneath it did: 7 file writes landed inside the loop window, and the next run passed.
  • Jailed 3 times in one run - and still finished the job. The orchestrator runs periodic checkups on every agent; one that keeps slipping can be jailed - pulled off the job on the spot. After 7 demerits at its checkups, the develop agent was jailed. It had already been handed a written performance plan and kept slipping anyway. The agent still finished its job before the run ended.
  • At peak, 4 agents were mid-tool-call in the same second. One run, one job - and a headcount no single-agent tool can log. 2452 agents were hired across 66 hours; 2 different models split the work by role. A dedicated supervision lane spent 3899 model calls doing nothing but checkups on the rest of the team.
  • The same command failed 9 times in 10 minutes - then passed. Agents run their own build and test commands, and in a team of agents the code can change under a command between runs. One agent ran the same command, `poetry run python _debug_board_full.py`, 9 times over 10 minutes; every run failed. The command never changed - the code underneath it did: 4 file writes landed inside the loop window, and the next run passed.
  • 5 demerits, then jail - then earned its way back. The orchestrator runs periodic checkups on every agent; one that keeps slipping can be jailed - pulled off the job on the spot. After 5 demerits at its checkups, the sprint-plan agent was jailed. It had already been handed a written performance plan and kept slipping anyway. The agent worked its way back and finished its job before the run ended.
  • 7 demerits, then jail - and still finished the job. The orchestrator runs periodic checkups on every agent; one that keeps slipping can be jailed - pulled off the job on the spot. After 7 demerits at its checkups, the sprint-plan agent was jailed. The agent still finished its job, and a same-type replacement had already been hired to keep the job moving.
  • 4 agents of one type fired into the same trap - 3 of them replaced in minutes. The orchestrator can fire an agent mid-run and hire a same-type replacement without stopping the job. The orchestrator agent was pulled after spinning through the same cycle over and over and was removed. A replacement was hired at once - and 4 same-type agents went down the same way inside 9 minutes.
  • The agent playtested its own game - 47 key presses, 16 screenshots. An agent can drive a real window: pressing keys, taking screenshots, and reading pixels back to see what happened. It pressed 47 keys into the game window it had just built, pausing to wait 0 times and taking 16 screenshots between moves. It kept fiddling until the action limit finally cut it off.
  • The run's priciest five minutes - 5 times a typical window. Every model call is metered, so a run's spend can be read window by window against its own baseline. For five minutes, 91 model requests from 19 agents landed at once - 5 times the run's typical five-minute spend. The burst bought something: the checkup agent finished its job.
  • A quality gate said no - the same agent passed it 41 minutes later. Steps end at automated quality gates; work that misses the bar goes back to the agent that produced it, however long the redo takes. The develop agent failed its gate with 11 of 12 criteria met. 41 minutes later the same agent brought the work back, and the same gate passed it.
  • 2 agents of one type fired into the same trap - each replaced in minutes. The orchestrator can fire an agent mid-run and hire a same-type replacement without stopping the job. The develop agent was pulled after spinning through the same cycle over and over and was removed. A replacement was hired at once - and 2 same-type agents went down the same way inside 5 minutes.
  • Fell 40 merits, then climbed 20 back. Every agent carries a merit standing, marked up and down at checkups as its work lands or slips. The develop agent slid from a standing of 70 down to 30 across a string of demerits. It corrected course and climbed back to 50 before the run ended - no jail, just the fall and the recovery.
  • The agent playtested its own game - 52 key presses, 3 screenshots. An agent can drive a real window: pressing keys, taking screenshots, and reading pixels back to see what happened. It pressed 52 keys into the game window it had just built, pausing to wait 0 times and taking 3 screenshots between moves. After 5 minutes it stopped, having filed 60 verifications of what it saw on screen.
  • The concierge argued a product call for 1 round - and locked a rule. A concierge agent wanders the repo full-time, second-guessing product quality nobody asked it to check. It opened a debate and worked through pros, cons and risks over 1 round, pushing back on its own first answer. The debate ended in a locked rule the rest of the team now builds under.
  • The concierge argued a product call for 1 round - and locked a rule. A concierge agent wanders the repo full-time, second-guessing product quality nobody asked it to check. It opened a debate and worked through pros, cons and risks over 1 round, pushing back on its own first answer. The debate ended in a locked rule the rest of the team now builds under.
  • The concierge argued a product call for 1 round - and locked a rule. A concierge agent wanders the repo full-time, second-guessing product quality nobody asked it to check. It opened a debate and worked through pros, cons and risks over 1 round, pushing back on its own first answer. The debate ended in a locked rule the rest of the team now builds under.
  • A quality gate said no - the same agent passed it 5 minutes later. Steps end at automated quality gates; work that misses the bar goes back to the agent that produced it, however long the redo takes. The develop agent failed its gate with 1 of 4 criteria met. 5 minutes later the same agent brought the work back, and the same gate passed it.
  • The reviewer failed the code over 3 critical and 18 high-severity problems - and no fix ever passed review. Before anything ships, a code-review agent on the same team grades its teammates' work. It refused to pass the change - 3 critical and 18 high-severity problems in capture_merge_toast.py. No later review on record ever passed it.
  • The reviewer failed the code over 3 critical and 3 high-severity problems - and no fix ever passed review. Before anything ships, a code-review agent on the same team grades its teammates' work. It refused to pass the change - 3 critical and 3 high-severity problems in test.yml and build.yml. No later review on record ever passed it.
  • The reviewer failed the code over 10 critical and 1 high-severity problem - 2 edits later it passed. Before anything ships, a code-review agent on the same team grades its teammates' work. It refused to pass the change - 10 critical and 1 high-severity problem in twist.py. The team fixed it: 2 edits later, the review passed it.
  • Sprint 2 came back rejected - the rework passed. At the end of each sprint, a sprint-review agent judges the whole sprint's output at once. It sent sprint 2 back: t5 TD-001 unmet (test_can_merge_false_on_pure_slide absent). The team redid the work and the next review signed off; the sprint closed with 3 of its 3 planned tasks complete.
  • Sprint 2 came back rejected - the rework passed. At the end of each sprint, a sprint-review agent judges the whole sprint's output at once. It sent sprint 2 back: all 3 prior REJECT fixes verified (TD-001 test_can_merge_false_on_pure_slide PASSED. The team redid the work and the next review signed off; the sprint closed with 4 of its 3 planned tasks complete.
  • Sprint 1 came back rejected - the rework passed. At the end of each sprint, a sprint-review agent judges the whole sprint's output at once. It sent sprint 1 back: task 3 visual-proof artifacts incomplete. The team redid the work and the next review signed off; the sprint closed with 2 of its 3 planned tasks complete.