Shared run

neondatabase/neon · claude-opus-4-8, claude-opus-5, gemini-3-flash, claude-sonnet-4

running

started Jul 24, 19:05 UTC · still running · pass@1 · tokens n/a · cost —

benchmark my first benchmark based on neon's source code

fingerprint 99b0ca43e718

{
  "api": "v1",
  "caps": {
    "agentWallClockMin": 30
  },
  "driver": "cloud-agent",
  "harness": "cloud-agent",
  "rulesIncluded": true,
  "promptTemplate": 2,
  "cursorAgentVersion": null
}

Model comparison

run in progress — 63/64 attempts done

claude-opus-4-816 attempts

claude-opus-515 attempts +1 in flight

gemini-3-flash16 attempts

claude-sonnet-416 attempts

judge verdicts (pass rate)

15/16 passed

11/15 passed

7/16 passed

5/16 passed

Tokens — agent usage

14.8M total — 13.4M in / 1.4M out

63 of 63 terminal attempts reporting

Agent-phase tokens only, per-model split on the cards below. Judge-call usage is not measured today. Missing usage reads n/a — never a silent 0.

claude-opus-4-8

16 attempts · avg duration 3m 56s

tokens 1.6k in / 291k out · mean 18.3k/attempt

judge (pass rate)15/16 passed

claude-opus-5

15 attempts (+1 in flight) · avg duration 6m 52s

tokens 1.8k in / 372k out · mean 24.9k/attempt

judge (pass rate)11/15 passed

gemini-3-flash

16 attempts · avg duration 4m 25s

tokens 13.3M in / 490k out · mean 862k/attempt

judge (pass rate)7/16 passed

judge = judge_verdict pass rate (passed/judged) — gpt-5.6-luna · reasoning=none (pinned, D22) · verdict rubric v2 (D27: PASS = a correct solution to the same problem as the reference, equivalent or a valid alternative; FAIL = anything less). The only scorer since D26 (judge-only grading).

Task	claude-opus-4-8	claude-opus-5	gemini-3-flash	claude-sonnet-4
PR #12668 · ycb_neon_pr12668	PASS	PASS	PASS	FAIL
PR #12684 · ycb_neon_pr12684	PASS	PASS	PASS	FAIL
PR #12688 · ycb_neon_pr12688	PASS	PASS	PASS	FAIL
PR #12694 · ycb_neon_pr12694	PASS	PASS	FAIL	FAIL
PR #12705 · ycb_neon_pr12705	PASS	FAIL	FAIL	FAIL
PR #12727 · ycb_neon_pr12727	PASS	FAIL	FAIL	FAIL
PR #12745 · ycb_neon_pr12745	PASS	—agent_running	FAIL	FAIL
PR #12754 · ycb_neon_pr12754	PASS	PASS	FAIL	FAIL
PR #12772 · ycb_neon_pr12772	FAIL	FAIL	FAIL	FAIL
PR #12777 · ycb_neon_pr12777	PASS	PASS	PASS	PASS
PR #12782 · ycb_neon_pr12782	PASS	PASS	PASS	PASS
PR #12785 · ycb_neon_pr12785	PASS	FAIL	FAIL	FAIL
PR #12793 · ycb_neon_pr12793	PASS	PASS	FAIL	PASS
PR #12819 · ycb_neon_pr12819	PASS	PASS	FAIL	FAIL
PR #12826 · ycb_neon_pr12826	PASS	PASS	PASS	PASS
PR #12899 · ycb_neon_pr12899	PASS	PASS	PASS	PASS