Shared run

DeepSWE v1.1

completedHarbor tests · deepswe_verdict91 repos

Run

Judge

Harbor tests · deepswe_verdict

Duration

264h 11m

Policy

pass@1

Tokens

197M in / 77.7M out

Model comparison

Harbor tests · deepswe_verdict

Pass ratedeepswe_verdict · tokens

Grok 4.6 — thinking levels

grok-4.6-lowthinking low

113 attempts

59.3%

67/113/113

28.1M

Tasks

Task × model113 tasks · 10 models · deepswe_verdict

Task	grok-4.6-lowlow	grok-4.6-mediummedium	grok-4.6default (high)	grok-4.6-xhighxhigh	composer-2.5	gpt-5.6-luna-nonenone	gpt-5.6-luna-lowlow	gpt-5.6-lunadefault (medium)	gpt-5.6-luna-highhigh	gpt-5.6-luna-xhighxhigh
ycb_KaTeX_c6c65dc8a71a3
ycb_abs_c92ae892b40dd
ycb_abs_ce8ed642c7b27
ycb_actionlint_c4dc4f9512031
ycb_adaptix_c507ac5276499
ycb_aiomonitor_c9ecdc69d78cf
ycb_anko_c7cfc85538dd0
ycb_anko_cfdde3365c5c5
ycb_arcane_c7ae95515ffd0
ycb_arktype_c538742262688
ycb_awilix_c1d194536c83f
ycb_bandit_c064c8c29b1cd
ycb_bandit_c5278a07cc3f6
ycb_bandit_cc8cd89919ffe
ycb_boa_c417aaee9a01f
ycb_cattrs_c7b7b6270efcd
ycb_clack_c9a631a228b14
ycb_claude-code-by-agents_c6f171188a941
ycb_cliffy_cfdcb3ecb0343
ycb_csstree_c7cbad6e70694
ycb_dasel_cdcc49f0dbf48
ycb_dateutil_ca402289f335f
ycb_drizzle-orm_cb8ef82daae06
ycb_dynamodb-toolbox_c41b820ce10e8
ycb_dynamodb-toolbox_cb9040eb55e9b
ycb_effect_cfa87c163fef6
ycb_eicrud_c3e60703973c2
ycb_etree_c68e599d64846
ycb_expr_c8eab9fe50296
ycb_fastapi_cec25bde46134
ycb_fastapi_cf047f3cc8951
ycb_fd_c2214f507bd96
ycb_geo_c2187fb13a611
ycb_go-critic_c05b092c1f75d
ycb_go-genai_c6cc04f7a38c6
ycb_go-git_c19b0a27490c5
ycb_goreleaser_cf475e17d9f68
ycb_gql_cf00f950bfa74
ycb_happy-dom_c83d982312459
ycb_happy-dom_c989b313836db
ycb_helm_c69425c15295c
ycb_helm_ca36acc02dd92
ycb_httpx_c1e88823e965a
ycb_httpx_c536fe1960672
ycb_httpx_ca2be829e010e
ycb_igel_cc328cf4ca071
ycb_ink_c1a206b940222
ycb_ipython_c62c7e6177993
ycb_kcp-go_c3d2e8d96033e
ycb_kea_c06249fda4b84
ycb_kgateway_cd7ee0617d2fd
ycb_kombu_c5b831c22d8f2
ycb_kombu_c6fcd795a2a2e
ycb_koota_c6b5da2caa687
ycb_koota_c9ede00400924
ycb_koota_c9f7bd22123fe
ycb_koota_cc720222cbf68
ycb_koota_cf41cb1aebf0c
ycb_kysely_c6fa76e2712a5
ycb_langchain_c72da081c4b65
ycb_mashumaro_caf6eb8ad0f30
ycb_meriyah_cfd1375a8111f
ycb_mnamer_c2512256d5dcb
ycb_mobly_c36c1bcf4c83c
ycb_narwhals_c1da3bfe424ce
ycb_numba_ca3bbe1bc39ec
ycb_obsidian-linter_c33655279db71
ycb_obsidian-linter_c634d972a1557
ycb_obsidian-linter_c79d33987c361
ycb_ofetch_cc00dd8dca774
ycb_onedump_c94540ddab9b3
ycb_opa_c803dca332d8b
ycb_opa_ca2ad7b3adb02
ycb_optique_c5d9169a4d5e7
ycb_oxvg_c6c6c599e7865
ycb_participle_c0d2a0d9e636a
ycb_pebble_c3ab7566cfbd7
ycb_pest_c5b9a7f31615c
ycb_prometheus_c72cd18ccfe7d
ycb_prometheus_c73b813596ac6
ycb_psd-tools_c1be9bd659621
ycb_pwntools_cdcb70f96a02d
ycb_python-statemachine_c93eddad18a56
ycb_query_c71316ce65055
ycb_quill_c75769548bf2e
ycb_returns_c052a91546733
ycb_scc_cff77e0d04305
ycb_scriggo_c7e8d4017d566
ycb_skrub_c5834722c4500
ycb_sql-formatter_c6c4d96ac940d
ycb_sqlfmt_c858d7638c3d8
ycb_sqlite-utils_c4680466db75f
ycb_superjson_c0498942f86ac
ycb_task_c6ddf5e4b6ba5
ycb_tengo_c31e2a0da365a
ycb_tengo_c555b6485df2b
ycb_termenv_c2704929eb5e7
ycb_testem_c4caaec9c8462
ycb_testem_c757a0cd121ff
ycb_textual_c55bc288e66fa
ycb_textual_cf0df36d458e3
ycb_tomlkit_cabf1b96631e7
ycb_true-myth_cced9da513173
ycb_ts-pattern_cdf6a7c59c66e
ycb_updo_c986dc81cf914
ycb_valibot_c613387568944
ycb_vitest_c1ff736d301d3
ycb_vulture_c186917eed42b
ycb_wasmi_ced122529480f
ycb_wazero_ce2ab98d88ed9
ycb_yaegi_c1217d5c5a4b3
ycb_yjs_ce38c63135043
ycb_ytt_c2594b1ed8857

GPT-5.6 Luna — thinking levels

gpt-5.6-luna-nonethinking none

113 attempts

5.3%

6/113/113

900k

gpt-5.6-luna-lowthinking low

113 attempts

12.4%

14/113/113

1.3M

gpt-5.6-lunathinking default (medium)

113 attempts

25.7%

29/113/113

3.4M

gpt-5.6-luna-highthinking high

113 attempts

46%

52/113/113

11.9M

gpt-5.6-luna-xhighthinking xhigh

113 attempts

55.8%

63/113/113

37.9M

How scoring works ▸How scoring works ▾

pass rate = passed / (judged + agent failures) — a timeout, empty diff, agent error, or blown budget counts as a FAIL without a verdict (D48); infra failures and owner-cancelled attempts are excluded from the rate and annotated. DeepSWE Harbor tests write deepswe_verdict (reward 1/0). A cell still awaiting tests has no Harbor row yet. Official apply_failed is reward 0, not infra.