Shared run

Gomes Elixir Bench

completedGolden-diff judge · judge_verdict50 repos

Run

Judge

Golden-diff judge · judge_verdict

Duration

33h 31m

Policy

pass@3

Tokens

1180M in / 131M out

Model comparison

Golden-diff judge · judge_verdict

Pass ratejudge_verdict · tokens

Grok 4.6 — thinking levels

grok-4.6-lowthinking low

231 attempts

71.4%

165/231/231

63.4M

Tasks

Task × model77 tasks · 21 models · judge_verdict

Task	grok-4.6-lowlow	grok-4.6-mediummedium	grok-4.6default (high)	grok-4.6-xhighxhigh	gemini-3.7-flash-lowlow	gemini-3.7-flash-mediummedium	gemini-3.7-flashdefault (high)	claude-sonnet-5-lowlow	claude-sonnet-5-mediummedium	claude-sonnet-5default (high)	claude-sonnet-5-xhighxhigh	composer-2.5	gpt-5.6-sol-nonenone	gpt-5.6-sol-lowlow	gpt-5.6-soldefault (medium)	gpt-5.6-sol-highhigh	gpt-5.6-sol-xhighxhigh	claude-opus-5-lowlow	claude-opus-5-mediummedium	claude-opus-5default (high)	claude-opus-5-xhighxhigh
ycb_absinthe_cc029a611640a
ycb_absinthe_cde0e411f3ca4
ycb_arbor_c1c63aaef2d19
ycb_ash_oban_c5d117ed2006d
ycb_ash_phoenix_c115df93dab63
ycb_aws-elixir_c3e2a31c8264d
ycb_aws-elixir_c6a333d6439fb
ycb_aws-elixir_c8d238b828edb
ycb_badge_forge_c38d45ec2fe7f
ycb_badge_forge_cd4f3ce6fd71b
ycb_broadway_dashboard_cc3f88329c050
ycb_broadway_kafka_c1cadc01c4bb7
ycb_canary_c618944e649db
ycb_coherence_caa5e9c0964c5
ycb_ecto_ch_c794351260772
ycb_ecto_job_c1e283852c64b
ycb_ecto_shorts_c9653ea798f2b
ycb_elixir-ls_c41fdf45b47b2
ycb_elixir-secure-coding_c19e47196759c
ycb_elixir-secure-coding_c38cf33cc0df7
ycb_elixir_c31b41ee71f20
ycb_elixir_cf92b24dd1413
ycb_error-tracker_c5f5d814e34c3
ycb_error-tracker_ce0ae6f5da7ba
ycb_ewebmachine_c01d38d8b26e9
ycb_flop_ca90f6016c18a
ycb_flop_cd5eedddd6ae7
ycb_flop_cd9518985d9ec
ycb_free_oban_ui_c0c06aff25187
ycb_gen_queue_oban_c1341caf5e9d0
ycb_grpc_c219a327538ce
ycb_grpc_c9844499e466f
ycb_grpc_cde97e87c5663
ycb_guardian_cb1a7c817da9c
ycb_hammer_c54162aba084d
ycb_hammer_c7d7967f898c2
ycb_hexpm_c31a99efa0c0b
ycb_hexpm_c38c434c658a5
ycb_hexpm_cf39d2a3774b6
ycb_livebook_cb2a8416d1490
ycb_newsletter_c7398ba873fa4
ycb_nostrum_c132209106ac7
ycb_oban-console_cd51bcc3df2e2
ycb_oban-console_cf7bd06fd8520
ycb_oban-console_cffee0c1b05fb
ycb_oban-events_c537baa4afe98
ycb_oban-events_cac675fcaf7dc
ycb_oban_chore_c84c5b0cf74b2
ycb_oban_chore_ccb842290f6db
ycb_oban_doctor_c544336d370fd
ycb_oban_doctor_cce018dbc25a7
ycb_oban_notifiers_phoenix_c98d2046ad99e
ycb_oban_web_cd0810e0f88e4
ycb_ocelot_c821e8113d3af
ycb_off_broadway_kafka_c5d8a4ece71d1
ycb_paginator_c575c44421756
ycb_paginator_c9e5b1d21b717
ycb_petal_components_c1228638d0660
ycb_petal_components_cc4256bf32ca4
ycb_petal_components_cc81b8543fae2
ycb_phoenix_c3f002b784a80
ycb_phoenix_live_view_c6b7765803c68
ycb_phoenix_live_view_ca432466a4f91
ycb_phoenix_live_view_cd0c52c5a51de
ycb_phoenix_slime_cf450fa355064
ycb_plug_logger_json_cdd03a2458401
ycb_plug_rails_cookie_session_store_c64910eb230c1
ycb_plug_rails_cookie_session_store_c82656751942f
ycb_prometheus-plugs_ca53521f96c1d
ycb_sentry-elixir_cb50b965b7a7f
ycb_spider_man_cb05b70718d91
ycb_sqlite_ecto2_c7fccf51dbfa2
ycb_surface_c1f105ed5e00c
ycb_surface_cde97d872276d
ycb_tidewave_phoenix_c8dbb44e2083a
ycb_warehouse_c398d21a33f91
ycb_warehouse_c559c487675ad

Gemini 3.7 Flash — thinking levels

gemini-3.7-flash-lowthinking low

231 attempts

14.7%

34/231/231

227M

gemini-3.7-flash-mediumthinking medium

231 attempts

23.4%

54/231/231

279M

gemini-3.7-flashthinking default (high)

231 attempts

21.2%

49/231/231

253M

Claude Sonnet 5 — thinking levels

claude-sonnet-5-lowthinking low

231 attempts

51.5%

119/231/231

3.2M

claude-sonnet-5-mediumthinking medium

231 attempts

61%

141/231/231

4.5M

claude-sonnet-5thinking default (high)

231 attempts

67.5%

156/231/231

7.7M

claude-sonnet-5-xhighthinking xhigh

231 attempts

70.1%

162/231/231

11.9M

GPT-5.6 Sol — thinking levels

gpt-5.6-sol-nonethinking none

231 attempts

74.5%

172/231/231

23M

gpt-5.6-sol-lowthinking low

231 attempts

81.4%

188/231/231

11.2M

gpt-5.6-solthinking default (medium)

231 attempts

91.3%

211/231/231

11.5M

gpt-5.6-sol-highthinking high

231 attempts

95.2%

219/230/231

14M

1 judge failed

gpt-5.6-sol-xhighthinking xhigh

231 attempts

95.7%

221/231/231

19.7M

Claude Opus 5 — thinking levels

claude-opus-5-lowthinking low

231 attempts

65.8%

152/231/231

5.6M

claude-opus-5-mediumthinking medium

231 attempts

76.6%

177/231/231

11M

claude-opus-5thinking default (high)

231 attempts

77.5%

179/231/231

17.7M

claude-opus-5-xhighthinking xhigh

231 attempts

74.8%

172/230/231

28.8M

1 judge failed

How scoring works ▸How scoring works ▾

pass rate = passed / (judged + agent failures) — a timeout, empty diff, agent error, or blown budget counts as a FAIL without a verdict (D48); infra failures and owner-cancelled attempts are excluded from the rate and annotated. Verdicts by grok-4.5 · effort=high · fast (pinned, D22) · verdict rubric v5 (D54). PASS = a correct solution to the same problem as the reference, equivalent or a valid alternative; FAIL = anything less. The only scorer since D26 (judge-only grading).