Powered by YourCodingBench — benchmark coding models on your own repo's commits. sign in
Shared benchmark
50 repos: DockYard-Academy/newsletter, ExHammer/hammer, Kraigie/nostrum, MikaAK/ecto_shorts, absinthe-graphql/absinthe, alejolcc/oban_chore, allocator-one/oban-events, ash-project/ash_oban, ash-project/ash_phoenix, aws-beam/aws-elixir, bbalser/off_broadway_kafka, bleacherreport/plug_logger_json, cconstantin/plug_rails_cookie_session_store, coryodaniel/arbor, cpjk/canary, duffelhq/paginator, elixir-broadway/broadway_dashboard, elixir-broadway/broadway_kafka, elixir-error-tracker/error-tracker, elixir-grpc/grpc, elixir-lsp/elixir-ls, elixir-sqlite/sqlite_ecto2, erlef/elixir-secure-coding, exercism/elixir, feng19/spider_man, getsentry/sentry-elixir, hexpm/hexpm, kbrw/ewebmachine, linqueta/oban-console, livebook-dev/livebook, mbuhot/ecto_job, mimiquate/ocelot, oban-bg/badge_forge, oban-bg/oban_notifiers_phoenix, oban-bg/oban_web, petalframework/petal_components, phoenixframework/phoenix, phoenixframework/phoenix_live_view, plausible/ecto_ch, prometheus-erl/prometheus-plugs, sevab/free_oban_ui, slime-lang/phoenix_slime, smpallen99/coherence, surface-ui/surface, system76/warehouse, tidewave-ai/tidewave_phoenix, tomasz-tomczyk/oban_doctor, trax-retail/gen_queue_oban, ueberauth/guardian, woylie/flop · created Aug 17, 2026 · 77 / 500 tasks
Golden-diff judge · judge_verdict
Golden-diff judge writes judge_verdict. 21 models have a rate.
Golden-diff judge · judge_verdict
| grok-4.6-lowlow | 165/231 passed | 231 | 0 |
| grok-4.6-mediummedium | 178/231 passed | 231 | 0 |
| grok-4.6default (high) | 182/231 passed | 231 | 0 |
| grok-4.6-xhighxhigh | 184/231 passed | 231 | 0 |
| gemini-3.7-flash-lowlow | 34/231 passed | 231 | 0 |
| gemini-3.7-flash-mediummedium | 54/231 passed | 231 | 0 |
| gemini-3.7-flashdefault (high) | 49/231 passed | 231 | 0 |
| claude-sonnet-5-lowlow | 119/231 passed | 231 | 0 |
| claude-sonnet-5-mediummedium | 141/231 passed | 231 | 0 |
| claude-sonnet-5default (high) | 156/231 passed | 231 | 0 |
| claude-sonnet-5-xhighxhigh | 162/231 passed | 231 | 0 |
| composer-2.5 | 202/231 passed | 231 | 0 |
| gpt-5.6-sol-nonenone | 172/231 passed | 231 | 0 |
| gpt-5.6-sol-lowlow | 188/231 passed | 231 | 0 |
| gpt-5.6-soldefault (medium) | 211/231 passed | 231 | 0 |
| gpt-5.6-sol-highhigh | 219/230 passed | 231 | 1 |
| gpt-5.6-sol-xhighxhigh | 221/231 passed | 231 | 0 |
| claude-opus-5-lowlow | 152/231 passed | 231 | 0 |
| claude-opus-5-mediummedium | 177/231 passed | 231 | 0 |
| claude-opus-5default (high) | 179/231 passed | 231 | 0 |
| claude-opus-5-xhighxhigh | 172/230 passed | 231 | 1 |
| PR / commit | Repo | Title | Merged | +/− | Source |
|---|---|---|---|---|---|
| b1a7c81 | ueberauth/guardian | feat(plug): select the verifying secret from the connection (#748) | Aug 17, 2026 | +771 −11 | manual |
| 31b41ee | exercism/elixir |
judge = pass rate, passed / (judged + agent failures) — a timeout, empty diff, agent error, or blown budget counts as a FAIL without a verdict (D48). grok-4.5 pinned (D22) · verdict rubric v5 (D54), the only scorer since D26. excluded = attempts outside the rate: infra failures (not the model's fault), owner-cancelled, and attempts still awaiting judge.
Drop this iframe on any page. It only resolves for a sealed public benchmark, and only plots runs the owner also made public.
<iframe src="https://yourcodingbench.com/embed/benchmarks/19a00b84-2dff-456c-989c-0f52fe482710" title="Gomes Elixir Bench score vs cost" width="100%" height="480" style="border:0;border-radius:12px;overflow:hidden;background:transparent" loading="lazy"></iframe>
| Status | Models | Tasks | Judge | Started | Duration |
|---|---|---|---|---|---|
| completed | grok-4.6-low, grok-4.6-medium, grok-4.6, grok-4.6-xhigh, gemini-3.7-flash-low, gemini-3.7-flash-medium, gemini-3.7-flash, claude-sonnet-5-low, claude-sonnet-5-medium, claude-sonnet-5, claude-sonnet-5-xhigh, composer-2.5, gpt-5.6-sol-none, gpt-5.6-sol-low, gpt-5.6-sol, gpt-5.6-sol-high, gpt-5.6-sol-xhigh, claude-opus-5-low, claude-opus-5-medium, claude-opus-5, claude-opus-5-xhigh | 77 | grok-4.6-low 165/231·grok-4.6-medium 178/231·grok-4.6 182/231·grok-4.6-xhigh 184/231·gemini-3.7-flash-low 34/231·gemini-3.7-flash-medium 54/231·gemini-3.7-flash 49/231·claude-sonnet-5-low 119/231·claude-sonnet-5-medium 141/231·claude-sonnet-5 156/231·claude-sonnet-5-xhigh 162/231·composer-2.5 202/231·gpt-5.6-sol-none 172/231·gpt-5.6-sol-low 188/231·gpt-5.6-sol 211/231·gpt-5.6-sol-high 219/230·gpt-5.6-sol-xhigh 221/231·claude-opus-5-low 152/231·claude-opus-5-medium 177/231·claude-opus-5 179/231·claude-opus-5-xhigh 172/230 | Aug 17, 12:59 | 33h 31m |
| Add practice exercise `line-up` (#1620) |
| Mar 11, 2026 |
| +329 −0 |
| manual |
| f92b24d | exercism/elixir | Add practice exercise `prism` (#1617) | Mar 10, 2026 | +631 −0 | manual |
| 19e4719 | erlef/elixir-secure-coding | feat: self-evaluation and updates for owasp module | Apr 5, 2025 | +269 −118 | manual |
| 38cf33c | erlef/elixir-secure-coding | fix: smart cell to_source should not include string representation of… | Apr 14, 2025 | +63 −265 | manual |
| 9e5b1d2 | duffelhq/paginator | feat: add support for all asc/desc sorting order combinations (#136) | Feb 7, 2022 | +456 −58 | manual |
| 575c444 | duffelhq/paginator | Get intial expression pagination working | May 17, 2023 | +337 −6 | manual |
| 7fccf51 | elixir-sqlite/sqlite_ecto2 | Update to support Ecto 2.2. (#193) | Aug 29, 2017 | +291 −289 | manual |
| d951898 | woylie/flop | support cursor pagination on nullable columns | Aug 15, 2026 | +385 −99 | manual |
| a90f601 | woylie/flop | support ordering by custom fields | Aug 13, 2026 | +399 −70 | manual |
| d5eeddd | woylie/flop | support filtering with field_dynamic | Aug 14, 2026 | +382 −81 | manual |
| 1e28385 | mbuhot/ecto_job | Adapt code to run with both Mysql and Postgresql | Sep 9, 2020 | +233 −80 | manual |
| 1c63aae | coryodaniel/arbor | chore: added .formatter.exs and format the codebase using Elixir 1.6 … | Nov 14, 2018 | +222 −154 | manual |
| 7943512 | plausible/ecto_ch | Standardize ClickHouse quoting (#304) | Aug 16, 2026 | +271 −74 | manual |
| 9653ea7 | MikaAK/ecto_shorts | refactor query builder api *breaking change* (#60) | Oct 16, 2024 | +325 −166 | manual |
| 1228638 | petalframework/petal_components | fix(dropdown): panel flips upward when the viewport leaves no room be… | Aug 15, 2026 | +649 −10 | manual |
| c4256bf | petalframework/petal_components | fix(data_table): put the menus in the page - stop chasing the viewpor… | Aug 7, 2026 | +432 −120 | manual |
| c81b854 | petalframework/petal_components | feat(data_table): row selection - tri-state header, morphing toolbar … | Aug 7, 2026 | +481 −59 | manual |
| 8dbb44e | tidewave-ai/tidewave_phoenix | Remove HTTP handling from core MCP logic | Aug 11, 2026 | +413 −369 | manual |
| f450fa3 | slime-lang/phoenix_slime | Add phoenix 1.3 phx prefixed mix tasks (#56) | Oct 25, 2017 | +463 −0 | manual |
| aa5e9c0 | smpallen99/coherence | Add back the gettext fix in ViewHelpers. closes #391 | Oct 14, 2018 | +207 −95 | manual |
| 115df93 | ash-project/ash_phoenix | improvement: Split up usage rules into sub-rules (#458) | Feb 11, 2026 | +261 −231 | manual |
| 01d38d8 | kbrw/ewebmachine | chore: fixed warning in 1.18 | Sep 17, 2025 | +160 −150 | manual |
| 618944e | cpjk/canary | Consistent API for plugs and hooks | Feb 11, 2025 | +193 −108 | manual |
| dd03a24 | bleacherreport/plug_logger_json | ready 0.7.0 release (#24) | Apr 9, 2019 | +234 −155 | manual |
| a53521f | prometheus-erl/prometheus-plugs | add formatter and update prometheus_ex dep requirement | May 15, 2018 | +395 −256 | manual |
| 64910eb | cconstantin/plug_rails_cookie_session_store | Update dependencies, fix tests and support Rails 4 and Rails 5 (#13) | Jan 15, 2019 | +309 −267 | manual |
| 8265675 | cconstantin/plug_rails_cookie_session_store | Starting of with current plug cookie store | Dec 29, 2014 | +499 −2 | manual |
| 1cadc01 | elixir-broadway/broadway_kafka | Implement shared client (#133) | Oct 11, 2023 | +552 −105 | manual |
| c3f8832 | elixir-broadway/broadway_dashboard | Fix crash when named using `:via` (#14) | Sep 8, 2021 | +220 −136 | manual |
| b05b707 | feng19/spider_man | enhance: easy to show throughput infos on livebook | Feb 8, 2024 | +275 −62 | manual |
| 5d8a4ec | bbalser/off_broadway_kafka | Updating broadway (#19) | Mar 6, 2020 | +197 −114 | manual |
| 398d21a | system76/warehouse | messy but working part picking grpc endpoint | Aug 27, 2021 | +329 −21 | manual |
| 559c487 | system76/warehouse | Fix bug with demand decreasing for non pulled Components (#14) | Feb 2, 2022 | +96 −247 | manual |
| d0810e0 | oban-bg/oban_web | Display awaitable signals in the job details page | May 11, 2026 | +283 −21 | manual |
| 5d117ed | ash-project/ash_oban | improvement: add support for snoozing and cancelling Oban jobs from a… | Feb 23, 2026 | +322 −30 | manual |
| e0ae6f5 | elixir-error-tracker/error-tracker | feat: Light theme (#179) | May 9, 2026 | +334 −44 | manual |
| 5f5d814 | elixir-error-tracker/error-tracker | Mute error occurrences (#130) | Mar 3, 2025 | +290 −55 | manual |
| 98d2046 | oban-bg/oban_notifiers_phoenix | Prepare for CI, docs, licensing, and publication | Nov 6, 2023 | +371 −16 | manual |
| cb84229 | alejolcc/oban_chore | Showing history (#9) | Jun 13, 2026 | +675 −53 | manual |
| 84c5b0c | alejolcc/oban_chore | Add tests (#6) | Jun 3, 2026 | +477 −39 | manual |
| ffee0c1 | linqueta/oban-console | feat: Add Jobs tests | Oct 21, 2024 | +393 −15 | manual |
| d51bcc3 | linqueta/oban-console | feat: Add Jobs tests | Oct 22, 2024 | +379 −28 | manual |
| f7bd06f | linqueta/oban-console | feat: Add Queues tests | Oct 21, 2024 | +374 −22 | manual |
| 1341caf | trax-retail/gen_queue_oban | Repository Setup (#1) | Oct 30, 2019 | +584 −0 | manual |
| d4f3ce6 | oban-bg/badge_forge | Add python with oban and enqueue from elixir | Jan 26, 2026 | +837 −0 | manual |
| 38d45ec | oban-bg/badge_forge | Generate badge pdfs with weasyprint | Jan 26, 2026 | +162 −174 | manual |
| 0c06aff | sevab/free_oban_ui | Implement basic Oban Jobs UI | Sep 18, 2024 | +718 −49 | manual |
| 821e811 | mimiquate/ocelot | feat: add basic ui | Apr 13, 2026 | +544 −0 | manual |
| 537baa4 | allocator-one/oban-events | refactor to remove :if, improve oban opts handling | Oct 25, 2025 | +429 −231 | manual |
| ac675fc | allocator-one/oban-events | add event struct additional metadata, testing helpers, update readme | Oct 24, 2025 | +486 −88 | manual |
| ce018db | tomasz-tomczyk/oban_doctor | Add SmartEngineNotConfigured check (#7) | Jan 29, 2026 | +491 −5 | manual |
| 544336d | tomasz-tomczyk/oban_doctor | Update checks for Oban named state groups and add doc references | Jan 25, 2026 | +282 −28 | manual |
| 7398ba8 | DockYard-Academy/newsletter | Send a welcome message to new subscribers | Jan 15, 2023 | +452 −0 | manual |
| de0e411 | absinthe-graphql/absinthe | perf: Avoid rewalking the tree for suspended fields (#1450) | Jul 14, 2026 | +738 −63 | manual |
| c029a61 | absinthe-graphql/absinthe | feat: add description support for operations and fragments (#1404) | Apr 3, 2026 | +608 −8 | manual |
| 41fdf45 | elixir-lsp/elixir-ls | folding_range: delete dead Token/TokenPair/SpecialToken tokenizer mod… | Jun 14, 2026 | +1 −606 | manual |
| b50b965 | getsentry/sentry-elixir | feat: support for log_byte and trace_metric_byte (#1134) | Jul 31, 2026 | +810 −83 | manual |
| de97e87 | elixir-grpc/grpc | feat: ETS-backed LB with per-request pick, remove persistent_term wri… | Jun 29, 2026 | +691 −207 | manual |
| 9844499 | elixir-grpc/grpc | fix(gun): keep named channels alive after original caller exits (#519) | May 14, 2026 | +524 −94 | manual |
| 219a327 | elixir-grpc/grpc | chore: speed-up test suite (#565) | Jul 27, 2026 | +227 −129 | manual |
| b2a8416 | livebook-dev/livebook | Escape environment variables in app server shell instructions | Aug 4, 2026 | +363 −168 | manual |
| 8d238b8 | aws-beam/aws-elixir | Update services based on release-2026-06-15 of AWS Go SDK | Jun 16, 2026 | +829 −5 | manual |
| 6a333d6 | aws-beam/aws-elixir | Update services based on release-2026-06-19 of AWS Go SDK | Jun 20, 2026 | +734 −56 | manual |
| 3e2a31c | aws-beam/aws-elixir | Update services based on release-2026-06-18 of AWS Go SDK | Jun 19, 2026 | +444 −48 | manual |
| 1322091 | Kraigie/nostrum | Dave (#719) | Mar 9, 2026 | +448 −149 | manual |
| 3f002b7 | phoenixframework/phoenix | Streamline generated AGENTS.md usage rules (#6775) | Aug 4, 2026 | +47 −327 | manual |
| a432466 | phoenixframework/phoenix_live_view | Handle writer errors like other entry errors (#4370) | Aug 14, 2026 | +701 −106 | manual |
| d0c52c5 | phoenixframework/phoenix_live_view | Fix max_entries not working properly with auto uploads (#4373) | Aug 14, 2026 | +449 −18 | manual |
| 6b77658 | phoenixframework/phoenix_live_view | Allow cancelling client-side live navigation (#4332) | Jul 27, 2026 | +438 −20 | manual |
| 1f105ed | surface-ui/surface | Add new events and move some event-related logic to surface_form | Sep 18, 2024 | +204 −385 | manual |
| de97d87 | surface-ui/surface | Fix compile/tests issues with Elixir 1.19 and LV 1.1 (#772) | Jan 29, 2026 | +230 −190 | manual |
| f39d2a3 | hexpm/hexpm | Select latest releases with stored SemVer keys (#1832) | Aug 15, 2026 | +718 −59 | manual |
| 31a99ef | hexpm/hexpm | Tell people why we removed their package or account (#1791) | Aug 1, 2026 | +643 −38 | manual |
| 38c434c | hexpm/hexpm | Fix removing packages from policy overrides (#1799) | Aug 4, 2026 | +596 −44 | manual |
| 54162ab | ExHammer/hammer | Add before_clean callback on bucket expiry (#140) (#177) | Mar 30, 2026 | +593 −71 | manual |
| 7d7967f | ExHammer/hammer | Fix key type inconsistency in backend implementations (#143) | Jul 18, 2025 | +218 −87 | manual |