YourCodingBench

How it works

1. 01

   Connect a repo

   Sign in with GitHub and pick the repository you want to benchmark — private repos included.

2. 02

   Mine merged PRs

   A heuristic funnel plus an LLM pass reconstructs the task each PR answered.

3. 03

   Curate

   Accept, edit, or reject suggested tasks — or force-include any PR from the catalog.

4. 04

   Freeze

   Verification locks an immutable benchmark version: determinism, discriminating tests, no leakage.

5. 05

   Run models

   The Cursor agent attempts every task in isolated Cursor Cloud Agent VMs, once per model. The model is the only variable.

6. 06

   Compare

   Every run lands in your dashboard — full transcripts, per-task scores, side-by-side model comparison.

The spine is live end to end: real merged PRs become tasks, cloud runs on Cursor Cloud Agents attempt them once per model, and each attempt is graded by a pinned LLM judge against the PR’s real diff — full transcripts and scores in the runs dashboard.

• First-party results store

  Runs land in the dashboard with full transcripts, diffs, usage, and per-task scores — no external accounts.

  Done

• Cloud runs end to end

  UI-triggered runs on Cursor Cloud Agents: history-free snapshot repos, one VM per attempt, server-side diff freeze — no grading VM at all.

  Done

• Judge-only grading

  A pinned judge (gpt-5.6-luna, reasoning=none) grades every completed attempt server-side against the reference diff; every run is exactly one attempt per task × model.

  Done

• Miner: ingest, funnel, scoring through Cursor

  PR ingestion, the heuristic funnel, and the LLM scoring pass all route through Cursor — gold-set labels are the open half.

  In progress

• Builder + freeze, transcript audit

  Curation UI and immutable frozen versions; the integrity transcript scan. (Env-pin pre-warm dropped — D28 provisions no environment.)

  Up next