Primary Top50 metric = verified distinct-rule bugs under the frozen logical evidence budget. Ties remain ties; tokens, cost, and elapsed time are diagnostics only.
| Source | Target | Overhead (variable blow-up) |
|---|
The model first freezes the 50 rules it considers most bug-prone, then investigates each rule in a fresh isolated episode with the same logical budget. You provide the API key; we independently re-check every submitted bug before it scores.
One Docker image: the solver (pred) plus the rule library, pinned to the benchmark version.
OpenAI, Anthropic, DeepSeek, a self-hosted endpoint — you bring the API key.
Source-only triage freezes a Top50. Each rule then gets 10 model generations, 12 shell actions, 24 pred calls (10 solves), and exactly two submit attempts.
Upload submission.json with benchmark.submit. No web form.
We re-derive every claimed bug with our own solver and keep only the ones that reproduce.
only reproducible bugs are scoredModels are ordered only by confirmed distinct-rule bugs at the same frozen contract. Equal bug counts are tied.
Each reduction turns problem A into an equivalent problem B. It's buggy on an input when solving that input directly gives a different answer than solving it through the reduction.
Every model uses the same target, prompt, Top50 size, per-rule logical counters, and two submit opportunities. Wall-clock time, network delay, tokens, and cost never break ties.
Run the dockerized runner against problem-reductions v0.6.0, then upload the submission.json it produces. Every certificate is re-verified by pred before it counts.
submission.jsonConfigure once, validate with one tiny real call, then run.
cp submission.env.example submission.env # model · key make preflight # validate config with one tiny real call make run # → ./out/submission.json
The backend picks it up and re-verifies every certificate.
export PRB_SUBMIT_URL=<intake endpoint>
PRB_ACCESS_APP="${PRB_SUBMIT_URL%/submit}"
PRB_ACCESS_TOKEN="$(cloudflared access login --no-verbose --auto-close "$PRB_ACCESS_APP")" \
python -m benchmark.submit --predictions out/submission.json
Self-reported counts are advisory; only confirmed distinct-rule bugs are ranked.
A controlled measurement of how well a model prioritizes likely-buggy rules and turns bounded evidence into verified counterexamples. Every claimed bug is independently re-verified server-side.
@misc{TODO_citation_key,
title = {TODO},
author = {TODO},
year = {TODO},
url = {TODO}
}