Curated benchmarks
The authoritative list lives in
benchgate/curated.txt. Edit that file to add or remove benchmarks; the gate script picks it up automatically.
Running locally
Threshold
The default regression threshold is 25%. To change it:- Globally for CI: update
THRESHOLD_PCTin.github/workflows/bench.yml. - For a single local run:
THRESHOLD_PCT=10 ./scripts/bench-gate.sh --check diff.txt.
CI workflow
Thebench workflow (.github/workflows/bench.yml) runs on every pull request and on pushes to main.
Key design decisions:
- Base caching: the base-branch benchmark output is cached by commit SHA so re-runs of the same PR do not re-benchmark the base.
- Cold-base fallback: when no cache exists the workflow checks out the base commit, runs the benchmarks, saves the cache, then returns to the PR SHA.
- PR comment: on pull requests the diff is posted as a PR comment so reviewers see the delta without downloading artifacts.
- Artifacts:
pr-bench.txt,base-bench.txt, anddiff.txtare uploaded as workflow artifacts (retained 90 days).
Adding a benchmark
- Write the benchmark in the appropriate package following existing conventions (see
internal/bench/for examples). - Add the benchmark name to
benchgate/curated.txtwith a comment explaining what regression it catches. - Run
BENCH_TIME=1x BENCH_COUNT=1 ./scripts/bench-gate.shlocally to confirm it appears and exits 0.
Noisy benchmark skip annotation
If a benchmark routinely produces more than 10% noise, add a# gate:skip line in benchgate/curated.txt: