Skip to content

Name each benchmark's winner and lint it - #243

Merged
JuanVqz merged 4 commits into
mainfrom
feature/benchmark-claims
Sep 26, 2026
Merged

JuanVqz merged 4 commits into
mainfrom
feature/benchmark-claims

Conversation

@JuanVqz

@JuanVqz JuanVqz commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Each benchmark now says which report should win through its method names: fastest, faster, fast, then slow, slower, slowest. The winner is listed first, so the claim is readable from the code, with no README prose needed.
  • A new lint rule checks it, so the results site (next) can compare every run against the file's claim.

Changes

Step 1: name every benchmark's winner in its method names

  • Every report is wrapped in a method named by rank, so they all pay the same call cost. Names are relative: slow only means slower than fast. Where a ranking could look odd, a one-line comment says why, for example in length-vs-size-vs-count.rb: "Array#size is an alias of Array#length, so these two should tie."
  • 11 files that ran code inline or used other names were converted. Their rankings follow CI run 35927028164 (26 jobs), comparing ratios within each job.
  • start_with-vs-substring-==.rb reported code strings. Each string now becomes a method, so its numbers are not comparable with older runs.
  • raise-vs-e2mmap.rb had two blocks that could not both claim fast. The custom exception block moved to raise-custom-vs-e2mmap.rb, and the README entry links and shows both.
  • 21 files that already used fast listed a slower report first. Only their report lines were reordered.
  • keys-include-vs-key.rb and values-include-vs-value.rb dropped .shuffle, which has been there since Add Hash#key? vs. Hash#keys.include? and Hash#value? vs. Hash#values.include? #154 with no reason given. It moved the looked-up item on every run, and in 9 of 20 test builds "zz" was a key, not a value. With the item at a fixed spot on ruby_3.4, value? won 1.67x at 10% in, and values.include? won 1.10x at 90% and 1.34x when missing. Both files now look up the entry halfway through.
  • concatenation.rb's comments now give measured object counts per call (1, 1, 2, 2, 3 on Ruby 3.4). The old "3 object" comments were wrong for << and concat.

Step 2: lint that each benchmark names one winner, first

  • A fourth rule in .github/scripts/lint-benchmarks.rb: in every Benchmark.ips block, exactly one report calls fastest (else faster, else fast), and it is the first report.
  • No workflow change: the lint job already runs this script, and benchmarks-ok requires it.
  • CONTRIBUTING explains the naming with length-vs-size-vs-count.rb.

Step 3: name every report by rank and define methods in that order

  • Every report now calls a rank-named method (avg, slow_dup, even_slower, slow2 and slow_1 to slow_5 are gone), in the order CI run 35927028164 measured.
  • Names grow outward from the line between fast and slow without skipping a step: fast, faster, fastest on the recommended side, slow, slower, slowest on the other. They give the order, not the size of the gap, which changes from one Ruby to the next (for example String#match? is 0.96x to 2.07x of =~).
  • Methods are defined in the same order as the reports.
  • proc-call-vs-yield.rb drops the "unused block" report: it never called the block, so it did less work and won on 17 of 26 jobs. Without it, yield wins on 22 of 26.
  • One-line comments where a ranking depends on the Ruby or should tie, for example Range#member? being an alias of Range#include?.
  • CONTRIBUTING's naming text and example follow the same rule.

Step 4: rerun the proc-call-vs-yield.rb README sample

  • The README still showed the dropped "unused block" report ranked first. The new sample is from ruby_4.0 and shows the same order as 22 of 26 CI jobs.

Test plan

  • The lint passes all 68 files locally (Ruby 3.4) and under Docker ruby_4.0, the command CI runs. Step 1 alone also passes the lint on main.
  • 20 test cases (kept locally) give the 11 expected problems, including two reports calling fast and a claimed report that is not first.
  • After Step 3, every changed file ran on ruby_2.1 and ruby_3.4 (crash check, 34 of 34 blocks), and report labels match main except the split raise file and the dropped "unused block".
  • After Step 1, the changed files ran on ruby_2.1 (crash check, all 35 blocks) and ruby_3.4 (real timing). On ruby_3.4 the claimed report won or tied within error in 34 of 35. The exception, start-string-checking-match-vs-start_with.rb (match? 1.09x), happens only on 3.4: start_with? wins on 24 of 26 jobs in run 35927028164, and the file says so.
  • Without the shuffle, on ruby_2.1, 3.4, 4.0, jruby_head and truffleruby_head: key? wins everywhere (575x to 23363x). value? wins on JRuby (1.61x) and TruffleRuby (3.33x) and ties on the three CRubies.
  • This PR changes .github/scripts/, so its own CI runs the full suite on all 26 jobs.

Not covered

  • The lint checks only the top name. A block with one fastest and two reports calling fast still passes, so "name them by rank" is a convention, not a check.
  • README sample outputs still show the old report order. The numbers move to the results site in a later PR.

Each benchmark now says which report should win through its method names:
`fastest`, `faster`, `fast`, then `slow`, `slower`, `slowest`. Every report
is wrapped in one of these methods, so they all pay the same call cost, and
the winner is listed first. Where a ranking could look odd, a one-line
comment says why.

- 11 files that ran code inline or used other names were converted.
- start_with-vs-substring-==.rb reported code strings; each string now
  becomes a method, so its numbers are not comparable with older runs.
- raise-vs-e2mmap.rb had two blocks that could not both claim `fast`; the
  custom exception block moved to raise-custom-vs-e2mmap.rb.
- 21 files that already used `fast` listed a slower report first; only
  their report lines were reordered.
- keys-include and values-include dropped `.shuffle`. It moved the
  looked-up item on every run, so the winner of values-include depended on
  where it landed. Both now look up the entry halfway through.

Rankings follow the results from CI run 35927028164.
A fourth lint rule: in every Benchmark.ips block, exactly one report calls
`fastest` (else `faster`, else `fast`), and it is the first report. The
claim then comes from the code, not from README prose, and the results site
can check it against every run.

The lint job already runs this script and benchmarks-ok requires it, so the
workflow does not change. CONTRIBUTING explains the naming with
length-vs-size-vs-count.rb.
Finish the naming from Step 1 across all files, so each one reads top to
bottom as its advice.

- Every report now calls a rank-named method: avg, slow_dup, even_slower,
  slow2 and slow_1 to slow_5 are gone. Names follow the order in CI run
  35927028164.
- Names grow outward from the line between fast and slow, never skipping
  a step: the recommended side uses fast, faster, fastest, the other side
  slow, slower, slowest. The names give the order, not the size of the gap,
  which changes from one Ruby to the next.
- Methods are defined in the same order as the reports.
- proc-call-vs-yield.rb drops the "unused block" report: it never called
  the block, so it did less work than the others and won on 17 of 26 jobs.
- One-line comments where a ranking depends on the Ruby or should tie
  (Range#member? is an alias of Range#include?).
- inheritance-check.rb explains its two raise lines, and
  inject-symbol-vs-block.rb drops a require "rubygems" only Ruby 1.8
  needed.
- CONTRIBUTING's naming text and example follow the same rule.
Step 3 dropped its "unused block" report, but the README still showed it
ranked first, which would give readers the wrong ranking. The new sample
is from ruby_4.0: yield, then block + yield (1.23x slower), then
block.call (1.41x slower), the same order as 22 of 26 jobs in CI run
35927028164.
@JuanVqz
JuanVqz marked this pull request as ready for review September 26, 2026 04:08
@JuanVqz
JuanVqz merged commit f579b70 into main Sep 26, 2026
29 checks passed
@JuanVqz
JuanVqz deleted the feature/benchmark-claims branch September 26, 2026 04:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant