gh-157742: Speed up int multiplication by not zeroing the whole result in x_mul() - #157743
eendebakpt wants to merge 4 commits into
Conversation
x_mul() memset()s the entire product to zero before the gradeschool loop runs, but the i == 0 pass stores z[0:size_b+1] outright rather than accumulating into it, and every later pass only reads digits that the previous pass has already written. Only the top size_a digits -- the carry positions -- actually have to start out zeroed. k_mul() arranges for size_a <= size_b before calling x_mul(), so this replaces a memset of size_a + size_b digits with one of size_a digits. The win is largest exactly where the old code wasted the most: for a single-digit multiplier the zeroing pass was as long as the entire multiplication. Peeling the i == 0 pass out of the loop also drops the redundant load of *pz from the carry dependency chain. The a == b squaring path is unchanged and keeps the full memset. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each schoolbook pass stores its final carry into z[i+size_b], a digit no earlier pass has touched, so it can be a plain store rather than an addition onto a pre-zeroed digit. With that, no digit of the result needs zeroing at all. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
The row 0 loop no longer has a PGO+LTO build, a separate build for every variant, min of 4 interleaved rounds:
The gain is smaller than in the PR description, which looks measured without PGO. Larger sizes are dominated by the code layout and tell us nothing. Between With the signal check restored the gain is about 1.10x, and only for |
|
@serhiy-storchaka Agreed the code layout can interfere with the benchmarks. The avoided sigcheck is indeed part of the gain. In #157744 I address the sigcheck, If that is merged I will revisit this PR (maybe indeed the gains turn out to be too small) |
x_mul()(schoolbook multiplication inObjects/longobject.c) starts bymemset-ing all digits of the result to zero, then accumulates row by row withcarry += *pz + *pb++ * f.The zeroing is not needed for the non-squaring path. Row 0 can store its
size_b + 1digits outright, and rowionly reads digits that rowi - 1already wrote; each carry slotz[i + size_b]is written exactly once before it is ever read. So no digit needs to start out zero, and the per-rowif (carry) *pz += carrybecomes a plain store.This matters most for the common big × small case (
size_a == 1). Karatsuba and lopsided multiplication callx_mul()for their base cases, so they gain a little too.The
mul 1x2coverts the important case of 64-bit ints.Benchmarks
pidigitsBenchmark script