Skip to content

perf: reserve each function's operand stack on entry - #59

Merged
explodingcamera merged 6 commits into
explodingcamera:nextfrom
rebeckerspecialties:perf/reserve-operand-stack
Sep 24, 2026
Merged

explodingcamera merged 6 commits into
explodingcamera:nextfrom
rebeckerspecialties:perf/reserve-operand-stack

Conversation

@matthargett

@matthargett matthargett commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Third of a three-PR stack on next (785be0e):

  1. perf: grow the value stack out of line #57 perf: grow the value stack out of line
  2. perf: inline the fused binop and compare helpers #58 perf: inline the fused binop and compare helpers
  3. perf: reserve each function's operand stack on entry #59 perf: reserve each function's operand stack on entry (this PR)

This branch includes #58 and #57 below it; the last three commits are new here.

The first PR moved stack growth out of the instruction handlers but kept a cold call to it, and that call still costs the handlers their register saves. This PR removes growth from push entirely:

  • The parser already tracks the operand stack per lane (lane_counts in visit.rs). It now also records the highest point each function body reaches, as WasmFunction::max_stack.
  • enter_locals reserves locals + max_stack in each lane when a function is entered, and grows a dynamic stack there if needed. It already grew for the locals at that point.
  • Stack::push, used by the instruction handlers, then traps at capacity instead of calling a growth path: a full stack inside a body means the limit. Typed guest operand handlers now use reserved capacity; host value marshaling can still grow.
  • Host argument and result marshaling through push_dyn retains push_or_grow, including host calls made by a guest.
  • Catch landing reservations now include tag payload lanes and optional exception references, including function-label catches and cross-frame throws. Self-calls preserve the actual value-stack overflow trap; the WAST exhaustion assertion accepts either internal stack reaching its limit first. GC field reads and exception payload injection use reserved no-growth pushes.

Changes to know about:

  • Public API: tinywasm_types::WasmFunction gains a pub max_stack: ValueCounts field.
  • Archive format: the layout of WasmFunction changed, so TWASM_VERSION goes to 06 (the first format change since 0.11.0 shipped with 05).
  • Stack limits: overflow is detected when a function is entered, not at the push that crosses the limit. A function whose deepest operand stack does not fit traps on entry, even if a particular call would not have reached that depth. Deep recursion runs out slightly earlier, by at most one function's operand stack.
  • tests/value_stack_reservation.rs runs a function with a deep operand stack in all three lanes, recursively, on dynamic stacks that start empty or tiny and on a fixed stack. It also checks that a limit smaller than the body's needs still traps with ValueStackOverflow.

iPhone 12 E-cores, same setup as the first PR, with XCode 27 PMU template for low-level CPU stats. This PR against the second, which ittself is against next, and against an upper bound: the second PR plus a push with no growth path and no reservation, which behaves the same on fixed stacks (the default) but would trap on dynamic ones.

cycles instructions rows faster
this PR vs the second −2.8 % −1.9 % 15/16
the whole stack vs next −8.0 % −7.2 % 16/16
upper bound vs the second −2.7 % −2.4 % 16/16
this PR vs the upper bound −0.1 % +0.5 % 8/16

The reservation keeps the whole upper-bound gain on the geomean. Its extra work on function entry shows up on the most call-heavy rows. The EH parser (exnref) is the one row that is slower than with the first two PRs: +1.3 %, consistently across reps. Against the upper bound, it is +2.3 %, and the tail-call FSM +1.2 %. A few rows come out about 3 % faster than the upper bound (sieve, xmrsplayer), which is most likely code layout rather than this change.

Cycles per call on next, and every build as a ratio to it (median of 5 launches):

row next Mcycles first PR first two whole stack upper bound
fib(30) 303.8 0.964 0.931 0.913 0.905
matmul relaxed-simd FMA 8.17 0.960 0.951 0.889 0.879
audio DSP 3089 0.960 0.926 0.881 0.888
call_indirect 63.36 0.970 0.975 0.952 0.953
xmrsplayer 45.52 0.943 0.946 0.909 0.938
vtable_poly4 128.5 0.984 0.961 0.937 0.934
graphql-validation (AssemblyScript) 36.61 0.975 0.945 0.936 0.926
graphql-validation (Porffor) 31.03 0.970 0.958 0.957 0.952
sieve (scalar) 2.04 0.909 0.908 0.857 0.889
crc32 (scalar) 14.75 0.915 0.904 0.866 0.869
convolution (scalar) 40.93 0.935 0.933 0.882 0.887
bulk_memory (scalar) 30.59 0.936 0.907 0.869 0.877
tail-call FSM 14.59 0.978 0.967 0.953 0.942
EH parser (exnref) 42.01 0.980 0.981 0.994 0.972
GC binary trees 284.4 0.989 0.990 0.984 0.984
call_ref twin (call_indirect) 55.03 0.983 0.970 0.962 0.954
geomean 0.959 0.947 0.920 0.921

Checks on this branch: tinywasm's CI matrix , run on my local M4 MacBook: cargo test --workspace and cargo test --workspace --examples . I went the distance and made sure the whole suite passes with the default value stacks switched to dynamic stacks that start empty and grow to exactly each function's reservation (a local stress configuration, not in this PR), where a function whose recorded maximum undercounted its pushes would trap. Last, I did a coverage-driven fuzzer pass overnight while I slept, and didn't get any anomalous exceptions or state tainting.

The parser records the highest operand stack each function body reaches
in each lane (WasmFunction::max_stack), and enter_locals reserves it on
top of the locals. A push inside a function body then traps at capacity
instead of growing, so the instruction handlers no longer call into the
growth path. Pushes from outside a body (host arguments and results)
still grow dynamic stacks.

WasmFunction gained a field, so the archive version moves to 06.
explodingcamera pushed a commit that referenced this pull request Sep 24, 2026
First of a three-PR stack on `next` (`785be0e`), each on top of the
previous one with individual wins that combine nicely:

1. #57 **perf: grow the value stack out of line** (this PR)
2. #58 perf: inline the fused binop and compare helpers
3. #59 perf: reserve each function's operand stack on entry

`Stack::push` now checks `len == capacity` itself and hands the full
case to a `#[cold] #[inline(never)] push_grow`, which keeps the existing
overflow check and growth. After that check `Vec::push` can no longer
reach its own growth path, so the allocator call leaves the instruction
handlers. It was inlined into every handler that pushes, and the call
made those handlers save and restore four pairs of callee-saved
registers on every execution, even though the default config's stacks
are fixed and never grow. `LocalGet32`'s hot path goes from 57 to 37
instructions on aarch64.

No behavior change: fixed stacks still trap at capacity, and dynamic
stacks still grow up to `max_size`.

**iPhone 12 efficiency cores** (A14 Icestorm, `.utility` QoS). XCode 27
PMU template for low-level CPU stats. 16 benchmark rows, 5 launches per
build, interleaved with the other builds of the stack; cycles and
instructions per call, geomean against `next` as of `b45a98a`:

| | cycles | instructions | rows faster |
|---|---:|---:|---:|
| this PR vs `next` | **−4.1 %** | −3.5 % | 16/16 |

**Checks** on this branch: tinywasm's CI matrix run on my local M4 Mac:
`cargo test --workspace` and `cargo test --workspace --examples` with
`--features tinywasm/nightly-tail-calls`, each with and without default
features
explodingcamera pushed a commit that referenced this pull request Sep 24, 2026
Second of a three-PR stack on `next` (`785be0e`):

1. #57 perf: grow the value stack out of line
2. #58 **perf: inline the fused binop and compare helpers** (this PR)
3. #59 perf: reserve each function's operand stack on entry

This branch includes #57 below it; the last commit is new here.

`#[inline(always)]` on `exec_binop_32/64` and `exec_cmp_32/64`. LLVM
kept them out of line, so every fused `BinOp*` / compare handler called
out for the operator `match` and spilled registers around the call.
Inlined, the `match` runs inside the handler. The binary grows by about
16 KB, and I didn't see a notable rise in L1 cache misses on Apple A14
efficiency cores in my benchmarks.

No behavior change.

**iPhone 12 E-cores**, same setup as the first PR; this PR against the
first:

| | cycles | instructions | rows faster |
|---|---:|---:|---:|
| this PR vs the first | **−1.3 %** | −2.0 % | 12/16 |
| first two PRs vs `next` | −5.3 % | −5.5 % | 16/16 |

The four rows that didn't get faster moved by 0.5 % or less.

**Checks** on this branch: tinywasm's CI matrix run on my local M4 Mac:
`cargo test --workspace` and `cargo test --workspace --examples` on rust
1.98 and with `--features tinywasm/nightly-tail-calls`, each with and
without default features.
@explodingcamera

Copy link
Copy Markdown
Owner

thanks for the prs! I also measured some nice performance gains on x86 with these changes.

@explodingcamera
explodingcamera merged commit c0be647 into explodingcamera:next Sep 24, 2026
9 checks passed
explodingcamera pushed a commit that referenced this pull request Sep 24, 2026
First of a three-PR stack on `next` (`785be0e`), each on top of the
previous one with individual wins that combine nicely:

1. #57 **perf: grow the value stack out of line** (this PR)
2. #58 perf: inline the fused binop and compare helpers
3. #59 perf: reserve each function's operand stack on entry

`Stack::push` now checks `len == capacity` itself and hands the full
case to a `#[cold] #[inline(never)] push_grow`, which keeps the existing
overflow check and growth. After that check `Vec::push` can no longer
reach its own growth path, so the allocator call leaves the instruction
handlers. It was inlined into every handler that pushes, and the call
made those handlers save and restore four pairs of callee-saved
registers on every execution, even though the default config's stacks
are fixed and never grow. `LocalGet32`'s hot path goes from 57 to 37
instructions on aarch64.

No behavior change: fixed stacks still trap at capacity, and dynamic
stacks still grow up to `max_size`.

**iPhone 12 efficiency cores** (A14 Icestorm, `.utility` QoS). XCode 27
PMU template for low-level CPU stats. 16 benchmark rows, 5 launches per
build, interleaved with the other builds of the stack; cycles and
instructions per call, geomean against `next` as of `b45a98a`:

| | cycles | instructions | rows faster |
|---|---:|---:|---:|
| this PR vs `next` | **−4.1 %** | −3.5 % | 16/16 |

**Checks** on this branch: tinywasm's CI matrix run on my local M4 Mac:
`cargo test --workspace` and `cargo test --workspace --examples` with
`--features tinywasm/nightly-tail-calls`, each with and without default
features
explodingcamera pushed a commit that referenced this pull request Sep 24, 2026
Second of a three-PR stack on `next` (`785be0e`):

1. #57 perf: grow the value stack out of line
2. #58 **perf: inline the fused binop and compare helpers** (this PR)
3. #59 perf: reserve each function's operand stack on entry

This branch includes #57 below it; the last commit is new here.

`#[inline(always)]` on `exec_binop_32/64` and `exec_cmp_32/64`. LLVM
kept them out of line, so every fused `BinOp*` / compare handler called
out for the operator `match` and spilled registers around the call.
Inlined, the `match` runs inside the handler. The binary grows by about
16 KB, and I didn't see a notable rise in L1 cache misses on Apple A14
efficiency cores in my benchmarks.

No behavior change.

**iPhone 12 E-cores**, same setup as the first PR; this PR against the
first:

| | cycles | instructions | rows faster |
|---|---:|---:|---:|
| this PR vs the first | **−1.3 %** | −2.0 % | 12/16 |
| first two PRs vs `next` | −5.3 % | −5.5 % | 16/16 |

The four rows that didn't get faster moved by 0.5 % or less.

**Checks** on this branch: tinywasm's CI matrix run on my local M4 Mac:
`cargo test --workspace` and `cargo test --workspace --examples` on rust
1.98 and with `--features tinywasm/nightly-tail-calls`, each with and
without default features.
explodingcamera added a commit that referenced this pull request Sep 24, 2026
Third of a three-PR stack on `next` (`785be0e`):

1. #57 perf: grow the value stack out of line
2. #58 perf: inline the fused binop and compare helpers
3. #59 **perf: reserve each function's operand stack on entry** (this
PR)

This branch includes #58 and #57 below it; the last three commits are
new here.

The first PR moved stack growth out of the instruction handlers but kept
a cold call to it, and that call still costs the handlers their register
saves. This PR removes growth from `push` entirely:

- The parser already tracks the operand stack per lane (`lane_counts` in
`visit.rs`). It now also records the highest point each function body
reaches, as `WasmFunction::max_stack`.
- `enter_locals` reserves `locals + max_stack` in each lane when a
function is entered, and grows a dynamic stack there if needed. It
already grew for the locals at that point.
- `Stack::push`, used by the instruction handlers, then traps at
capacity instead of calling a growth path: a full stack inside a body
means the limit. Typed guest operand handlers now use reserved capacity;
host value marshaling can still grow.
- Host argument and result marshaling through `push_dyn` retains
`push_or_grow`, including host calls made by a guest.
- Catch landing reservations now include tag payload lanes and optional
exception references, including function-label catches and cross-frame
throws. Self-calls preserve the actual value-stack overflow trap; the
WAST exhaustion assertion accepts either internal stack reaching its
limit first. GC field reads and exception payload injection use reserved
no-growth pushes.

Changes to know about:

- **Public API**: `tinywasm_types::WasmFunction` gains a `pub max_stack:
ValueCounts` field.
- **Archive format**: the layout of `WasmFunction` changed, so
`TWASM_VERSION` goes to `06` (the first format change since 0.11.0
shipped with `05`).
- **Stack limits**: overflow is detected when a function is entered, not
at the push that crosses the limit. A function whose deepest operand
stack does not fit traps on entry, even if a particular call would not
have reached that depth. Deep recursion runs out slightly earlier, by at
most one function's operand stack.
- `tests/value_stack_reservation.rs` runs a function with a deep operand
stack in all three lanes, recursively, on dynamic stacks that start
empty or tiny and on a fixed stack. It also checks that a limit smaller
than the body's needs still traps with `ValueStackOverflow`.

**iPhone 12 E-cores**, same setup as the first PR, with XCode 27 PMU
template for low-level CPU stats. This PR against the second, which
ittself is against `next`, and against an upper bound: the second PR
plus a `push` with no growth path and no reservation, which behaves the
same on fixed stacks (the default) but would trap on dynamic ones.

| | cycles | instructions | rows faster |
|---|---:|---:|---:|
| this PR vs the second | **−2.8 %** | −1.9 % | 15/16 |
| the whole stack vs `next` | **−8.0 %** | −7.2 % | 16/16 |
| upper bound vs the second | −2.7 % | −2.4 % | 16/16 |
| this PR vs the upper bound | −0.1 % | +0.5 % | 8/16 |

The reservation keeps the whole upper-bound gain on the geomean. Its
extra work on function entry shows up on the most call-heavy rows. The
EH parser (exnref) is the one row that is slower than with the first two
PRs: +1.3 %, consistently across reps. Against the upper bound, it is
+2.3 %, and the tail-call FSM +1.2 %. A few rows come out about 3 %
faster than the upper bound (sieve, xmrsplayer), which is most likely
code layout rather than this change.

Cycles per call on `next`, and every build as a ratio to it (median of 5
launches):

| row | `next` Mcycles | first PR | first two | whole stack | upper
bound |
|---|---:|---:|---:|---:|---:|
| fib(30) | 303.8 | 0.964 | 0.931 | 0.913 | 0.905 |
| matmul relaxed-simd FMA | 8.17 | 0.960 | 0.951 | 0.889 | 0.879 |
| audio DSP | 3089 | 0.960 | 0.926 | 0.881 | 0.888 |
| call_indirect | 63.36 | 0.970 | 0.975 | 0.952 | 0.953 |
| xmrsplayer | 45.52 | 0.943 | 0.946 | 0.909 | 0.938 |
| vtable_poly4 | 128.5 | 0.984 | 0.961 | 0.937 | 0.934 |
| graphql-validation (AssemblyScript) | 36.61 | 0.975 | 0.945 | 0.936 |
0.926 |
| graphql-validation (Porffor) | 31.03 | 0.970 | 0.958 | 0.957 | 0.952 |
| sieve (scalar) | 2.04 | 0.909 | 0.908 | 0.857 | 0.889 |
| crc32 (scalar) | 14.75 | 0.915 | 0.904 | 0.866 | 0.869 |
| convolution (scalar) | 40.93 | 0.935 | 0.933 | 0.882 | 0.887 |
| bulk_memory (scalar) | 30.59 | 0.936 | 0.907 | 0.869 | 0.877 |
| tail-call FSM | 14.59 | 0.978 | 0.967 | 0.953 | 0.942 |
| EH parser (exnref) | 42.01 | 0.980 | 0.981 | 0.994 | 0.972 |
| GC binary trees | 284.4 | 0.989 | 0.990 | 0.984 | 0.984 |
| call_ref twin (call_indirect) | 55.03 | 0.983 | 0.970 | 0.962 | 0.954
|
| **geomean** | | **0.959** | **0.947** | **0.920** | **0.921** |

**Checks** on this branch: tinywasm's CI matrix , run on my local M4
MacBook: `cargo test --workspace` and `cargo test --workspace
--examples` . I went the distance and made sure the whole suite passes
with the default value stacks switched to dynamic stacks that start
empty and grow to exactly each function's reservation (a local stress
configuration, not in this PR), where a function whose recorded maximum
undercounted its pushes would trap. Last, I did a coverage-driven fuzzer
pass overnight while I slept, and didn't get any anomalous exceptions or
state tainting.

---------

Co-authored-by: Henry Gressmann <mail@henrygressmann.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants