Repeated opcache_reset() under concurrency permanently grows ZCSG(map_ptr_last), inflating every newly spawned php-fpm worker

[thisjrodriguez]

Description

Description

On a busy php-fpm server, an application that calls opcache_reset() on many requests makes every newly spawned worker start bigger and bigger, without limit, until FPM is restarted.

The memory is a single large zero-filled anonymous mapping allocated by zend_map_ptr_extend() when a worker loads its first script from SHM. It is not visible to userland: on the affected production server PHP reported peak request memory of 2-38 MB while each worker's RSS was ~1.1 GB from birth.

Each opcache_reset() permanently increases ZCSG(map_ptr_last); it never shrinks. Workers born later allocate (and zero) the whole table.

Concurrency is required to reproduce. With resets happening while the pool is idle, growth is negligible (500 resets => +2 MB). With concurrent traffic and workers being spawned/reaped, the table grows on every reset.

I have not pinned down the exact code path for the concurrent case. My reading is that accel_activate() resets ZCSG(map_ptr_last) via zend_reset_cache_vars() only in the process that performs the restart, while workers that were busy keep their old (large) CG(map_ptr_last) and write it back to SHM on the next cache_script_in_shared_memory() (zend_persist.c: ZCSG(map_ptr_last) = CG(map_ptr_last);). That part is a hypothesis; the measurements below are not.

Real-world impact

A PHP application called opcache_reset() on ~4,700 requests/day (~1 every 16 s) on a shared server with ~850 req/min of traffic:

• growth measured in production: ~0.3 MB per reset (app of ~6,000 cached scripts)
• 21 hours after the last FPM restart, every newly spawned worker was ~960 MB at birth
• php-fpm total RSS: 52 GB across ~50 workers; a graceful reload brought it back to 2.4 GB
• the previous day the box (128 GB RAM) hit OOM, the kernel started killing processes and sites went down
• OPcache SHM is shared by all pools of that PHP version, so unrelated accounts on the same server were inflated too

Reproducer

Attached script: starts its own php-fpm (temp dir, own socket), serves a small app (2,500 autoloaded classes with inheritance/interfaces/traits) and sends concurrent FastCGI requests, optionally issuing opcache_reset() concurrently. Python 3.6+, stdlib only.

python3 php_opcache_reset_repro.py <php-fpm> <opcache.so|-> 600 noreset
python3 php_opcache_reset_repro.py <php-fpm> <opcache.so|-> 600 reset

It prints, before and after, the max worker RSS and the max anonymous mapping in a worker (where the map_ptr table lives).

Results (600 cycles, 6 concurrent requests each)

PHP	Build	mode	max worker RSS	max anon mapping
8.3.33	Debian/Sury	noreset	20 -> 22 MB	4 -> 2 MB
8.4.24	Debian 13	noreset	21 -> 22 MB	6 -> 2 MB
8.5.8	static-php-cli	noreset	42 -> 44 MB	19 -> 19 MB
8.1.34	cPanel ea-php	reset (598)	19 -> 87 MB	4 -> 57 MB
8.1.34	CloudLinux alt-php	reset (598)	20 -> 85 MB	4 -> 54 MB
8.3.33	Debian/Sury	reset (592)	20 -> 94 MB	4 -> 65 MB
8.3.33	cPanel ea-php	reset (598)	19 -> 84 MB	6 -> 55 MB
8.4.24	Debian 13	reset (600)	20 -> 97 MB	6 -> 66 MB
8.4.25	cPanel ea-php	reset (600)	20 -> 89 MB	6 -> 58 MB
8.4.25	CloudLinux alt-php	reset (595)	21 -> 87 MB	6 -> 55 MB
8.5.8	static-php-cli	reset (597)	43 -> 111 MB	19 -> 61 MB

Growth is linear and does not level off: 1,800 resets give ~200 MB (8.3.33) and ~206 MB (8.5.8) of anonymous mapping per worker.

Note this is distinct from GH-8646, whose fix (zend_map_ptr_reset() at request end when CG(interned_strings) is non-empty) only applies when OPcache is disabled. Here OPcache is enabled and the growth happens through opcache_reset().

Expected result

Repeated opcache_reset() should not permanently increase per-worker memory. After a reset the map_ptr table should return to a size proportional to the code actually cached.

Actual result

ZCSG(map_ptr_last) grows on every reset and never shrinks, so every worker spawned afterwards allocates and zero-fills an ever larger table, until php-fpm is restarted/reloaded.

PHP Version

Reproduced on 8.1.34, 8.3.33, 8.4.24, 8.4.25 and 8.5.8 (Debian, Sury, static-php-cli, cPanel ea-php, CloudLinux alt-php). The relevant code in ext/opcache/ZendAccelerator.c looks unchanged in master.

Operating System

AlmaLinux 10 (production), Debian 13 (local repro)

[dm-jnpstr]

We hit the same growth in production on PHP 8.3.30 FPM (Debian packages, pm = static, 192 children per node, opcache.revalidate_path = 1). Our deploys called opcache_reset() through cachetool over FastCGI, about 6–7 times a day. I think the code path is the one below. Line numbers are from the current PHP-8.3 branch.

Mechanism

1. Persisting a script takes map_ptr slots from the compiling process's own counter and publishes it: ZCSG(map_ptr_last) = CG(map_ptr_last) (zend_persist.c:1385). We measure about 13 slots per script.
2. On restart, only the process that performs it rewinds, with zend_map_ptr_reset() and zend_reset_cache_vars() (ZendAccelerator.c:2693). Every other live worker takes the "SHM was reinitialized" branch (ZendAccelerator.c:2728). That branch clears only the realpath and PCRE caches, so the worker keeps its old, high CG(map_ptr_last).
3. The first of those workers to persist a script allocates on top of the stale value and republishes it. ZCSG(map_ptr_last) never goes down, and it grows by one generation per reset.
4. Every new worker extends its table to ZCSG(map_ptr_last) on its first cached-script load (zend_accelerator_util_funcs.c:384). zend_activate() then zeroes the whole table on every request (zend.c:1271), so the CPU cost grows along with the memory.

Collected data

• One worker on a busy node held a single anonymous mapping of 112,452 kB. That is 3,514 × 32 KiB + 4 KiB, which is what perealloc in 4,096-slot steps plus the allocator header gives. The prediction from step 4 is 154 resets × 7,606 cached scripts × ~12.9 slots × 8 B ≈ 121 MB. A worker 1 s old, before its first request, had 12 MiB.
• Across 14 test nodes, AnonPages against (resets since FPM start × cached scripts) fits with r = 0.978. Resets alone give r = 0.82.
• CPU per request on the worst node went from 21.3 ms to 26–27 ms over three weeks. It stayed flat on nodes that were reloaded often.
• pm.max_requests does not help, because each new worker re-extends on its first request.

Workaround

A graceful FPM reload (SIGUSR2, with process_control_timeout > 0) creates a new master and a new SHM segment, so no stale worker survives. We stopped calling opcache_reset() under traffic. Deploys now reload FPM only when the cache needs it.

Possible fix (untested)

The rewind at ZendAccelerator.c:2693 might also be enough for the other workers. In the "SHM was reinitialized" branch, a worker is at request activation and holds no slot for a live request. So calling zend_map_ptr_reset() and zend_reset_cache_vars() there as well would let every process restart from the startup baseline. I have not checked whether anything else relies on those slots across a restart.