[rush] rushd: fix the bugs found by dogfooding the Rush daemon (stacked on #6046) - #6102
Merged
Sean Larkin (TheLarkInn) merged 266 commits intoSep 30, 2026
Conversation
…ed under common/config (and 3 more) Swarm integration step 1; original commit 433c87f24f (merge of swarm/r05 at 190d77ea17). Commits folded into this step (4): - 40b5b24d72 [rush-lib] Workspace fingerprint: ignore build state of projects nested under common/config - 1880e7b2b7 [rush-daemon] Warm set: keep resource-free retained results under memory pressure and idle expiry - 07e908d32d [rush-daemon] Admission: don't spend a default wait budget behind a graph load or across an admitted request's execution - 190d77ea17 [rush-lib] Move the nested common/config fingerprint test to the end of its suite Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…ce per project (and 1 more) Swarm integration step 2; original commit 8553b2be55 (merge of swarm/r07 at d94054e04d). Commits folded into this step (2): - 57dd6a4b45 [rush-daemon] Warm set: read status once per maintenance pass, not once per project - d94054e04d [rush-lib] Share rig profile loads across projects in the uncached rush-project.json load Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…nments (and 3 more) Swarm integration step 3; original commit 3b2d24277d (merge of swarm/r06 at 6c10e241a4). Commits folded into this step (4): - 803345396b [rush-lib] Keep RUSH_DAEMON* variables out of lifecycle script environments - b49d8b10cb [rush-daemon] Keep one daemon across per-session environment differences - 36892176cb [rush-daemon-transport] Reap detached operation groups of a daemon that died uncleanly - 6c10e241a4 [rush-daemon] Run each operation with its requester's per-session environment Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…tible plugins (and 1 more) Swarm integration step 4; original commit 1e786a4252 (merge of swarm/r01 at 162ff05eaf). Commits folded into this step (2): - 5fe505bf1a [rush-lib] Let daemon engines serve builds with declared daemon-compatible plugins - 162ff05eaf [rush-lib] Walk each plugin package once in the daemon runtime fingerprint Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…ing client needs it Swarm integration step 5; original commit 02f83a99ae (merge of swarm/r05 at bd7ac88caa). Refs: board 207. Commits folded into this step (1): - bd7ac88caa [rush-daemon] Abort a shared iteration's leftover work once no remaining client needs it (board 207) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
Swarm integration step 6; original commit 519ae33c8d. Refs: task 50, board 115. ShellOperationRunner runs a phase's `:incremental` script whenever the operation has a last state. Native `rush build` executes one iteration, so only watch mode ever had one, and watch mode turns cache writes off. rushd keeps one operation graph across requests, so from its second request on a daemon `build` ran `_phase:build:incremental` (in odsp-web, `heft run --only build --` without --clean) with cache writes on. Outputs of a deleted or renamed source file stayed in lib/ and were cached under the key native Rush computes for the clean build, so later native builds restored them (t04 board 115, board 450). ShellOperationRunnerPlugin now sets the incremental command only in watch mode. Every other command runs the initial script, exactly as a single `rush build` does, which is option (a) of task 50. Native `rush build` is unchanged (it never has a last state) and watch mode keeps `:incremental` with writes off. Tests: a rush-lib unit test runs a repeated operation with and without watch mode, and a daemon integration test runs three warm builds of a project with an `:incremental` script. Before the fix the second daemon build logged "Invoking (incremental): node build.cjs --incremental" followed by "Successfully set cache entry". (cherry picked from commit e382d2ac80f881451b7d527ea9a97e3037880923) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
Swarm integration step 7; original commit f7ac88dc9d (merge of swarm/r07 at 55a16f14ef). Refs: F1 coalescer, board 555. GATE OK ch01 board 930 (tree 41b2227). CONFIRMED by ch03 board 681, t06 board 701/board 749, t01 board 861. Only 55a16f14ef; b6692d09a5 (task 74) is not gated. Commits folded into this step (1): - 55a16f14ef [rush-daemon] Share input captures between concurrent requests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
… snapshot is taken Swarm integration step 8; original commit bc03023091 (merge of swarm/r06 at 29f86b181c). Refs: task 77. ch01 GATE OK board 1019 (tree 80c9659); claim board 698, CONFIRMED by o04 board 738, o03 board 742, o05 board 748. Commits folded into this step (2): - a644819daf [rush-lib] Don't write a build cache entry for inputs saved while the snapshot was taken - 29f86b181c [rush-lib] Make the parameter identity test independent of the host's RUSH_PARALLELISM Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
… join that batch Swarm integration step 9; original commit d0ed461613 (merge of swarm/r05 at 199c69347c). Refs: board 371, task 72. ch01 GATE OK board 1152 (gated tree 5c2b6b6). Commits folded into this step (1): - 199c69347c [rush-daemon] Don't let a request received during a batch's reconcile join that batch (board 371) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…ait timeout Swarm integration step 10; original commit 908dcc1f27 (merge of swarm/r05 at 868e4b9299). Refs: task 7 part 3, board 891. ch01 GATE OK board 1196 (gated tree 55698be). Commits folded into this step (1): - 868e4b9299 [rush-daemon] Admission: another request's graph load doesn't spend any finite wait timeout Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…rify untrusted ones Swarm integration step 11; original commit 4456f17460 (merge of swarm/r07 at 53654af37c). Refs: board 401, task 74, board 982, board 1024. ch01 GATE OK board 1197 (gated tree 3014a03). Commits folded into this step (3): - b6692d09a5 [rush-lib] Keep cache trust for unselected retained results and re-verify untrusted ones - 492126a051 [rush-lib] Re-verify retained results built against unverified dependency outputs without the build cache - 53654af37c [rush-daemon] Test that input captures are shared only between equal fingerprint environments Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…ep the daemon alive when git hash-object fails Swarm integration step 12; original commit cc5c6aab7e (merge of swarm/r01 at 575cbf5698). Refs: board 601, task 80, board 899. ch01 GATE OK board 1272 and board 1273 (gated tree b691997). Commits folded into this step (3): - c11ee10320 [rush-cli-client] Show engine and plugin warnings in agent output - c77aa1dbef [package-deps-hash] Handle a git hash-object failure while ls-files and status still run - 575cbf5698 [package-deps-hash] Name the git command that failed in the error message Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…and own-start memo Swarm integration step 13; original commit 0347cd0dbb. Refs: tasks 67, D and 62. ch01 GATE OK board 1389; o05 board 1360 (task 62 on ch01-t62: 32/32 served in 3 of 3 bursts); t01 board 1106/board 1186. Commits folded into this step (6): - 3dfb75ec86 [node-core-library] Don't grant a LockFile twice when lockfile birthtimes tie - 074f207217 [node-core-library] Keep the LockFile of a running process with another time zone or locale - 4e43a1ef8f [node-core-library] Don't set dirtyWhenAcquired when the holder releases the lock during a check - ad9c1211b4 [node-core-library] Read the start times of other LockFile processes from /proc on Linux - 698af4ef92 [node-core-library] Run ps for the start time of the current LockFile process only once on Linux - 9249e85d63 [rush-client-core] Test clients that start the daemon at the same time Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
… it was received Swarm integration step 14; original commit f2ec9ffc26. ch01 GATE OK board 1430 (tree ae49a66 with o07's stack); t06 CONFIRMED board 1365; o05 board 1412 (cold split gone in auto-start mode). Commits folded into this step (1): - a4de3fe5fe [rush-daemon] Share running input captures that started after a request was received Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…ation cannot load Swarm integration step 15; original commit 69f30e84ab (merge of swarm/r01-t70 at c1b5c95adb). Scope: task 70. ch01 GATE OK board 1522 (tree 3a564b4; WorkspaceRequestLifecycle.ts resolved with r07's board 1231 recipe: the Reuse capture stays tolerant and keeps task 94's receipt); o01 board 1291; r01 board 1250. Commits folded into this step (5): - 08aed7efd2 [package-deps-hash] Handle a git hash-object failure while ls-files and status still run - 3d786c1f78 [package-deps-hash] Name the git command that failed in the error message - a11aa6b774 [rush-lib] Name the project whose configuration an engine cannot load - 24cf595d6a [rush-lib] Report a missing project dependency file to an engine's host - c1b5c95adb [rush-daemon] Hand requests to in-process Rush when a project's configuration cannot load Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…g the run is not trusted as up to date Swarm integration step 16; original commit d6ac7c43f3 (merge of swarm/r06-t92 at 72c2f0e4f8). Scope: task 92. ch01 GATE OK board 1563 (tree 5c0b0ac); r06 board 1270; CONFIRMED by t03, t04, ch05 and o03. Commits folded into this step (1): - 72c2f0e4f8 [rush-lib] Re-run operations whose inputs changed during the iteration, also with cache writes off Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
Swarm integration step 17; original commit 9a58fc2f85 (merge of swarm/r01-t103 at 93eacefb71). Scope: task 103. ch01 GATE OK board 1565 (tree e234c8d); o04 CONFIRMED board 1477; r01 board 1440. This is upstream row 16's commit. Commits folded into this step (1): - 93eacefb71 [rush-lib] Release the package manager lock when installing the package manager fails Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…ost-connection message and a private runtime folder Swarm integration step 18; original commit 261e4320dd (merge of swarm/r03-t87 at d9366621c8). Scope: tasks 4+5, 68, 86 and 87. ch01 GATE OK board 1675 (tree f85ce62). Commits: bb1d272738 (tasks 4+5), 959cbb84ee (task 68), a68e4d96a5, 696dba1d3e (task 86) and d9366621c8 (task 87). Commits folded into this step (5): - bb1d272738 [rush-lib] Hand off a by-name resolvable rush-lib and load built-in cache plugins from deploy outputs - 959cbb84ee [rush-client-core] Resolve a retained startup reservation next to a ready daemon - a68e4d96a5 [rush-client-core] Resolve a remaining startup reservation before a plain daemon stop - 696dba1d3e [rush-client-core] Explain a daemon that exited before delivering a result - d9366621c8 [rush-daemon] Meet one daemon per checkout whatever TMPDIR or XDG_RUNTIME_DIR says (task 87) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
… instead of running Rush in-process Swarm integration step 19; original commit a81a00520d (merge of o07/startup-fallback at 668753ad4d). Scope: task 95. Brings 647a8bf902 [rush-client-core] and 668753ad4d [rush-cli-client]. Gate: ch01 GATE OK board 1731 (tree e4a0062); burst: t01 board 1716. Commits folded into this step (2): - 647a8bf902 [rush-client-core] Wait for a daemon that a live process is still starting before permitting in-process Rush - 668753ad4d [rush-cli-client] Don't run Rush in-process while daemon startup is still pending Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…hile their files are unchanged Swarm integration step 20; original commit 0f987a5b62 (merge of swarm/r07-t93-on-s10 at 18900fe380). Scope: task 93. Brings 143f21a891 and 18900fe380 [rush-lib]. Gate: ch01 GATE OK board 1869 (tree 8892a71); m01 CONFIRMED board 1738 and board 1818; t04 board 1794. Commits folded into this step (2): - 143f21a891 [rush-lib] Reuse workspace input digests and project configurations while their files are unchanged - 18900fe380 [rush-lib] Test that a project configuration that failed to load is loaded afresh once it is fixed Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
… full in agent output, and rushd's startup wait as a pipe line Swarm integration step 21; original commit 478f4d3af0 (merge of swarm/r04-t83-a81 at 4d1a045fca). Scope: task 83. Brings task 83's chain as resolved onto a81a00520d: 244ed8b1f5 (merge of swarm/r04-t83-clip @9d71dbe9bb, the clip fix for board 1539) and 4d1a045fca, with board 711 (1886d80936), tasks 79, 71 and 50 and the restart-drain timeout. Gate: ch01 GATE OK board 1872 (lifts board 1778); combined with task 93, tree d0e8df0, full suite board 1888. Commits folded into this step (21): - 2e7d278d8f [rush-daemon] Report the full log path of failed and warning operations - 3b91eec856 [rush-cli-client] Keep spawned test clients on the default output when tests run in an agent shell - 1ae03fc555 [rush-cli-client] Agent output that holds at repository scale - 862ed5e676 [rush-cli-client] Do not report a daemon shutdown as a user cancellation - 39cae5f22b [rush-cli-client] Send no terminal width for a pseudo-terminal without a size - 5f3c36fe17 [rush-daemon] Describe a rejected request without the debug messages of the graph load (task 71) - 563e541fdf [rush-cli-client] Cap a multi-line error message in agent output (task 71) - 2377740ab9 [rush-daemon] Keep the loaded graph when a request only selects an unknown project (task 71) - e382d2ac80 [rush-lib] Run the initial script for every command outside watch mode (task 50, board 115) - b7bba00fb3 [rush-cli-client] Never leave agent output silent on a pipe (task 79) - feacc54bf7 [rush-daemon] Report a restart drain as a queue position (task 79) - 1886d80936 [rush-cli-client] Report Ctrl+C during graph preparation as cancelled (board 711) - ae31023aad [rush-cli-client] Report an empty selection in agent output (board 634) - bacfa3d5af [rush-daemon] Do not apply the client-default queue timeout to a restart drain (task 83) - 5f1c7b4f1b [rush-cli-client] Keep "daemon admission failed (<code>)" in the agent summary; word an empty selection like native Rush (board 785, board 634) - 3b5dc578d6 [rush-daemon] Document the restart drain and its client-default timeout (task 83) - cbf8a3a45e [rush-daemon] Apply the client-default timeout to a restart drain while a rushx script runs or for later requests (task 83) - 6cd768c6b3 [rush-daemon] Suggest stopping the rushx script when a restart drain times out behind one (task 83) - b2f9d78f1f [rush-daemon] Name the waived time when a restart drain times out (task 83) - 9d71dbe9bb Write the daemon's reason for an admission failure in full in agent output (task 83, board 1539) - 4d1a045fca Write rushd's startup wait as a pipe line in agent output (task 83 on task 95) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
Swarm integration step 22; original commit 6bc6dd8f83 (merge of swarm/r04-t108-a81 at 14faa4e5be). Scope: task 108. Brings task 108 items 1 and 3 onto task 83: agent output prints a failed operation's first diagnostics as soon as that operation finishes, no `running` line before 10 s for quick requests, and each TypeScript error once (t07 board 1930: L4 17 of 41 on 4d1a045fca against 24 of 41 on s11; 38 of 41 with items 1 and 3 on their old base, board 1692). Gate: ch01 GATE OK board 1980 (tree 9079647) Commits folded into this step (1): - 14faa4e5be Report failed operations as they fail in agent output (task 108 items 1 and 3) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…operation graphs Swarm integration step 23; original commit e16ed01d3a (merge of swarm/r01-t85-a81 at 4066e8f5d7). Scope: task 85. Brings task 85 onto task 108 items 1+3: the daemon keeps a result that a plugin such as fstrace found up to date across requests, and checks a Skipped result again when it was marked unverifiable. m01 CONFIRMED the gated tip (board 2024: 72 of 72 hot requests "1066 no op") and measured 24 of 24 hot B6 requests as "no operations needed" (board 2073); ch04 CONFIRMED 6e39ea6527 (board 2003) and posted the product-default A/B (board 2132: -0.24 s [-0.57, +0.09]). ch01's combined suite on this tree passes (board 2150: rush-lib 1169/1, a timing flake in InstallRunScripts.test that task 85 doesn't touch, 5 of 5 on rerun; rush-daemon 515/0, rush-cli-client 367/0). Gate: ch01 GATE OK board 1955, combined suite board 2150 (tree 1c6a3bf) Commits folded into this step (2): - 95c8ece420 [rush-lib] Retain results that a plugin found up to date in long-lived operation graphs - 4066e8f5d7 [rush-lib] Check a Skipped result again if it was marked unverifiable Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
… hooks (and 7 more) Swarm integration step 24; original commit 31a8db3e65 (merge of swarm/r01-t85-x106b at 4835a8700e). Scope: tasks 85 and 106. Brings task 106 (guarded :incremental builds in the daemon) and task 84 at de9a503a50 through r01's 85x106 resolution. This is step 1 of the path in board 2046 and board 2161; step 2 brings task 106's test-only tip b767c64dac. Merging b767c64dac straight onto task 85 conflicts in CacheableOperationPluginRetainedResults.test.ts. Gate: ch01 GATE OK board 2161 (tree 8250469) Commits folded into this step (8): - c7400b3f9c [rush-lib] Pass getOperationEnvironment to the before/after iteration hooks - e07159a38f [rush-lib] Let the daemon run :incremental scripts when a guard allows it - 90a76234e5 [rush-lib] Register RUSH_DAEMON_INCREMENTAL_BUILDS as a known environment variable - b1647a952a [rush-lib] Test that --only consumers of a retained incremental result skip cache writes - aed31544c2 [rush-lib] Test that a terminated command drops its incremental base - 69e33c03a8 [rush-lib] Format the incremental build changes with Prettier - c3afda3587 [rush-lib] Test that a run whose inputs changed while it ran leaves no incremental base - de9a503a50 [rush-lib] Check dependencies reached through same-project phases in the incremental guard Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…s it Swarm integration step 25; original commit 7d0ffb955b (merge of swarm/r06-t106 at b767c64dac). Scope: task 106. Brings task 106's test-only tip onto step 1 (31a8db3e65). With task 106 the daemon runs an operation's <phase>:incremental script, on top of the outputs of its last successful run in the daemon, when only files that the operation builds were edited since then and its output folders are unchanged; anything else runs the initial command. A result built on an incremental base is never written to the build cache. Kill switch: daemon.incrementalBuilds false in rush.json, or RUSH_DAEMON_INCREMENTAL_BUILDS=0. Task 84 passes getOperationEnvironment to the before/after iteration hooks. Second agents: t03 CONFIRMED (board 2067, 37 of 39; TAMPERI is the declared limitation) and t04 CONFIRMED (board 2100). ch01 on this tree (board 2161): rush-lib 1228/0, rush-daemon 516/0, rush-cli-client 367/0; 50 mutants, 42 killed; 33 of 33 end-to-end rows against a native cache-off oracle. Known costs, per ch01's rulings in board 2161: revisiting a state restores less from the cache (Rule 2), and an in-place rewrite of an output file by another writer is not detected (TAMPERI; task 139, r06's 641c39f120 in board 2171). Gate: ch01 GATE OK board 2161 (tree b1c0c2e) Commits folded into this step (1): - b767c64dac [rush-lib] Test that an incremental result taints cache writes through an operation without a script Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
….json may still be changing Swarm integration step 26; original commit 8a1d1ee8dd (merge of swarm/r07-t93-nit at f5826544d1). Scope: task 93 NIT. Test-only: 7 lines in RushProjectConfiguration.test.ts, no source change. It kills ch01's task 93 mutant M13 (board 1869): with the mutant the new assertion fails, 25/1, and 18900fe380's test file lets it survive, 26/0. Full rush-lib on the cherry-pick onto b1c0c2e: 1228/0 (board 2178). The deployed daemon files don't change, so snapshot s14 (7d0ffb955b) is unaffected. Gate: ch01 GATE OK board 2178 (tree 09d9052) Commits folded into this step (1): - f5826544d1 [rush-lib] Test that a configuration is read again while only its rig.json may still be changing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…tput file
Swarm integration step 27; original commit 6eae8fadad (merge of swarm/r06-t106 at 641c39f120).
Scope: task 139.
Brings r06's fold-in on task 106 (board 2171), in place of b767c64dac. The output signature of an operation now records each output file's inode, size and mtime as well as the folder listing, so when another writer appends to or rewrites an output file in place, the next build runs the initial command ("its output folders changed since its last successful run") instead of an incremental run on the stale file. This closes t03's TAMPERI. A touch that changes only mtime now also costs one initial run. r06 measured about 4 µs per output file per read.
ch01 on tree 1eb87abdd2 (board 2211): rush-lib 1231/0, rush-daemon 516/0, rush-cli-client 367/0; b767c64dac's manifest fails 4 of the new tests; 10 mutants, 6 killed (N1, N3 and N6 survive; task 138 adds their tests); tamper end-to-end rows 9/9 against 2/9 before, full rows 33/33, cache-off rows 7/7. On this branch the tree is 1eb87abdd2 plus r07's test-only 8a1d1ee8dd change, byte for byte.
Gate: ch01 GATE OK board 2211 (tree 1eb87abdd2)
Commits folded into this step (1):
- 641c39f120 [rush-lib] Detect in-place rewrites of output files in the incremental guard
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…ntent-hashed outputs, invalidation and skip record Swarm integration step 28; original commit 0d30a5164c (merge of swarm/r06-t138 at dbd5d99147). Scope: task 138 and t106d. Brings r06's task 138 tests (board 2302) and t106d (4c9f23e650, board 2233). Only 3 test files change (+204/-18): ProductionDaemonRequestResolver.test.ts, IncrementalExecutionGuardPlugin.test.ts and OperationOutputManifest.test.ts. No product code changes, so no snapshot is needed. ch01 on tree 45c2c5e (board 2364): rush-lib 1241/0 (1 skipped); rush-daemon 516/0 with t106d. With the new tests, each of the 10 guard and manifest mutants (G9-G12, G19, G24, L1, N1, N3, N6) fails its intended test; all 10 pass the old tests. t106d's kill-switch test fails against a product mutant that drops the daemon.incrementalBuilds check at PhasedScriptAction.ts:666, and 6eae8fadad's version of it passed that mutant. Gate: ch01 GATE OK board 2364 (tree 45c2c5e) Commits folded into this step (2): - 4c9f23e650 [rush-daemon] Make the test of daemon.incrementalBuilds: false edit a source file - dbd5d99147 [rush-lib] Test the incremental guard's configuration inputs, content-hashed outputs, invalidation and skip record Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…it starts, so reloads don't wait for it (board 113) (and 5 more) Swarm integration step 29; original commit 3857bc9926 (merge of swarm/r05-stack-int at c35a77cc5c). Brings r05's stack (board 1931): - Task 55: a served rushx script releases the lifecycle gate once it starts, so reloads don't wait for it. - Task 7 part 3 text: admission timeouts name the client's wait timeout and the time that isn't counted. - Task 96: a rushx script that arrives while a restart is pending waits for the restart instead of postponing it, and admission failures in legacy output and rushx-client print the daemon's reason. - Task 98: a request spends its wait timeout only while it waits for other requests. - The race fix (c35a77cc5c, restacked from 9370662d5f): a client that follows a daemon restart connects to the successor that the restarting daemon launches instead of racing it. ch01 on tree b500e451d4 (board 2429): build rc 0 with 0 warnings; rush-daemon 544/0, rush-lib 1231/0, rush-cli-client 374/0, rush-client-core 129/0 on its own (2 known flakes in the full run), protocol 166/0, transport 73/0, reporter 519/0, wire-e2e 4/0. The red checks fail as intended without each change. 36 of 43 mutants are killed; the 7 survivors are test gaps and change no claim. The end-to-end rows R55, R96 and R96A pass on the stack and fail on s14b's files. Gate: ch01 GATE OK board 2429 (tree b500e451d4 on 6eae8fadad; 87e0e3a together with task 138's merge, in either order) Commits folded into this step (6): - c8cd36f574 [rush-daemon] A served rushx script releases the lifecycle gate once it starts, so reloads don't wait for it (board 113) - 4a5a13bddf [rush-daemon] Name the client's wait timeout and uncounted time in admission timeouts - cafa954eb4 [rush-daemon] A rushx script that arrives while a restart is pending waits for it instead of postponing it (task 96) - 7ba7021ed6 [rush-cli-client] Admission failures in legacy output and rushx-client print the daemon's reason (task 96) - 4ddb1fdbfe [rush-daemon] Spend a request's wait timeout only while it waits for other requests (task 98) - c35a77cc5c [rush-client-core] A client that follows a daemon restart connects to the successor the restarting daemon launches instead of racing it (task 96) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…has work Swarm integration step 30; original commit a4de651852 (merge of swarm/r07-t76 at 6b32aa340e). Scope: task 76. Brings r07's task 76 (board 1730; o05 CONFIRMED board 1904). A warm-set maintenance pass takes the native repository lease only when it has something to evict, a watcher change to apply, or a failed observation to retry. An idle daemon no longer takes common/temp/rush#<pid>.lock every 30 s, and a native command that holds the lock no longer puts the warm set in 'native-busy'. Two commits: 40a9f8f5eb (product) and 6b32aa340e (tests for the two lease exemptions a pass relies on). ch01 on tree fe0b6a4 (board 2511): rush-daemon 549/0 (544 + 5 new tests). The old WorkspaceWarmSet.ts fails exactly the 5 new tests. 15 of 16 WorkspaceWarmSet.ts mutants are killed; B6 (no protected check in #shouldEvict) survives and is a test NIT. E2E on ch01's own daemons: 0 lock holds in 150 s idle (5 before), 0 daemon lock attempts while native holds the lock 40 s (12 before, with 'native-busy'). Gate: ch01 GATE OK board 2511 (tree fe0b6a4) Commits folded into this step (2): - 40a9f8f5eb [rush-daemon] Warm set: take the repository lock only for a maintenance pass that has work - 6b32aa340e [rush-daemon] Warm set: test the two lease exemptions a pass relies on Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…ut a process Swarm integration step 233; original commit a1a015252b (merge of swarm/r06-t344 at 5cdc913318). Scope: task 344, 321 test NIT: no-process tests for the spawn mark writer. Brings 344 (branch swarm/r06-t344). Second agent: r03 board 8813. Evidence: ch01's GATE OK board 9622. Batch E2j, item 5 of 8. Gate: ch01 GATE OK board 9622 (tree 205fcb0) Commits folded into this step (1): - 5cdc913318 rush-daemon-transport: test the spawn mark and own group reads without a process Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
Swarm integration step 234; original commit 17a943c265 (merge of swarm/r07-t341 at 6bd98980ae). Scope: task 341, 113 follow-up: tests that kill two survivors. Brings 341 (branch swarm/r07-t341). Second agent: t06 board 9032. Evidence: ch01's GATE OK board 9622. Batch E2j, item 6 of 8. Gate: ch01 GATE OK board 9622 (tree fbdd18d) Commits folded into this step (1): - 6bd98980ae [rush-lib] Test two cases of deferred build cache writes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…d or retest Swarm integration step 235; original commit bf1ac546df (merge of swarm/r04-t340 at 665d8ec1b5). Scope: task 340, 108 README NIT. Brings 340 (branch swarm/r04-t340). Second agent: t07 board 8441. Evidence: ch01's GATE OK board 9622. Batch E2j, item 7 of 8. Gate: ch01 GATE OK board 9622 (tree 1c0a571) Commits folded into this step (1): - 665d8ec1b5 [rush-cli-client] Document the silent stop by a served reload, rebuild or retest Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
Swarm integration step 236; original commit ca8c052981 (merge of swarm/r04-t345 at 857fc03605). Scope: task 345, 340 follow-up: say which operations a served request stops. Brings 345 (branch swarm/r04-t345, 3 commits). Second agent: t07 board 9019 (m01 METRICS board 9124). GATE NIT: mutant M345-ANY-POSITION survives; the test-only follow-up is task 368 (D64 board 9574). Evidence: ch01's GATE OK board 9622. Batch E2j, item 8 of 8. Gate: ch01 GATE OK board 9622 (tree ed25ba3) Commits folded into this step (3): - f9ccf77215 [rush-daemon-protocol] Add a stopping flag to IDaemonContinuingOperations - 406b0d8325 [rush-daemon] Name the continuing operations that a queued request makes rushd stop - 857fc03605 [rush-cli-client] Say which operations rushd stops for a served command Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
Swarm integration step 237; original commit 3e71185392 (merge of swarm/o08-d551-t2 at 66bf274454). Scope: task 350, D55.1 T2: rig-package's missing -rig suffix error names the wrong field. Brings 350 (branch swarm/o08-d551-t2). Second agent: o03 board 9087 (CLAIM board 8944). Evidence: ch01's GATE OK board 9991. Batch E2k part 1 (K1), item 1 of 5. Gate: ch01 GATE OK board 9991 (tree da36e22) Commits folded into this step (1): - 66bf274454 rig-package: name the package in the missing "-rig" suffix error Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…oject's script Swarm integration step 238; original commit dc83bf8af5 (merge of swarm/r03-d551-t1 at e89a36bdea). Scope: task 349, D55.1 T1: quote custom parameter values in shell commands. Brings 349 (branch swarm/r03-d551-t1). Second agent: o08 board 9188. Evidence: ch01's GATE OK board 9991. Batch E2k part 1 (K1), item 2 of 5. Gate: ch01 GATE OK board 9991 (tree 73b7b3b) Commits folded into this step (1): - e89a36bdea [rush-lib] Quote custom parameter values for the shell that runs a project's script Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
Swarm integration step 239; original commit 83cc7cfc3d (merge of swarm/o07-d551-t5 at 5de59478c8). Scope: task 353, D55.1 T5: a warm TypeScript watch worker sees edits after a folder is recreated. Brings 353 (branch swarm/o07-d551-t5). Second agent: t03 board 9224. Evidence: ch01's GATE OK board 9991. Batch E2k part 1 (K1), item 3 of 5. Gate: ch01 GATE OK board 9991 (tree def670f) Commits folded into this step (1): - 5de59478c8 [rush-lib] Read a warm worker's input folders before its command starts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…already waits first Swarm integration step 240; original commit 5f436fbe16 (merge of swarm/r04-t361 at 5156e5a0c6). Scope: task 361, 345 follow-up: a request queued before a failed build becomes preemptible. Brings 361 (branch swarm/r04-t361). Second agent: t07 board 9533 (RESULT board 9532). Evidence: ch01's GATE OK board 9991. Batch E2k part 1 (K1), item 4 of 5. Gate: ch01 GATE OK board 9991 (tree 964d518) Commits folded into this step (1): - 5156e5a0c6 [rush-daemon] Name the operations that rushd stops to a request that already waits first Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…d selects (and 1 more) Swarm integration step 241; original commit 0a7af6a3df (merge of swarm/r03-row23 at 4a746d0348). Scope: task 363, row 23: rush-bridge-cache-plugin acts only on the selected operations. Brings 363 (branch swarm/r03-row23, 2 commits). Second agent: ch05 board 9400. Evidence: ch01's GATE OK board 9991. Batch E2k part 1 (K1), item 5 of 5. Gate: ch01 GATE OK board 9991 (tree 228f843) Commits folded into this step (2): - 92b277d814 [rush-bridge-cache-plugin] Act only on the operations that the command selects - 4a746d0348 [rush-bridge-cache-plugin] Test a selected operation whose record is disabled Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
Swarm integration step 242; original commit b20e9983fa (merge of swarm/r01-t117-p2 at ab4690c4b3). Refs: task 117. Scope: task 117 Part 2: prepare the next engine in the background. Brings R (branch swarm/r01-t117-p2). Second agent: r02 board 9649 (RESULT board 9647). Evidence: ch01's GATE OK board 10189. Batch E2k part 2 (K2), item 1 of 9. Gate: ch01 GATE OK board 10189 (tree 3c32e43) Commits folded into this step (1): - ab4690c4b3 rush-daemon: Prepare the next engine in the background (task 117) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
Swarm integration step 243; original commit fb7eaa0bbf (merge of swarm/r01-t117-p2-count at 49b2a1b5c6). Refs: board 9212. Scope: task 117 Part 2 follow-up: count the capture's calls in BackgroundPrepare's tests. Brings F (branch swarm/r01-t117-p2-count). Second agent: r02 board 9865 (RESULT board 9863). Evidence: ch01's GATE OK board 10189. Batch E2k part 2 (K2), item 2 of 9. Gate: ch01 GATE OK board 10189 (tree c3f57b7) Commits folded into this step (1): - 49b2a1b5c6 rush-daemon: Count the capture's calls in BackgroundPrepare's tests (board 9212) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
… preparing Swarm integration step 244; original commit 66229fc83a (merge of swarm/r01-t359 at 99625a4eb6). Refs: task 359. Scope: task 359: a preparation failure before the reload stops preparing. Brings T (branch swarm/r01-t359). Second agent: ch05 board 9916 (RESULT board 9915). Evidence: ch01's GATE OK board 10189. Batch E2k part 2 (K2), item 3 of 9. Gate: ch01 GATE OK board 10189 (tree 65d1ae3) Commits folded into this step (1): - 99625a4eb6 rush-daemon: Test that a preparation failure before the reload stops preparing (task 359) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
Swarm integration step 245; original commit 1faa9f9d1f (merge of swarm/r02-d551-pr-hygiene at a82c9c190f). Scope: task 355, PR hygiene: drop internal references from test names and comments. Brings 355 (branch swarm/r02-d551-pr-hygiene). Second agent: ch05 board 9628 (RESULT board 9627). Evidence: ch01's GATE OK board 10189. Batch E2k part 2 (K2), item 4 of 9. Gate: ch01 GATE OK board 10189 (tree cfc98bc) Commits folded into this step (1): - a82c9c190f Drop internal references from test names and comments Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…MPDIR is Swarm integration step 246; original commit 5ab3a13dda (merge of swarm/r02-d551-t4 at d1b14ef3c3). Scope: task 352, D55.1 T4: rush-daemon test socket paths stay short however long TMPDIR is. Brings 352 (branch swarm/r02-d551-t4). Second agent: r01 board 9940 (RESULT board 9937). Evidence: ch01's GATE OK board 10189. Batch E2k part 2 (K2), item 5 of 9. Gate: ch01 GATE OK board 10189 (tree 77ba7e3) Commits folded into this step (1): - d1b14ef3c3 Keep rush-daemon test socket paths short however long TMPDIR is Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
… responded Swarm integration step 247; original commit b41e167dbc (merge of swarm/r04-t364 at 455b9b1940). Scope: task 364: the silence line names `rush-client daemon status`. Brings 364 (branch swarm/r04-t364). Second agent: t07 board 9930 (RESULT board 9929). Evidence: ch01's GATE OK board 10189. Batch E2k part 2 (K2), item 6 of 9. Gate: ch01 GATE OK board 10189 (tree a9bde89) Commits folded into this step (1): - 455b9b1940 [rush-cli-client] Name `rush-client daemon status` when rushd has not responded Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
Swarm integration step 248; original commit 86204d0550 (merge of swarm/r06-d551-t3 at 298b23b0d2). Scope: task 351, D55.1 T3: package-deps-hash hashes the paths that git hash-object can't read. Brings 351 (branch swarm/r06-d551-t3). Second agent: t04 board 10037. Evidence: ch01's GATE OK board 10189. Batch E2k part 2 (K2), item 7 of 9. Gate: ch01 GATE OK board 10189 (tree 772d755) Commits folded into this step (1): - 298b23b0d2 package-deps-hash: hash the paths that "git hash-object" can't read Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
…r path against the working directory (and 1 more) Swarm integration step 249; original commit 5aecba33b6 (merge of swarm/r06-t366 at 03a65659c2). Scope: task 366, 351 follow-ups: sign an input that is a link to no regular file by the link itself. Brings 366 (branch swarm/r06-t366, 2 commits). Second agent: t04 board 10037. Evidence: ch01's GATE OK board 10189. Batch E2k part 2 (K2), item 8 of 9. Gate: ch01 GATE OK board 10189 (tree 506a5b0) Commits folded into this step (2): - 720874a782 [package-deps-hash] Resolve a relative RepoStateCache temporary folder path against the working directory - 03a65659c2 [rush-lib] Sign an input that is a symbolic link not leading to a regular file by the link itself Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
… 1 more) Swarm integration step 250; original commit 051bfee3e4 (merge of swarm/o01-t365 at fb77adfde9). Scope: task 365, C1 and C2: rush-daemon tests print no environment in failed assertions, and delete secret-shaped variables first. Brings 365 (branch swarm/o01-t365, 2 commits). Second agent: r01 board 9962 (RESULT board 9901). Evidence: ch01's GATE OK board 10189. Batch E2k part 2 (K2), item 9 of 9. Gate: ch01 GATE OK board 10189 (tree 090484c) Commits folded into this step (2): - df1f14139a rush-daemon: keep the environment out of failed test assertions - fb77adfde9 rush-daemon: delete secret-shaped environment variables before each test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
Sean Larkin (TheLarkInn)
marked this pull request as ready for review
September 30, 2026 11:53
Sean Larkin (TheLarkInn)
merged commit Sep 30, 2026
8c902ee
into
thelarkinn-dogfood-rush-daemon
1 check passed
Sean Larkin (TheLarkInn)
deleted the
thelarkinn/rushd-swarm-bug-fixes
branch
September 30, 2026 11:54
Sean Larkin (TheLarkInn)
added a commit
that referenced
this pull request
Sep 30, 2026
Replace the 489 change files that this branch added with one change file for each of the 18 projects it changes, with shorter comments. Also delete 27 change files that came from main commits 1f51ac8..982b33b. Main has already published and deleted them, and the squash merge of #6102 dropped the merge of main that contained them, so merging this branch would add them back. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
Sean Larkin (TheLarkInn)
added a commit
that referenced
this pull request
Sep 30, 2026
The branch tree already contains main up to 982b33b: the squash merge of #6102 was built on a merge of 982b33b, but the squash dropped that parent. This merge restores the ancestry without changing the tree, so the next merge of origin/main only brings the commits after 982b33b. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #6046, so the base is
thelarkinn-dogfood-rush-daemon. It stays a draft until it's reviewed. The bugs that aren't fixed yet are listed below, with where each one stands.Summary
This PR fixes the Rush daemon (rushd) bugs found by an overnight dogfooding run in a large monorepo. Agents built and tested each fix, and a second agent reproduced the result before it landed.
maininto [rush] Dogfood the Rush daemon in rushstack: allow command-scoped plugins, add snapshot workflow and guide #6046's branch. Its 15maincommits appear in this PR's diff until [rush] Dogfood the Rush daemon in rushstack: allow command-scoped plugins, add snapshot workflow and guide #6046 takes the same merge. To see only this PR's changes, use 530c72f9ee...thelarkinn/rushd-swarm-bug-fixes.Bugs (166 reported)
Check off each bug as you review it. Each link is a commit that fixes the bug.
Fixed in this PR (130)
Same change as the row for board 2284: a run that writes nothing to stderr deletes the old error log, so no cache entry packs a stale one.
The daemon now stays up after a served install or update that fails before it changes the installation. A served update that rewrites the repo's scripts and then fails does what native Rush does.
Introduced by an earlier commit in this PR, cd7a2a5b73, so it isn't in the base branch.
Introduced by an earlier commit in this PR, cd7a2a5b73, so it isn't in the base branch.
Introduced by an earlier commit in this PR, cd7a2a5b73, so it isn't in the base branch.
The rush-lib side adds a per-request key; the dogfood repo's telemetry plugin reads it in its own change.
a|bor parentheses break the command (native Rush too). (board 944) — 0e7f4ceab8Introduced by an earlier commit in this PR, 6a05b01dee, so it isn't in the base branch.
Step 69's folder check fixes the deleted-and-recreated form: before each warm run, Rush compares the identity of every folder that holds an input file and closes the worker if one changed. Step 239 also reads those folders before a new worker's first command starts, so files moved into a recreated folder during that first run are caught too.
Introduced and fixed by commits that this step folds together, so neither the base branch nor any commit of this PR has it.
Introduced by an earlier commit in this PR, 0b978371ea, so it isn't in the base branch.
Introduced by an earlier commit in this PR, 071dc65872, so it isn't in the base branch.
Introduced by an earlier commit in this PR, ab2fb7786b, so it isn't in the base branch.
rush-lib queues a watch iteration for the request, and heft-webpack5-plugin waits for pending watch events before it flushes.
Introduced and fixed by commits that this step folds together, so neither the base branch nor any commit of this PR has it.
rush-lib half: its cache-write guard trusts a skip that a plugin marks as verified at the operation's own state hash. The dogfood repo's fstrace plugin carries the other half.
Introduced by an earlier commit in this PR, 5b89b0c3ff, so it isn't in the base branch.
:incrementalscript without a guard reason and leaves stale outputs. (board 1893) — d2dc437514watchFile: useFsEventsOnParentDirectory, heft's TypeScript watch program can stop noticing a source file that was missing at a program update, even after the file comes back. (board 2854, task 171) — 98c5d18bc9rush update-cloud-credentials --delete). (board 2891, task 172) — fe18240a4e--clean. (board 4403, task 236) — 22e0c0b5f4.gitattributesfile that the index marks assume-unchanged or skip-worktree. (board 4661, board 4779, task 252) — 5a508b0a1c, b2cd1ada36Introduced by an earlier commit in this PR, 0b978371ea, so it isn't in the base branch.
Introduced by an earlier commit in this PR, d2dc437514, so it isn't in the base branch.
azureBlobStorageConfiguration.storageEndpoint, which the Azure cache plugin reads, so build-cache.json can't set a custom storage endpoint. (task 144) — 02e9125d14disableBuildCache, because the phased-command schema leaves it out. (board 3503, task 205) — 1534a71c4arushcommand, or the client's in-process fallback) can crash with an unhandled EPIPE and exit code 1 when the reader of its stdout exits (for example| head), and can orphan the running operation. (task 254) — 919fb7aedcps(0.15–0.23 s each) even when the holder's pid has no /proc entry. (task 263) — 3009e26d1fdaemon stopand the first SIGTERM can fail to end a daemon whose request is stuck in an await that ignores cancellation. (board 1712, task 115) — c97c702aa8daemon stop, a ref'd timer or handle can keep the daemon process alive after it has released its socket. (board 2199, task 147) — c97c702aa8, 3c0d3d9936Introduced by an earlier commit in this PR, ab2fb7786b, so it isn't in the base branch.
Introduced by an earlier commit in this PR, 5a1996bb74, so it isn't in the base branch.
Error: write EPIPEstack for every reply to a client that has already gone. (task 260) — 1fed617104, 52f68f1625--wait-timeout 600. (task 245) — 1ef258f07eIntroduced by an earlier commit in this PR, be85a49b8a, so it isn't in the base branch.
Introduced by an earlier commit in this PR, adf57d5899, so it isn't in the base branch.
--onlyselection. (board 1533, board 1586, board 1775) — 4502875149rejects.toMatchObjectchecks in ProductionDaemonRequestParse and ProductionDaemonEnvironmentIdentity print the request they resolved to,not.toHaveBeenCalled()checks in DaemonInstallationChange, FailedNativeMutation and ProductionDaemonEnvironmentIdentity print the calls' arguments, and an identity check in PhasedRequestBatching prints both environments. (board 9212, board 9262, board 9282, board 9291, board 9306) — 5ff84efc5dIntroduced by earlier commits in this PR, 4388b98bae, 23c3e88258, be85a49b8a, 268c8665d6, 660c20eb13 and efb546253c, so it isn't in the base branch.
temporaryFolderPathas given, so it creates its private index relative to the process's working folder while Git resolves GIT_INDEX_FILE against the repository root. When the two differ, Git reads an empty index, and the repo state silently drops files that exist only in the index and reports uncommitted changes. Rush passes its temp folder, which is relative only if RUSH_TEMP_FOLDER is. (board 9276) — eea0048cdc, 832fe1294dIntroduced by an earlier commit in this PR, 0b978371ea, so it isn't in the base branch.
Fixed, with a known residual (4)
Residual: when old and new Rush versions share a repository, an old and a new process that create their lock files in the same millisecond can leave the lock with no holder. Every command in that tie can then fail with "Another Rush command is already running in this repository." although none holds the lock, and the next command gets it. That needs a release note, not code.
Residual: when old and new Rush versions share a repository and use different TZ or locale settings, an old process can still treat a lock that a new process holds as stale, delete it and take the lock. That needs a release note, not code.
Step 248 hashes the paths that Git lists but can't read from disk, and step 249 signs an input link that leads to no regular file by the link itself. So none of the three triggers fails any more: a sparse-checkout or assume-unchanged path that's missing on disk, an untracked link that dangles, and one that points at a folder. Residual: a file removed from disk that the index marks skip-worktree or assume-unchanged still costs the first build's cache write, as a row below says.
Introduced by an earlier commit in this PR, 9f5b48595f, so it isn't in the base branch. Step 249 fixes the link form: an input link that leads to no regular file is signed by the link itself, so it no longer blocks the write. Residual: a file removed from disk that the index still marks skip-worktree or assume-unchanged still blocks the first build's cache write, and a later build writes the entry. No wrong output, only a lost write and an extra rebuild.
Not fixed yet (3): 1 in progress and 2 held back for a missing test
The dogfood repo works around it. A fix is written, and it passed every second-agent check but one: its tests don't catch a weaker version of the fix. A test fix is drafted but wasn't checked in time, so the fix isn't in this PR.
rush installwrites the wrappers for common/git-hooks into the worktree's private Git folder (.git/worktrees/<name>/hooks). Git never reads hooks from there. So a commit in a linked worktree runs no Rush hooks or, if the main worktree has runrush install, runs the main worktree's scripts.Git.getHooksFolder()uses the worktree's Git folder, whilegetIsHooksPathDefaultAsync()correctly compares against the shared one. (board 9425, board 9426)In the base since [rush] Properly handle Git worktrees #994, which fixed the crash in [rush] Git hooks feature causes Rush commands to fail in worktrees #990 by using the worktree's Git folder. Not a daemon bug; the run found it because its agents commit from linked worktrees. Reproduced with Git 2.55.0. The fix needs a design first: the shared hooks folder serves every worktree, and each wrapper finds its script by a path relative to the hooks folder, which would then point into whichever worktree ran the install. Not fixed yet.
Introduced by an earlier commit in this PR, 48930441c7, so it isn't in the base branch. Measured with 16 fragmented files: a small file's write, rename and stat, started 500 ms into the clones, took 1.2 s with 16 clones in flight and 1 ms with 2. A fix is written, but its check for output files that change while they're cloned isn't tested yet on the path rushd uses, so it isn't in this PR.
Covered by guidance, with no code change (1)
Fixed outside rushstack: the dogfood repo's own plugins, tools, scripts and config (22)
In the run's own tooling and the dogfood repo's telemetry dashboard (6)
env -i. (board 1391, board 1382)Notes for review
ODSP_TELEMETRY_TAGis added to the daemon's environment lists in 1b71530939. It's specific to the dogfood repo's telemetry. Generalize it or drop it before this merges.