feat(serverless): coordinate early health checks once per container

[justinwlin]

Why

Serverless workers used to run health checks at serverless.start(), which can happen after a large model has loaded. An unhealthy worker could spend that time loading a model before it failed.

This PR moves the hardware checks that are safe to run early to the first import runpod. Checks that need the running worker stay at worker start.

Normal Serverless path

Import runpod
    |
    v
Check RAM, disk, CUDA version, and GPU hardware
    | failure -> Exit unhealthy
    v pass
Load the model
    |
    v
Start the worker
    |
    v
Recheck disk; check network, CUDA initialization,
GPU compute, and customer-registered checks
    | failure -> Exit unhealthy
    v pass
Accept jobs

Disk is checked twice because loading a model can use the space that was available at import time.

Other paths and controls

• Realtime workers: No fitness checks, matching their behavior on main. RUNPOD_REALTIME_PORT excludes them from the early pass.
• Local/test runs: No early pass for RUNPOD_TEST, --test_input, or --rp_serve_api. Early checks also require both RUNPOD_ENDPOINT_ID and RUNPOD_WEBHOOK_GET_JOB.
• RUNPOD_DEFER_FITNESS_CHECKS=true: Run the checks at worker start instead of import time.
• RUNPOD_SKIP_FITNESS_CHECKS=true: Disable all fitness checks.
• Child processes: After a successful early pass, RUNPOD_EARLY_FITNESS_CHECKS_DONE=1 is inherited by children. They skip the early GPU probe if they import runpod again. A failed pass does not set the marker.

The early trigger is in runpod/__init__.py, so it does not depend on importing runpod.serverless eagerly. Changes to check thresholds after import are applied at worker start; disk is rechecked there after an early pass. The network check also runs at worker start, targets the worker API host, and retries within a bounded time. Registration and worker-start setup failures use the same unhealthy-exit path.

This PR also fixes logger redaction and moves health modules to runpod/_health/, with aliases at their old import paths. When feat/apps-sdk is merged into main, its lazy __init__ will need to carry the run_import_checks() call.

Validation

After rebasing on main, the full local suite passed on Python 3.11: 787 tests, 6 subtests, 93.97% coverage. A real multiprocessing spawn test confirms that a child inherits the marker and runs no early probes.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Auto-registration failures can currently escape the hard-exit failure path, undermining the “must not hang” operational guarantee during worker startup.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Moves most serverless worker fitness checks earlier (at import runpod.serverless) to fail unhealthy workers before model load, while keeping CUDA-context-creating checks deferred to start() and adding env flags to skip/defer behavior.

Changes:

• Add an import-time startup pass (run_startup_fitness_checks) gated by worker env, with once-per-process deduping and deferred-check support.
• Introduce global skip/defer env flags and more consistent “truthy” env parsing; add nvidia-smi timeout for CUDA detection.
• Add/expand tests and docs to cover startup timing, deduping, deferred checks, and late-config warnings.

File summaries

File	Description
tests/test_serverless/test_worker.py	Asserts worker loop still runs fitness checks.
tests/test_serverless/test_utils/test_cuda.py	Updates CUDA availability test expectations for timeout=5.
tests/test_serverless/test_modules/test_fitness/test_startup.py	New test suite validating import/start timing, deferral, dedupe, and config warnings.
tests/test_serverless/test_modules/test_fitness/conftest.py	Resets new startup/dedupe global state between tests.
runpod/serverless/utils/rp_cuda.py	Adds bounded nvidia-smi probe with timeout to avoid hangs.
runpod/serverless/modules/rp_system_fitness.py	Marks CUDA-init and benchmark checks as deferred-to-worker-start.
runpod/serverless/modules/rp_gpu_fitness.py	Uses shared truthy env flag parsing for skip behavior.
runpod/serverless/modules/rp_fitness.py	Implements startup pass, deduping, skip/defer env flags, and late-config warnings.
runpod/serverless/init.py	Triggers startup checks at import (worker-only no-op otherwise).
README.md	Updates high-level behavior summary for new timing model.
docs/serverless/worker_fitness_checks.md	Documents import-time checks, deferral, and new env flags.
ARCHITECTURE.md	Updates architecture docs for new timing and hard-exit behavior.

Review details

• Files reviewed: 12/12 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[deanq]

Review from /code-review (correctness + cleanup pass). Four findings, most centered on moving os._exit(1)-capable checks to import time. Lines re-anchored to the current diff.

[justinwlin]

bugbot run

[justinwlin]

Update: lock file replaced with an inherited env marker, realtime hook removed

Summary of what changed since the last round of review, and how the four review points map to the current code.

What the PR does now, in one paragraph

import runpod in a Serverless worker runs the four hardware checks that are safe before the handler exists (RAM, disk, CUDA version, native GPU test). A failure exits the process right there, before any model loads. Network, torch CUDA init, GPU benchmark, and customer checks still run at start(). When the import pass passes, the process sets RUNPOD_EARLY_FITNESS_CHECKS_DONE=1, and every child it launches inherits that and skips the pass.

Why the lock file is gone

The shared /tmp lock, procfs identity, and persisted failure state solved one problem: child processes that re-import the handler (vLLM spawn workers, multiprocessing, subprocesses) repeating the 30-second GPU probe. An inherited env var solves the same problem with no shared state, no lock waits, and no sticky failures. The realtime lifespan hook came out too, since realtime workers never ran fitness checks on main and that should be its own decision.

Review points

Point	Status	Where
Import gate misses local mode and --test_input	Fixed	is_early_check_eligible requires both worker env vars and skips RUNPOD_TEST, --test_input, --rp_serve_api
apps-sdk lazy-loads serverless, so the trigger never fires early	Fixed	Trigger moved to top of runpod/__init__.py; independent of serverless import. apps-sdk will need to carry the one-line call when main is merged in
Network check at import turns cold-boot network delay into a crash loop	Fixed	_network_check is @defer_to_worker_start and retries within one bounded budget
Late-config comparison is overhead on the import pass	Fixed	_refresh_late_config only runs on the worker-start pass
Registration failures escape the hard-exit path (Copilot)	Fixed	Registration is atomic; worker-start setup errors go through _fail_worker

Examples

Healthy worker. Logs from a real RTX 4090 Serverless worker on this branch:

Running 4 fitness check(s)...
GPU binary test passed: 1 GPU(s) healthy (CUDA 12.8)
Memory check passed: 184.90GB available (of 247.52GB total)
Disk space check passed: 19.78GB free (98.9% available)
CUDA version check passed: 12.8 (minimum: 11.8)
All fitness checks passed. (599.98ms)
IMPORT_DONE 3.82s marker=1
--- pretend model load ---
--- Starting Serverless Worker |  Version 1.12.1.dev23 ---
Running 3 fitness check(s)...
Network connectivity passed: Connected to api.runpod.ai:443
CUDA initialization passed: 1 device(s) initialized successfully
GPU compute benchmark passed: Matrix multiply completed in 108ms

Four checks inside the import, three at start(), nothing runs twice.

Broken worker. Same endpoint with RUNPOD_MIN_MEMORY_GB=99999 to force a failure:

Running 4 fitness check(s)...
GPU binary test passed: 1 GPU(s) healthy (CUDA 12.8)
Fitness check failed: _memory_check | RuntimeError: Insufficient memory: 207.61GB available, 99999.0GB required
Worker is unhealthy, exiting.

Exit about one second into import runpod. The model-load line never prints. The platform marked the worker UNHEALTHY within a minute.

Child process. A handler that spawns a multiprocessing child, which re-imports the handler and therefore runpod:

Running 4 fitness check(s)...        <- parent, once
All fitness checks passed. (416.35ms)
PARENT import 2.29s marker=1
PARENT import 1.59s marker=1         <- child re-import, no checks
('child sees marker', '1')

Opt-outs. RUNPOD_DEFER_FITNESS_CHECKS=true restores the old worker-start timing. RUNPOD_SKIP_FITNESS_CHECKS=true disables all checks. Local --test_input runs and RUNPOD_TEST are exempt automatically.

Validation

• 694 tests pass locally, 93% coverage, CI green on 3.10 through 3.14.
• Live on a Pod (RTX 4090): healthy import, forced failure, spawn child, and each exemption flag.
• Live on a Serverless endpoint (ADA_24): healthy cold start with a completed job, then a forced-failure release observed crash-looping at import.

[justinwlin]

@copilot review

[justinwlin]

  Process starts
      |
      v
  import runpod
      |
      +-- Realtime, local/test run, or checks disabled?
      |       |
      |       +-- Yes --> Skip early checks
      |
      +-- Checks deferred until worker start?
      |       |
      |       +-- Yes --> Wait until worker start
      |
      +-- Otherwise, run early checks:
              RAM → disk → CUDA version → GPU hardware
                  |
                  +-- Failure --> Exit unhealthy
                  |
                  +-- Pass --> Mark early checks done
                                 |
                                 v
                          Load the model
                                 |
                                 v
                          Start the worker
                                 |
                                 +-- Recheck disk space
                                 +-- Check network, CUDA initialization,
                                     GPU compute, and custom checks
                                        |
                                        +-- Failure --> Exit unhealthy
                                        |
                                        +-- Pass --> Accept jobs

  Child processes inherit the “early checks done” marker, so they skip the
  early checks when they import runpod again.

[daveseddon-runpod]

Nice work — the safety reasoning behind this change is the hard part, and it shows.

👍 What's good

• Safe/unsafe split is right: @defer_to_worker_start on _cuda_init_check/_benchmark_check keeps a CUDA context out of the pre-fork import path (vLLM/DeepSpeed), with docstrings explaining why. Network check deferred too.
• os._exit vs sys.exit documented with the real reason (live non-daemon threads blocking cooperative shutdown).
• Child marker RUNPOD_EARLY_FITNESS_CHECKS_DONE replacing the lock-file/procfs machinery — clean simplification; "marker only ever means a parent passed" is a tidy invariant.
• Security: log.secret() now fully [REDACTED] (won't str() the value) — closes the CodeQL clear-text finding and the old length side-channel.
• sys.modules aliases keep every old import path working.
• Latch-on-success registration (_register_builtins rolls back so malformed config re-raises loudly, not silently) — good catch, and it's tested.
• Docs (README, ARCHITECTURE, worker_fitness_checks.md) updated in step.

🔴 Discuss before merge

• import runpod can now os._exit(1) any importer in a worker container — not just the worker (sidecar scripts, spawn re-imports, a debug shell). Can we (a) call this out in _startup.py's docstring / README gotchas, and (b) confirm the gate (RUNPOD_ENDPOINT_ID + RUNPOD_WEBHOOK_GET_JOB minus test flags) is tight enough that no standard-image tooling trips it? Biggest behavioral change here.
• _report_unhealthy dropped the shared HTTP client + WORKER_ID for raw requests.Session() + os.environ["RUNPOD_POD_ID"]. Intended? It loses any TLS/proxy/UA/retry defaults the shared client carries, and if $RUNPOD_POD_ID is in the URL but unset, it now sends nothing (vs a WORKER_ID fallback). Best-effort, so low impact — but lazily reusing the shared client would be safer.
• Type the check metadata (ties to modern types): flags live as dynamic attrs (_runpod_defer_to_worker_start, _runpod_recheck_at_worker_start, _runpod_builtin); mypy --check-untyped-defs flags every read. A small @dataclass/Protocol wrapping callable + flags makes the states typed and discoverable and kills the "typo in a getattr default silently disables a check" class of bug.

🟡 Medium

• nvidia-smi probed 3+ times on the startup path (cuda.is_available(), gpu.auto_register_gpu_check(), system._get_cuda_version()), each with a multi-second timeout. Cache detection (e.g. lru_cache) — a DRY win that also speeds the startup this PR targets.
• configure() reassigns 5–7 module globals via global (PLW0603). With the above, a frozen config object built from env once would be more testable and type-checked.
• binary_error typing (gpu.py): inferred FileNotFoundError | None, later reassigned Exception. Annotate binary_error: Exception | None = None.

🟢 Low / nits

• _logger.py uses Optional[str]; project is >=3.10, so str | None is idiomatic (ruff UP045) and drops the typing import.
• ruff PLR2004/PLR0912 in _get_cuda_version — a named constant / small split reads better.
• No CHANGELOG.md entry despite new env vars + import-time behavior — worth one (confirm repo convention).
• Remaining ruff hits (PLC0415, S110, S607, PERF203) are intentional/correct here — noting them only to show they were reviewed, not missed.

🔒 Security

• Redaction improvement is solid; corner cases (empty, 1-char, None, __str__-raises) are tested.
• _network_probe_target avoids logging URL creds and fails safe on an unparseable URL.
• _report_unhealthy truncates reason to 256 chars to the same host as the heartbeat — low risk; note the exception message could carry paths.
• No new privilege: RUNPOD_SKIP_FITNESS_CHECKS is env-gated like existing controls.

✅ Tests

• Strong coverage: eligibility, realtime exclusion, deferred-vs-early, marker inheritance (real multiprocessing spawn), disk-recheck-at-start, network retry/bounded-close, malformed-config re-raise, redaction corner cases.
• test_secret_redacts_short_empty_and_object_values is effectively table-driven. Two asks to match our convention: (a) give each row an explicit description/expected so failures name the case; (b) fold the separate _network_check/URL-fallback tests into one @pytest.mark.parametrize table.

Overall: strong PR. Only real gates are the two 🔴 items; the rest are improvements.

Static analysis run (commands + results)

$ ruff check --select E,F,W,B,UP,SIM,C4,ASYNC,S,PERF,RUF,PL --line-length 100 runpod/
# 36 findings, by rule:
  12  PLC0415  import should be at top-level         (intentional: deferred to break import cycle)
   7  UP045    use `X | None` for annotations        (_logger.py — modernize, drop `Optional`)
   7  PLW0603  `global` update in configure()        (system.py / gpu.py — see Medium)
   3  S607     partial executable path ("nvidia-smi")(accepted: PATH lookup of a system tool)
   2  S110     try/except/pass                       (accepted: best-effort before os._exit)
   2  PERF203  try/except within loop                (nvcc/nvidia-smi fallback — minor)
   1  PLW2901  loop var `line` overwritten           (gpu.py:82 — harmless)
   1  PLR2004  magic value `2`                       (system.py:243)
   1  PLR0912  too many branches (13>12)             (_get_cuda_version)

$ mypy --check-untyped-defs --explicit-package-bases runpod/_health/ runpod/_logger.py runpod/_startup.py
runpod/_health/fitness.py:121 error: "Callable[..., Any]" has no attribute "_runpod_defer_to_worker_start"
runpod/_health/fitness.py:131 error: "Callable[..., Any]" has no attribute "_runpod_recheck_at_worker_start"
runpod/_health/fitness.py:261 error: "Callable[..., Any]" has no attribute "_runpod_builtin"
runpod/_health/fitness.py:289 error: "Callable[..., Any]" has no attribute "_runpod_builtin"
runpod/_health/gpu.py:88/90/106/110 error: loose dict typing in _parse_gpu_test_output ("object" ...)
runpod/_health/gpu.py:259/263 error: binary_error type too narrow (FileNotFoundError | None <- Exception)

The mypy check-metadata errors are the concrete backing for 🔴 #3; UP045 backs Low nit #1; PLW0603 backs Medium; the binary_error errors back Medium. The remaining ruff hits are the intentional-and-accepted set noted under Low.