Add host-callback contract tests from the stream-callback survey

[0z5a]

Incremental review: Files changed against main.

Scope

Related to #311. This does not close it.

Scope approved in: not yet confirmed. #311 asks for stream callbacks, and its concrete design overlaps #2058 (host_launch, assigned to @Andy-Jost), so I asked which slice is independent before writing anything: #311 (comment) and #2058 (comment). Pending that answer, this PR adds only contract tests for the host-callback helper that already exists on main — no public API, no implementation change.

This PR changes: one new test file, cuda_core/tests/graph/test_host_callback_contracts.py — three behaviours of _resolve_host_callback / the Python trampoline that the existing suite did not pin down.

Out of scope: an ordinary-stream callback entry point (Stream.add_callback, a cuStreamAddCallback wrapper, or another trampoline) — that is #2058's design; the stream-side ordering/lifetime matrix of #311, which cannot be asserted until that entry exists; capture/replay coverage.

Contract

The three cases, and what they fix in place:

• Exception reporting. The trampoline is cdef void _py_host_trampoline(void* data) noexcept with gil, so a Python exception raised inside a callback cannot cross back into the driver. Observed on this commit: the exception is delivered to sys.unraisablehook, graph.launch() and stream.sync() return normally (no CUDA error), the node after the callback still executes, and a later launch still runs its callback. The test asserts exactly that, rather than promising a mechanism the API does not document.
• Raw user_data. user_data given as an int is the caller's own pointer and reaches a ctypes callback unchanged (_resolve_host_callback copies only the bytes form). The test asserts pointer identity and the byte read through it; the caller keeps owning the storage, and there is deliberately no "freed early but still expected to work" case.
• Two devices. Two devices, two graphs each, each graph with its own stream, device buffer and host slot: every callback observes exactly its own value, and the set of callbacks is exact.

Evidence

• Base SHA: 8b6e9f52db1074a580fc1678c71e7c75c42db2ea
• Tested SHA: 7cdc6395632c181d543f819a7f75c8056625592f — the commit in this PR (cuda_core/tests/graph/test_host_callback_contracts.py, sha256 21b71759af2f3a359f70c385cc8171127390c3a8846ed49074669d811a05111a); the branch carries that same commit object.
• CUDA Python components: cuda-core 1.2.1.dev112+g8b6e9f52d built from this tree (pip install --no-build-isolation -e cuda_core), cuda-bindings 12.9.8, Cython 3.2.9, Python 3.10.12, CUDA toolkit /usr/local/cuda
• GPUs: 2 × NVIDIA L20 (SM89), driver 595.91.07, NVML 13.595.91.07; devices 0 and 1 of the host (GPU-836d0605-… / GPU-739e6fcb-…)
• Deterministic tests: cuda_core/tests/graph/test_host_callback_contracts.py — 3 passed; the file was run 20 consecutive times to check for flakiness (all green)
• Second host (8 x RTX 5090, Python 3.12.13, cuda-bindings 13.4.2, CUDA 13.4 headers), same commit: 3 passed in 1.04 s, cuda_core/tests/graph 502 passed, 33 skipped, and cuda_core/tests in full 4317 passed, 0 failed — so the three assertions hold on both a 2-GPU and an 8-GPU machine
• Regression: cuda_core/tests/graph — 502 passed, 33 skipped (three consecutive runs); cuda_core/tests/system + cuda_core/tests/test_stream.py — 124 passed, 130 skipped, 8 xfailed, 984 subtests passed
• Native tests: all three cases execute real graphs on the two L20s; nothing is mocked, and the multi-device case covers the two GPUs that were free on the host
• Skipped or blocked cases: BLOCKED: dependency — C01–C05 and C09 of the Add stream callback support #311 matrix (single-stream callback, repeated callbacks, two streams per GPU, two GPUs × two streams, two host threads, owning vs borrowed stream) all need the ordinary-stream entry point that main does not have yet; reported as blocked rather than asserted or failed. OUT OF SCOPE — capture/replay (C13).
• Duplicates deliberately not added: prerequisite-data visibility and post-callback ordering are already covered by tests/graph/test_graph_definition_integration.py::test_heat_diffusion and test_graph_definition.py::test_instantiate_and_execute_host_callback; callable lifetime by test_graph_definition_lifetime.py::test_python_callable_callback_survives_del and friends; signature rejection and bytes user_data by the test_host_callback_* cases; cleanup coalescing and shutdown safety by test_user_object_cleanup_is_coalesced_on_python_thread, test_pending_call_queue_saturation_preserves_cleanup and test_pending_cleanup_is_safe_during_python_shutdown.

Limits

• No implementation change is included: the survey reproduced no defect in the existing helper, so there is nothing to fix yet. If the stream-side work later reproduces one, the fix should be a separate, narrow change.
• Callback concurrency is not asserted, and must not be: CUDA does not define ordering between independent streams or callbacks, so the tests only check each graph's own order, count and data.
• No fault injection was attempted (no XID/ECC/GPU-lost, no driver unbind, no context corruption) — that is not acceptable on a shared host, and the tests therefore do not claim those paths.
• Exceptions raised inside a ctypes callback are not covered: the exception cannot cross the C ABI, so such a case would be an invalid use rather than a contract.
• sys.unraisablehook is process-global, so the reporting test is marked thread_unsafe, following the existing precedent in tests/test_program.py.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[0z5a]

Re-verified on a second host (8 × RTX 5090)

No code change since the first push — this is the same commit 7cdc63956 — but the three cases were re-run on a second machine to check that they are not tuned to one stack: 8 × RTX 5090 (SM120), driver 580.82.07, Python 3.12.13, cuda-bindings 13.4.2, CUDA 13.4 headers.

• test_host_callback_contracts.py: 3 passed in 1.04s (same file, sha256 21b71759… as in the commit).
• cuda_core/tests/graph on that host: 502 passed, 33 skipped.
• cuda_core/tests in full: 4317 passed, 0 failed.

The first host was 2 × L20 with Python 3.10 and cuda-bindings 12.9.8, so the three contract assertions (exception reporting through sys.unraisablehook, caller-owned int user_data delivered unchanged, two devices keeping their own callback parameters) hold on both a 2-GPU and an 8-GPU machine.

Still unchanged and still waiting on the split answer: the stream-side matrix (C01–C05, C09) remains BLOCKED: dependency until the ordinary-stream callback entry from #2058 exists, and capture/replay stays out of scope.