Add isolated sessions: a fresh Python process per session, from a prepared template

CHANGELOG.md

@@ -1,6 +1,41 @@
 # Changelog
 
-## 5.0.1 (2026-09-22)
+## 5.1.0 (unreleased)
+
+### Added
+
+- **Isolated sessions** - `py_session:template/1` prepares a Python
+  environment once (interpreter, `paths`, `imports`, `preload`, `env`,
+  `hash_seed`, `rlimits`) and `py_session:new/1` gives a fresh child process
+  per session that starts from it and shares no state with any other
+  session: module globals, `sys.modules`, environment, threads, working
+  directory. By default a session is forked from a zygote that already ran
+  the imports and preload, so it is ready in a few milliseconds instead of
+  the ~50 ms of a new interpreter plus its imports; `start => spawn` starts
+  a new interpreter per session (optionally from a `warm` pool) for code
+  that cannot be forked. A session is an isolated context: calls, callbacks,
+  calls back into the same session, interrupts, loops and `pass_fd` work
+  unchanged. `close/1` kills it; a session whose process dies answers with
+  the reason until closed and is never restarted. `run/5` runs one call in a
+  new session; `refresh/1` rebuilds the template after a deploy; `info/1`
+  reports zygotes, live sessions and forks. `start => reimport` runs each
+  call in a fresh module dictionary on a worker or owngil context instead,
+  the way Temporal's Python SDK isolates a workflow run: the function's
+  module is imported again per run, and the standard library, `imports` and
+  `passthrough` modules are shared. See `docs/sessions.md`.
+- `clear_env` and `hash_seed` options for isolated contexts: the child sees
+  only the variables named in `env`, and every child built with the same
+  seed hashes strings and orders sets the same way.
+
+### Fixed
+
+- On an owngil context, a Python function that called `erlang.call`, where
+  the Erlang callback called the same context again, hung until the request
+  timeout. The context thread waited for the callback on its pipe while the
+  nested call sat in its queue. It now waits inline and serves the nested
+  call, as worker contexts do since 5.0.1.
+
+
 
 ### Fixed
 

c_src/py_callback.c

@@ -1657,9 +1657,14 @@ static PyObject *erlang_call_impl(PyObject *self, PyObject *args) {
     bool has_context_suspension = (tl_current_context != NULL && tl_allow_suspension &&
                                    !loop_running);
     bool has_context_handler = (tl_current_context != NULL && tl_current_context->has_callback_handler);
+    /* An owngil context also has a callback handler (for its other
+     * threads), but a request with a caller must wait inline too: on the
+     * handler pipe a callback calling back into this context would queue
+     * behind the request that waits for it. */
     bool has_context_inline = (tl_current_context != NULL && !tl_allow_suspension &&
                                tl_current_context->has_current_caller &&
-                               !has_context_handler && !loop_running);
+                               (!has_context_handler || tl_current_context->is_subinterp) &&
+                               !loop_running);
 
     if (has_context_inline) {
         Py_ssize_t nargs = PyTuple_Size(args);
@@ -3794,7 +3799,7 @@ static int create_erlang_module(void) {
         PyDict_SetItemString(log_globals, "__builtins__", builtins);
 
         /* Import erlang module into globals so the code can reference it */
-        PyObject *sys_modules = PySys_GetObject("modules");
+        PyObject *sys_modules = PyImport_GetModuleDict();  /* the interpreter's table, a dict even when sys.modules is replaced */
         if (sys_modules != NULL) {
             PyObject *erlang_mod = PyDict_GetItemString(sys_modules, "erlang");
             if (erlang_mod != NULL) {
@@ -3896,7 +3901,7 @@ static int create_erlang_module(void) {
         PyDict_SetItemString(ext_globals, "__builtins__", builtins);
 
         /* Import erlang module into globals so the code can reference it */
-        PyObject *sys_modules = PySys_GetObject("modules");
+        PyObject *sys_modules = PyImport_GetModuleDict();  /* the interpreter's table, a dict even when sys.modules is replaced */
         if (sys_modules != NULL) {
             PyObject *erlang_mod = PyDict_GetItemString(sys_modules, "erlang");
             if (erlang_mod != NULL) {
@@ -3953,7 +3958,7 @@ static int create_erlang_module(void) {
         PyDict_SetItemString(atom_globals, "__builtins__", builtins);
 
         /* Import erlang module into globals so the code can reference it */
-        PyObject *sys_modules = PySys_GetObject("modules");
+        PyObject *sys_modules = PyImport_GetModuleDict();  /* the interpreter's table, a dict even when sys.modules is replaced */
         if (sys_modules != NULL) {
             PyObject *erlang_mod = PyDict_GetItemString(sys_modules, "erlang");
             if (erlang_mod != NULL) {
@@ -4043,7 +4048,7 @@ static int create_erlang_module(void) {
         PyDict_SetItemString(sd_globals, "__builtins__", builtins);
 
         /* Import erlang module into globals so the code can reference it */
-        PyObject *sys_modules = PySys_GetObject("modules");
+        PyObject *sys_modules = PyImport_GetModuleDict();  /* the interpreter's table, a dict even when sys.modules is replaced */
         if (sys_modules != NULL) {
             PyObject *erlang_mod = PyDict_GetItemString(sys_modules, "erlang");
             if (erlang_mod != NULL) {

c_src/py_nif.c

@@ -3166,6 +3166,13 @@ static void *ctx_thread_main_owngil(void *arg) {
         ctx->request_term = req->request_data;
         ctx->reactor_buffer_ptr = req->reactor_buffer_ptr;
         ctx->local_env_ptr = req->local_env_ptr;
+        /* The caller serves erlang.call made by this request: the call
+         * waits inline and serves nested requests (ctx_call_erlang_inline),
+         * as on a worker context thread */
+        ctx->has_current_caller = req->async_mode;
+        if (req->async_mode) {
+            ctx->current_caller = req->caller_pid;
+        }
         ctx->response_ok = false;
         ctx->response_term = 0;
 
@@ -3195,6 +3202,7 @@ static void *ctx_thread_main_owngil(void *arg) {
         ctx->request_term = 0;
         ctx->reactor_buffer_ptr = NULL;
         ctx->local_env_ptr = NULL;
+        ctx->has_current_caller = false;
 
         /* Deliver result - async (message to caller) or blocking (condvar) */
         if (req->async_mode) {

docs/code-map.md

@@ -16,6 +16,9 @@ exercised by suites). Guides are in `docs/`, suites in `test/`. Start with
 | `py_context` | The API every mode answers (`call/eval/exec`, `interrupt`, `kill`, loops, `pass_fd`), the reply protocol and the pid to NIF reference table; `init/4` hands the process to `py_context_embedded` or `py_isolated` | live | context-affinity, workers, interrupts | `py_context_SUITE`, `py_context_process_SUITE`, `py_interrupt_SUITE`, `py_worker_loop_SUITE` |
 | `py_context_embedded` | Process body for `worker` and `owngil` mode: the receive loop, callbacks (suspension and pipe), worker loops | live | architecture, state-machines | same |
 | `py_isolated` | `gen_statem` driving a child process over the socket; restart policy | live | isolated | `py_isolated_*_SUITE` |
+| `py_child` | Helpers shared by the processes that drive a Python child: executable lookup, socket listen/accept, frames, port env, rlimit flags | live | isolated | `py_isolated_*_SUITE` |
+| `py_session` | Isolated sessions: a fresh child per session from a template (`template/1`, `new/1`, `close/1`, `run/4`, `refresh/1`), or a fresh module dictionary per run (`start => reimport`) | live | sessions | `py_session_SUITE` |
+| `py_session_template`, `py_session_sup` | A template: zygotes that fork sessions (`priv/py_zygote.py`) or warm spawned sessions; exit reports to the session contexts | live | sessions | `py_session_SUITE` |
 | `py_context_router` | Pools and scheduler-affinity routing | live | pools, context-affinity | `py_context_router_SUITE`, `py_pool_SUITE` |
 | `py_context_sup`, `py_context_init` | Supervisor of contexts; starts the default pool at boot | live | pools | (through the above) |
 | `py_nif` | Erlang stubs and docs for every NIF | live | api-reference | all |
@@ -79,8 +82,10 @@ loop, channels and servers.
 | `_erlang_impl/_mode.py` | Detects how Python is running (embedded, free-threaded, child) | all |
 | `_erlang_impl/_etf.py` | Pure-Python ETF codec with the `py_convert.c` mapping | isolated child |
 | `_erlang_impl/_isolated.py` | Child runtime: socket frames, reader thread, re-entrant main loop, interrupt signal, asyncio loop, the `erlang` shim | isolated child |
+| `_erlang_impl/_reimport.py` | Re-import runs for `py_session` templates with `start => reimport`: a fresh module dictionary per run, swapped per thread | embedded (worker, owngil) |
 | `_erlang_impl/_shm.py` | `SharedMemory` and `SharedBuffer` wrappers over mmap | all |
 | `py_isolated_child.py` | Child launcher: rlimits, parent-death signal, cgroup join, connect | isolated child |
+| `py_zygote.py` | Session template zygote: imports and preload once, forks one child per session, reports exits | session zygote |
 | `test_erlang_loop.py`, `test_async_task.py`, `test_channel_ref.py`, `tests/` | Python-side tests of the loop, tasks and channels | test |
 
 ## Tests (`test/`)

docs/decisions/0009-isolated-sessions.md

@@ -0,0 +1,61 @@
+# 0009: Sessions fork from a prepared zygote
+
+Since 5.1.0. Code: `src/py_session.erl`, `src/py_session_template.erl`,
+`priv/py_zygote.py`, `src/py_isolated.erl` (session origin), `src/py_child.erl`.
+
+## Situation
+
+Callers such as a durable-execution engine need each run of a Python
+function to start from the same state and to leave nothing behind for the
+next run. An isolated context keeps its interpreter between calls, and a
+new one costs a cold interpreter start plus every import (about 50 ms
+before the first import). Temporal's Python SDK isolates a workflow run by
+re-importing its module in a fresh `sys.modules` inside a shared process;
+Restate's does not isolate invocations at all and relies on the platform
+(a container, a Lambda microVM). Neither gives a fresh process per run.
+
+## Decision
+
+A template prepares an interpreter once, in a zygote: a single-threaded
+child that runs the imports and preload, builds the child runtime (the
+`_isolated.Runtime` and the `erlang` module) without a socket, then forks
+one child per session. The forked child connects that runtime to its own
+socket and continues as a normal isolated child, so a session is an
+isolated context (`py_isolated` with `session => true`). The zygote reports
+each child's exit on its control socket; the template forwards it to the
+session's context. `start => spawn` starts a normal child per session
+instead, with an optional warm pool, for templates that cannot be forked.
+
+A session is never restarted: after its child dies it answers every
+request with the reason until it is closed. It runs in its own scratch
+directory, its stdio is detached from the zygote's port, and it sees only
+the template's environment.
+
+`start => reimport` is the light variant for worker and owngil contexts:
+no process per session, the function's module imported again in a fresh
+module dictionary per run, swapped per thread as Temporal's workflow
+sandbox does. It isolates module state only and is offered as that.
+
+Not chosen: a subinterpreter per session (about 13 ms, shares the process
+environment, working directory, hash seed and C-extension state, and PyO3
+extensions refuse to load), CRIU (Linux only, needs privileges and PID
+namespaces), and Wasm images (no native C extensions).
+
+## Consequences
+
+- A session costs a fork and a connect (a few milliseconds) instead of an
+  interpreter start and the imports.
+- The zygote must stay single threaded: a template whose imports start a
+  thread is refused. On macOS, modules that load Objective-C cannot be
+  forked safely; `start => spawn` is the way out.
+- All sessions of a template share its hash seed and its prepared state;
+  per-session randomness relies on Python's at-fork reseeding.
+- `erlang` functions fail during preload: the runtime is not connected
+  until a session exists.
+- A zygote that dies is rebuilt; its orphaned sessions are watched by pid
+  (`py_isolated` probes `kill(pid, 0)`), since nobody reports their exit.
+- A re-import template replaces `sys.modules` and `builtins.__import__` in
+  its interpreter with per-thread stand-ins. C code that reads the
+  interpreter's own module table (the C `pickle`) does not see a run's
+  modules; the NIF's own lookups use `PyImport_GetModuleDict()` for that
+  reason.

docs/decisions/overview.md

@@ -16,3 +16,4 @@ what was decided, what it costs, and where the code is.
 | [0006](0006-shared-memory-over-iommap.md) | Bulk data through iommap regions, handles as plain tuples | 5.0.0 |
 | [0007](0007-remove-legacy-execution-paths.md) | One execution path per mode; the legacy API is removed | 5.0.0 |
 | [0008](0008-pipe-io-rules.md) | Pipe I/O is non-blocking, deadlined and waited with poll | 3.1.0, 5.0.0 |
+| [0009](0009-isolated-sessions.md) | Sessions fork from a prepared zygote | 5.1.0 |

docs/isolated.md

@@ -24,6 +24,9 @@ memory bound. The public API is the one you already use with `worker` and
 | Startup | microseconds | milliseconds | ~40 ms |
 | Zero-copy `py_buffer`, channels, `erlang.schedule`, object refs | yes | yes | no (see Limits) |
 
+To give every request or job its own fresh child, started from prepared
+imports in a few milliseconds, use [Isolated Sessions](sessions.md).
+
 ## Start a context
 
 ```erlang
@@ -50,6 +53,8 @@ Options of `py_context:new/1` specific to this mode:
 | `rlimits` | `#{}` | `#{as => Bytes, cpu => Seconds, nofile => N}`, applied with `setrlimit` before any user code |
 | `cgroup` | none | Path of a cgroup v2 directory the child joins (limits written by you: `memory.max`, `cpu.max`, `pids.max`) |
 | `env` | `#{}` | Extra environment variables for the child |
+| `clear_env` | `false` | When `true` the child inherits nothing from the VM's environment: it sees only `env` |
+| `hash_seed` | `random` | `PYTHONHASHSEED` for the child (0 to 4294967295): the same seed gives the same `set` and `dict`-of-`str` iteration order in every child |
 | `paths` | `[]` | Extra `sys.path` entries (registered `py_import` paths and imports are applied too) |
 | `preload` | none | Code run once in the child before anything else |
 | `kill_after` | `1000` | Milliseconds between a soft interrupt and `SIGKILL` |

docs/security.md

@@ -158,7 +158,9 @@ child process:
 ```
 
 A crash kills only the child, `py_context:kill/1` is total, and rlimits or
-cgroups bound resources. See [Isolated Contexts](isolated.md).
+cgroups bound resources. See [Isolated Contexts](isolated.md). When each
+request must also start from a clean state, with nothing left by the
+previous one, give it its own session: see [Isolated Sessions](sessions.md).
 
 That is a boundary against Python *failing*, not against Python *reaching*.
 The child runs as the same user as the node, so it can read and write