deer-flow/backend/packages/harness/deerflow/sandbox/AGENTS.md

Sandbox System (packages/harness/deerflow/sandbox/)

Interface: Abstract Sandbox with execute_command(command, env=None), read_file, write_file, list_dir, glob, and grep. grep accepts either one text file or a directory tree. The optional env injects per-call environment variables (request-scoped secrets — see Request-Scoped Secrets below); LocalSandbox merges it via subprocess.run(env=...) and AioSandbox routes env-bearing commands through the bash.exec(env=...) API on a fresh session. Provider Pattern: SandboxProvider with acquire, acquire_async, get, release lifecycle. Async agent/tool paths call async sandbox lifecycle hooks so Docker sandbox creation, discovery, cross-process locking, readiness polling, and release stay off the event loop. Environment policy (sandbox/env_policy.py): execute_command no longer inherits the full os.environ. build_sandbox_env() scrubs secret-looking names (*KEY*/*SECRET*/*TOKEN*/*PASS*/*CREDENTIAL*) from the inherited environment before layering injected request secrets on top, so platform credentials (e.g. OPENAI_API_KEY) never leak into skill subprocesses. Benign vars (PATH, HOME, LANG, VIRTUAL_ENV, ...) are preserved. Implementations:

• LocalSandboxProvider - Local filesystem execution. acquire(thread_id) returns a per-thread LocalSandbox (id local:{thread_id}) whose path_mappings resolve /mnt/user-data/{workspace,uploads,outputs} and /mnt/acp-workspace to that thread's host directories, so the public Sandbox API honours the /mnt/user-data contract uniformly with AIO. acquire() / acquire(None) keeps the legacy generic singleton (id local) for callers without a thread context. Per-thread sandboxes are held in an LRU cache (default 256 entries) guarded by a threading.Lock. Public, custom, legacy, and managed integration skill mappings point at stable enabled-only projection roots rather than raw skill directories.
• AioSandboxProvider (packages/harness/deerflow/community/) - Docker-based isolation. Active-cache and warm-pool entries are checked with the backend during acquire/reuse; definitively dead containers are dropped from all in-process maps so the thread can discover or create a fresh sandbox instead of reusing a stale client. Backend health-check failures are treated as unknown, not dead; local discovery likewise treats an unverifiable container as not adoptable and falls through to create rather than failing acquire. get() remains an in-memory lookup for event-loop-safe tool paths — it never touches the ownership store (that would be blocking IO on the event loop); ownership is published on acquire/reclaim and refreshed off the event loop by the dedicated renewal thread (_renew_owned_leases). uses_thread_data_mounts defaults to backend detection (LocalContainerBackend=True, remote/provisioner backends=False), while the optional sandbox.thread_data_mounts boolean takes precedence for deployments that guarantee the Gateway and sandbox share the same thread user-data directories. Setting it true skips upload-time sandbox acquire/sync; a false positive leaves uploads unavailable to the sandbox. Local-container and hostPath-provisioner mounts use the same stable skill projection roots; PVC-backed skills remain governed by the operator-supplied PVC layout until PVC materialization is implemented. Readiness probes and agent_sandbox clients classify loopback/private IPs, single-label cluster hosts, and Docker/Podman internal hostnames as direct control-plane destinations and set trust_env=False; external FQDNs and public IPs retain environment proxy support.
• E2BSandboxProvider (packages/harness/deerflow/community/e2b_sandbox/) provides E2B remote isolation. New sandboxes receive a one-shot upload from the enabled-only public, custom, legacy, and managed integration projections. Existing E2B VMs keep their creation-time snapshot because E2B has no shared host mount. Acquire and release share a per-user and thread lock. The provider lock does not cover remote IO. burst_limit adds capacity only for the burst policy. The wait policy fails the turn after acquire_timeout. The runtime does not retry the turn automatically. E2B acquisition uses a bounded executor. Waiting calls do not consume the default asyncio executor. The reject policy can evict one warm VM before it returns an error. With memory ownership, replicas limits one Gateway process. Redis ownership shares one <ownership.key_prefix>:e2b-capacity Hash, making the limit (plus a bounded burst) deployment-wide. Lua atomically manages VM and in-flight-create entries; missing or unavailable state fails closed. E2B reservation metadata repairs interrupted creates. Inventory replacement is revision-CAS guarded, and incomplete inventories never remove entries; complete omissions get a grace period. Uncertain cleanup keeps a tombstone slot. Shutdown tracks owned remote operation IDs. Discovery can find a VM from another Gateway. Shutdown closes an unowned discovery client without destroying its VM. Release ends its transition count when the VM enters the warm pool. Local client cleanup does not consume a second slot. A create that returns after shutdown retries one failed kill through a new client. An unconfirmed remote ID stays tracked. reset() uses full shutdown semantics. It destroys tracked active and warm VMs. It wakes capacity waiters. Callers cannot reuse the old provider instance. A background startup pass and periodic reconciliation list provider-tagged remote sandboxes within page/item/time budgets, probe every candidate until a healthy canonical sandbox is found, adopt only through the shared ownership store, and reap duplicates/orphans only after their configured grace/TTL and an atomic del: claim. Lease renewal is independent of reconciliation. Failed canonical adoption clears both its capacity reservation and acquire-intent marker even if a peer takes ownership between the initial claim and bootstrap cleanup. Shutdown kills only IDs whose leases are owned by this provider instance; peer-owned clients are merely closed.
  • Cross-instance ownership store (aio_sandbox/ownership/, #4206): gateway instances sharing a container backend coordinate container ownership through a pluggable lease store, selected by sandbox.ownership.type (memory | redis) and resolved like stream_bridge (factory.py, lazy per-branch import, redis optional extra, DEER_FLOW_SANDBOX_OWNERSHIP_REDIS_URL env escape hatch; a set DEER_FLOW_STREAM_BRIDGE_REDIS_URL implies a multi-instance deployment and infers redis). memory is single-instance only and declares supports_cross_process = False.
    • A lease answers "who reaps this container", not "who may use it". That splits the interface in two: take() transfers ownership on the acquire path (a container is deterministic per user/thread, so consecutive turns legitimately land on different instances — a conditional claim there would strand the thread until the previous lease expired), while claim() succeeds only if the container is unowned or already ours and gates every adopt/reap path. release() never clears a peer's lease.
    • A lease carries a state, and that is what makes the destroy window safe. own: = responsible for this container; del: = tearing it down (claim(..., for_destroy=True)). take() is refused against a del: lease, so a container cannot be re-acquired between a destroy path's claim and its container stop. Without the two states an unconditional take() would silently overwrite the destroyer's claim and the peer's stop would land on a container the new owner had already handed to an agent — i.e. #4206 again. That pairing is what replaced the previous same-host flock guard, which is gone; Redis makes the scope genuinely multi-instance instead of same-host. A destroyer that dies mid-stop leaves a del: marker that lapses with the TTL. On the acquire path a refused take raises SandboxBeingDestroyedError: the reuse/reclaim paths drop the container and cold-start, and the discover path propagates (falling through to create would collide with the not-yet-removed container name).
    • The del: state has to be held for the stop, not just written before it. The two states alone do not make flock redundant — a held lock cannot expire, whereas a lease can, and claim(..., for_destroy=True) writes the marker with the ordinary lease TTL. Nothing else refreshes it: renew() extends only own: and deliberately reports a teardown as LOST, and the destroy paths drop the sandbox from the maps _renew_owned_leases iterates. So a container stop that outlived the TTL let the marker lapse, a peer's take() succeeded against the still-running container, and the stop then landed on the turn that had just been handed it — the exact window del: exists to close, reopened by its own expiry. _held_teardown_lease wraps every del:-marked stop — _destroy_warm_entry, destroy(), and _drop_unhealthy_sandbox — and re-claims the marker every renewal_interval_seconds until the stop returns. _drop_unhealthy_sandbox needs it most: it untracks before claiming (under its expected_info TOCTOU guard), so _renew_owned_leases cannot see the id either. The final release is the heartbeat's own last act, not the caller's — a refresh claim still in flight when the context exits (the store's socket timeout bounds it, but it can be mid-round-trip) would otherwise land after a caller-side release and rewrite del: on a container whose stop already completed, stranding a fresh take() (or rolling back a fresh create) until the TTL. Releasing from inside the heartbeat, after its loop stops, sequences the release strictly after the last refresh, so no claim can follow it; the context join is bounded and, on a genuine wedge, defers the release to that thread rather than clearing the marker itself. This covers a failed stop too (the container is probably still up, and a marker left behind refuses its own thread's take() until the TTL lapses); destroy() still lets the error propagate out of the with — shutdown() logs per sandbox off it. RedisOwnershipStore sets a socket_timeout so no store round trip — and so no heartbeat refresh — can block unbounded, keeping that deferred release finite. This needs no abnormal backend: the schema bounds only renewal_interval_seconds (> 0) and ttl_multiplier (>= 2), so a legal config puts the TTL below a normal container stop. LocalContainerBackend._stop_container now passes a timeout to subprocess.run (_STOP_TIMEOUT_SECONDS) so a wedged daemon cannot block unbounded — that bounds the residual window independently of the ownership layer, for the case where the del: marker lapses mid-stop (a store outage longer than the TTL) and the stop then lands on a container a peer has been handed. A timed-out stop propagates rather than being swallowed like a CalledProcessError: the container is probably still running, so reporting a clean stop would drop the warm entry and leak it. The TTL stays finite on purpose — the heartbeat dies with the process, so a destroyer that crashes mid-stop still releases the container one TTL later instead of marking it undestroyable forever. Raising a separate teardown TTL instead would only be sufficient if it were bounded above every backend's real stop deadline.
    • Fail-closed both directions. Establishment: a sandbox whose ownership cannot be published is never handed out (a just-created container is destroyed rather than leaked as an adoptable orphan) — acquiring raises OwnershipBackendError, matching the stream bridge's fail-hard v1 policy. Reaping: a store that cannot answer is treated as peer-owned, so an outage never turns live peer containers into orphans. Renewal is the deliberate exception: an unanswerable store there means unknown, not lost, so _refresh_ownership keeps the sandbox and retries — failing closed on that path would evict every live sandbox on every instance the moment the store blinked. The TTL still bounds how long a genuinely dead owner holds a lease. Both paths that stop a container they still track — destroy() and _destroy_warm_entry — claim before untracking, so a refused claim cannot leave a container running and untracked. (_drop_unhealthy_sandbox untracks first, under its expected_info TOCTOU guard, then claims before the stop; a refused claim there leaves the container to the next reconcile, which re-adopts it after the grace.)
    • A lease excludes peers, never ourselves — same-process exclusion is the provider's job. claim() and take() both succeed against this instance's own own: lease by design (that is what lets a destroy path claim what it already owns), so del: says nothing to this process's other threads. Every reaper — idle checker, renewal, warm eviction, unhealthy drop — decides outside self._lock, because a store round trip must not be held under the lock that guards every acquire; so each one acts on a decision its own acquire path may already have invalidated. Two guards cover the two directions, and both live in AioSandboxProvider, not the store:
      • Reaping (_reserve_local_teardown / _local_teardown): the reaper marks the id, and every promote path — _reuse_in_process_sandbox, _reclaim_warm_pool_sandbox, _register_discovered_sandbox — refuses a marked id exactly as it refuses a peer's del: (drop and cold-start). The "is this still reapable?" check runs in the same critical section as the mark, passed down as a still_reapable predicate rather than run by the caller beforehand: checking first and marking second is the window, not a narrower version of it. This matters most where the entry deliberately stays visible during the stop — both warm reapers defer their pop so a refused claim cannot lose the container — and where the maps are cleared first (_drop_unhealthy_sandbox), which leaves backend discovery as the open path. On main the mixin's _evict_oldest_warm / _reap_expired_warm popped under the lock, so the deferred pop is what made this reachable.
      • Forgetting (_acquire_epoch): when renew() reports LOST the peer legitimately wins, so here the promote is the thing to detect. _publish_ownership bumps a per-id acquire epoch; _renew_owned_leases and release() snapshot it before the round trip and hand it to _forget_lost_sandbox, which skips the pop if it moved. Object identity is not enough: the reuse path re-publishes ownership while handing out the same tracked AioSandbox, so an identity check sees nothing and the pop closes a client mid-turn.
      • A guard must become visible no later than the transition it guards. The epoch cannot satisfy that for take(): the takeover is durable before take() returns (redis has committed the SET while the reply is in flight), and the epoch can only be written afterwards, so a renewal holding an older LOST walks through the gap, drops the maps, and closes the client the acquire is about to hand back — acquire then returns an id whose get() is None. _publish_ownership therefore publishes an intent mark (_acquire_inflight) under _lock before the round trip; the epoch covers the other half, "an acquire completed since you decided". _forget_lost_sandbox honours the intent mark unconditionally, not only when an epoch is supplied — "no epoch" must not read as "no guard".
      • A reservation must cover the removal, not just the stop. _destroy_warm_entry pops the warm entry itself, inside the reservation. Releasing the reservation when the stop returns and letting the caller pop afterwards leaves a gap where the container is stopped, the entry is still in _warm_pool, and nothing marks it — a reclaim there hands out a dead container. The pop stays deferred relative to the stop (a refused or failed stop keeps the entry), just no longer relative to the reservation.
      • A check taken before a round trip must be retaken after it. _reuse_in_process_sandbox re-verifies both its map entry and the local teardown reservation, _reclaim_warm_pool_sandbox re-checks the reservation, and _register_discovered_sandbox re-checks before installing its client, all after publishing ownership. Before the intent mark is set a renewal's LOST is both current and correct, so the forget can legitimately remove the entry the acquire decided to hand out; independently, a local reaper can reserve an id while reuse is outside _lock for its health/store calls and deliberately leaves the map entry present until its destroy claim succeeds. Falling through re-discovers or cold-starts instead of returning/installing a client for either stale decision. The pre-round-trip checks remain as early-outs that skip backend and store work on an already-doomed entry.
      • Adoption is a promote too. _reconcile_orphans honours the reservation: a container being torn down is untracked and still running, which is exactly the shape that loop adopts, and neither the claim (ours) nor the recovery grace (skipped entirely on memory, where supports_cross_process is False) excludes it.
      • Active and warm are exclusive, and only a promote can violate it. Both register paths pop _warm_pool inside the same locked section that inserts into _sandboxes: a warm entry for an id is stale the moment that id becomes active, and leaving it gives the container two reapers — _reap_expired_warm judges it by the warm timestamp and never consults _last_activity, so it stops a container an agent is using while _sandboxes still hands out its client. Reachable because reconciliation adopts into the warm pool inside the register's publish → track window, and on memory it adopts on sight (_adoptable_after_grace short-circuits when supports_cross_process is False, so an id carrying this process's own lease reads as adoptable). On main the track was a single locked insert with nothing before it, so the window did not exist. A non-destroy claim() is the one case the store does police against its own owner: it refuses to overwrite our own del:, because the stop it marks is already in flight and downgrading the marker would let a take() hand out a container about to die. Enforced in both backends (Lua and Python) so they cannot drift.
    • Renewal is independent of idle_timeout (_start_lease_renewal, own daemon thread; TTL = renewal_interval_seconds × ttl_multiplier). Renewal used to ride on the idle checker, which __init__ only starts when idle_timeout > 0 — so idle_timeout: 0 ("keep warm VMs until shutdown", a documented config) let every lease lapse. Liveness and reaping must not share a switch. Renewal covers warm entries as well as active ones; losing a lease drops the sandbox from this instance's maps without touching the container (_forget_lost_sandbox) — destroying it there would be the very cross-instance kill this store prevents.
      • A warm teardown is the local exception to that forget rule: _destroy_warm_entry deliberately keeps the entry in _warm_pool until the backend stop succeeds, while its own del: marker makes ordinary renew() report LOST. _forget_lost_sandbox therefore honours _local_teardown; otherwise the renewal thread can pop the retained entry mid-stop and a failed stop leaves a running container untracked.
    • renew() distinguishes lapsed from lost (RenewOutcome), and the two must not be collapsed. LAPSED means the lease is simply absent — nobody took it — so _refresh_ownership re-establishes it; LOST means a peer holds it and it is never re-taken. Treating an absent lease as lost meant a Redis restart without persistence (every key gone) evicted every in-flight sandbox on every instance at once.
      • Renewal's fail-open rule covers both store round trips. If renew() returns LAPSED but the follow-up claim() cannot answer, ownership is still unknown rather than lost, so the provider keeps the sandbox and retries. The ordinary _claim_ownership helper remains fail-closed for adopt/reap callers and is intentionally not used for this re-claim.
    • Teardown join budget covers refresh plus release. Redis bounds each ownership operation at five seconds, and context exit can catch the heartbeat in one final refresh before its finally performs the final release. _TEARDOWN_JOIN_TIMEOUT_SECONDS is therefore 12 seconds — greater than both sequential operation bounds — so a normal pair of socket timeouts does not emit the deferred-release warning; a still-running heartbeat continues to own the release safely.
    • An absent lease means the same thing on both paths, and reconciliation must say so too. The LAPSED rule above only covers an owner renewing its own lease; on its own it does not make state loss safe, because reconciliation reads the same absent key as "orphan, adopt". After a Redis flush (restart without persistence, or eviction under maxmemory) every owner is alive and merely pre-renewal-tick, so whichever instance reconciles first would adopt every live container, each real owner's next renewal would report LOST, and it would drop a sandbox mid-turn for the adopter to idle-destroy — #4206 through the back door. _adoptable_after_grace closes it: an untracked container must be seen unowned (owner(), a read-only peek — the atomic claim() is still what actually gates adoption) across a full lease TTL before it can be adopted, tracked per container in _unowned_since. That rebuilds the delay the flush erased — a live owner republishes within one renewal interval, shorter than the TTL by construction (ttl_multiplier >= 2) — while a genuinely crashed owner never republishes, so its containers are still adopted one grace later rather than leaking. A republished lease resets the grace; a pausing-only timer would still expire over a live owner's lease. The grace is skipped when supports_cross_process is False: no peer can hold a lease such a store would show us, so single-instance deployments keep instant orphan cleanup, and a grace could not help a multi-worker gateway on memory anyway (peers are invisible to each other's leases with or without it).
    • The memory store is single-instance only and says so via supports_cross_process = False; the provider logs a warning at startup when the configured store cannot see peers. A multi-worker gateway on memory has no cross-process coordination at all — same contract as stream_bridge's memory backend. This is why the redis inference matters: it reads app_config.stream_bridge and the env var, in the same order the bridge's own resolver does, so any deployment already pointing the bridge at Redis (i.e. every multi-instance one) gets a redis ownership store without extra config.
    • get() stays a pure in-memory lookup and must never call the store (that is blocking filesystem/network IO on the event loop); anchored by tests/blocking_io/test_aio_sandbox_get.py, which injects a deliberately-blocking probe store so the anchor keeps its teeth regardless of the configured backend. Tests: tests/test_sandbox_ownership_store.py (store contract, defined once for both backends — the redis tier is @pytest.mark.integration, uses DEER_FLOW_TEST_REDIS_URL when set, and otherwise self-skips without a reachable Redis. Backend CI provisions Redis, so the merge gate executes the real Lua tier; there is no fake-redis tier because a fake would not execute the Lua exclusions) and tests/test_sandbox_orphan_reconciliation.py (provider behaviour, two providers sharing one store).
• BoxliteProvider (packages/harness/deerflow/community/boxlite/) - BoxLite micro-VM isolation. The boxlite runtime is optional (deerflow-harness[boxlite]) and lazy-imported only when this provider is selected. The provider owns one private asyncio event loop on a daemon thread because BoxLite handles are loop-affine; sync Sandbox calls marshal onto that loop with run_coroutine_threadsafe. Boxes are named deterministically from user_id:thread_id, released into an in-process warm pool after each agent turn, and reclaimed only by the same user/thread. Warm-pool health checks use a short explicit timeout and forward that timeout through both BoxLite exec(timeout=...) and the private-loop .result(timeout) bridge so a hung VM cannot pin the per-thread acquire lock indefinitely. sandbox.replicas caps active + warm VMs per gateway process; if capacity is exhausted, only warm-pool VMs are evicted. sandbox.idle_timeout stops idle warm VMs after the configured seconds. reset() is intentionally a lightweight registry clear for reset_sandbox_provider() and does not close boxes, stop the idle reaper, or close the private loop; full teardown remains shutdown().
• TenkiSandboxProvider (packages/harness/deerflow/community/tenki/) - Tenki cloud microVM isolation. The tenki-sandbox SDK is optional (deerflow-harness[tenki]) and lazy-imported (_import_client) only when this provider is selected. Unlike Boxlite, the SDK is synchronous, so the adapter calls it directly with no event-loop bridge. File transport uses Tenki's native sandbox.fs API (read_text/read_stream/write_stream/mkdir/stat) — binary-safe and streaming, no base64/shell hop; only directory/content search (list_dir/glob/grep) shells out to busybox-portable find/grep, parsed with the shared deerflow.sandbox.search helpers like community/e2b_sandbox. Sandboxes run as the unprivileged tenki user, so DeerFlow's /mnt/user-data prefix is remapped under a writable HOME (_resolve_path) and best-effort sudo-symlinked at bootstrap. Boxes are named deterministically from sha256(user_id:thread_id)[:16] (64-bit, matching E2B; the warm pool is keyed by this id alone with no full-seed fallback), released into an in-process warm pool, and reclaimed only by the same user/thread after a liveness check. A terminal session error (named SDK errors plus builtin ConnectionError/BrokenPipeError/EOFError) routes through _invalidate_sandbox to evict the dead microVM. Cross-process orphan reconciliation is a follow-up (single-process warm pool today).

Shared warm-pool lifecycle: community sandbox providers that keep released sandboxes alive for fast reuse share deerflow.community.warm_pool_lifecycle.WarmPoolLifecycleMixin. The mixin owns the common DEFAULT_IDLE_TIMEOUT=600, IDLE_CHECK_INTERVAL=60, DEFAULT_REPLICAS=3, idle-checker loop, warm-pool expiry, oldest-warm eviction, replica counting, and soft-cap logging. Providers remain responsible for their own active registries, creation/discovery, health checks, and destroy hook (_destroy_warm_entry): AIO destroys SandboxInfo through its backend; Boxlite closes loop-affine BoxliteBox handles; Tenki closes the microVM session (TenkiSandbox.close, which terminates the remote sandbox). AIO keeps active-idle cleanup outside the mixin and delegates only warm-pool expiry to the shared helper.

Virtual Path System:

• Agent sees: /mnt/user-data/{workspace,uploads,outputs}, /mnt/skills
• Physical: backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/...; raw skills stay under deer-flow/skills/ and managed integration storage, while sandboxes read backend/.deer-flow/skills_view/public/ and backend/.deer-flow/users/{user_id}/skills_view/{custom,legacy,integrations}/
• Translation: LocalSandboxProvider builds per-thread PathMappings at acquire time. Sandbox-backed readers resolve only /mnt/user-data/... in the tool layer; skills, ACP workspaces, and configured custom mounts stay virtual so the provider mount table remains the single source of acquire-time identity and visibility. Full reads, ranged reads, and read-before-write hashing share this path. tools.py keeps replace_virtual_path() / replace_virtual_paths_in_command() as a defense-in-depth layer (and for path validation). AIO has the directories volume-mounted at the same virtual paths inside its container, so both implementations accept /mnt/user-data/... natively.
• Detection: is_local_sandbox() accepts both sandbox_id == "local" (legacy / no-thread) and sandbox_id.startswith("local:") (per-thread)

Sandbox Tools (in packages/harness/deerflow/sandbox/tools.py):

• bash - Execute commands with path translation and error handling. For LocalSandbox (host bash), POSIX output is captured through bounded pipe-drain threads and stdin is /dev/null, so a backgrounded long-lived process (server &) returns immediately instead of blocking the turn on an inherited pipe, while unredirected background output is drained without growing anonymous temp files. Commands that read stdin get immediate EOF. The command runs in its own process group with a wall-clock timeout (sandbox.bash_command_timeout, default 600s); on timeout the whole group is killed and the agent gets a notice telling it to background long-lived processes. The bash tool description itself also instructs the model to background long-lived processes (e.g. servers) up front so it doesn't waste the turn waiting on a foreground server. See LocalSandbox.execute_command / _run_posix_command and bash_tool's docstring.
• ls - Directory listing (tree format, max 2 levels)
• glob - Find files or directories below a root directory with bounded results
• grep - Search one text file or recursively search a directory, with optional glob filtering and bounded line-level results
• read_file - Read file contents with optional line range
• write_file - Write/append to files, creates directories; overwrites by default and exposes the append argument in the model-facing schema for end-of-file writes; subject to the read-before-write gate when read_before_write.enabled (see Middleware Chain)
• str_replace - Substring replacement (single or all occurrences); same-path serialization is scoped to (sandbox.id, path) so isolated sandboxes do not contend on identical virtual paths inside one process; subject to the read-before-write gate when read_before_write.enabled (see Middleware Chain)