NanPan 13edf25b89
fix(streaming): fence memory writes after retirement or END (#7766)
* Rebase stream lifecycle fix onto upstream main

* Preserve gateway config snapshot and recovered stream retention guard
2026-11-17 09:59:18 +08:00

5.1 KiB

Runtime (deerflow/runtime/)

Executes runs: admission, the run worker, the stream bridge (memory and Redis), checkpoint access, the run event store and journal, and the server-installed capabilities of a run. Background: runtime details, run event stream, streaming, checkpoint retention.

Verify

cd backend
python -m pytest tests -q -k "run or stream or checkpoint or event_store or journal"
make test-blocking-io

Streams and reconnect

  • A replay gap is not the end of a run. When the retained window no longer covers a reconnect cursor, the bridge reports a gap and the reader reloads state.

  • Reconnect cursors are client input. Parse them strictly and answer a malformed one as bad input.

  • Redis END publication is one Lua script. The producer-token check, the END detection and the append happen atomically. Restart all Gateways together when the stream mode changes, because old and new producers cannot share a stream.

  • Subscribers get their own copy. The memory bridge copies events outside its condition lock, and caches copy on ingest and on read, so one consumer cannot change what another sees.

  • Keep registry locks short. Heartbeats take the same locks and must not queue behind slow work.

  • Subgraph frames keep their namespace. A bare values frame from a subgraph would overwrite the thread view of an SDK client. Consumers that only handle the root run ignore namespaced frames.

  • Memory writers fence the captured generation. After waiting for the stream Condition, check retired and ended before allocating an event ID or appending; rejected writes consume no sequence number, and END is idempotent for that generation.

Checkpoints

  • The channel mode is fixed for the life of the process. A reader in full mode rejects a delta checkpoint; it does not guess.
  • Read and change state through CheckpointStateAccessor with the thread's effective schema. It preserves mode gates, parentage and pending writes.
  • Delta resume and rollback rewrite the current head. They do not fork. Reject a missing tuple or pending sends before the first write.

Admission and terminal commit

  • Only the worker that owns the run commits its terminal state. A lost or expired lease fences later writes, so a recovered run is not overwritten.
  • Register completion before cleanup awaits. Cancellation during cleanup is then safe, and an idempotent request gets the stored result replayed.
  • Terminal cleanup always finishes. Streams, journals and graph references are closed even when the run is cancelled.
  • A provider tool-call ID is not an owner. Providers reuse IDs across parent runs. The server-generated execution ID owns the registry entry, polling, cancellation and cleanup; the provider ID only correlates what the user sees.

Run events and journal

  • RunJournal is changed on its owning event loop only. A recorder failure fails open, and a slow recorder must not delay tool state.
  • Usage replay may enrich a message, never add or remove one. A provider can report the end of the same model call twice.
  • Event-store mutations of one thread share one fence (an asyncio lock plus a PostgreSQL lock). The fence serializes writers. It does not tell an old run incarnation from a new one.
  • The JSONL store is single-process. It splits on LF only, because Unicode line separators inside a field are data. A started mutation drains through cancellation. The sequence watermark is saved before a run is deleted, and a corrupt watermark fails closed.
  • The in-memory store copies caller-owned content at admission.
  • Event reads pass the trusted user identity, and the owner on a RunRecord matches the durable row.
  • An evidence receipt needs the worker's own final acknowledgement. An unknown or cancelled write outcome yields no receipt.

Goals

  • Goal synthesis is read-only. runtime/goal_synthesis.py composes the /goal from-chat objective from the recent agreement; the Gateway route owns the save. It treats the conversation as data, applies the evaluator's PII redaction and oneshot-LLM path, excludes evidence tool results, keeps the newest turn within its own 12,000-character budget even when a single paste exceeds it, and caps the compaction summary at half so recent constraints retain room.

Server-installed capabilities

  • Capabilities exist for one request. The scheduler capability, the conversation reader and client capability hints are installed by the server. A copy that a caller sends is rejected in both context carriers, and none is restored from a checkpoint.
  • Run metadata and hidden messages are display data. They grant nothing.
  • A scheduled goal is installed in a fresh thread only and is removed before the terminal status commits.
  • Retry the goal evaluator only for a malformed verdict from a normal model completion. Other failures are final.
  • A truncated workspace snapshot cannot prove that a file is absent.