Merge branch 'main' into rayhpeng/hexagonal-feedback-slice

Rebase the feedback migration onto main's new chain tip: main added
0008_thread_operation_kind (also chained after 0007), so
0008_feedback_tags becomes 0009_feedback_tags with
down_revision=0008_thread_operation_kind, keeping the chain linear.
The five head-pin test conflicts resolve to 0009_feedback_tags on top
of main's versions (preserving the new operation_kind assertions).
Verified: 42 migration/bootstrap tests, 54 feedback tests, and the
full frontend suite (797) pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

.github/workflows/e2e-tests.yml

@@ -54,6 +54,12 @@ jobs:
         env:
           SKIP_ENV_VALIDATION: '1'
 
+      - name: Run auth recovery E2E tests
+        working-directory: frontend
+        run: pnpm exec playwright test -c playwright.auth.config.ts
+        env:
+          SKIP_ENV_VALIDATION: '1'
+
       - name: Upload Playwright report
         uses: actions/upload-artifact@v4
         if: ${{ !cancelled() }}

AGENTS.md

@@ -50,6 +50,7 @@ deer-flow/
 ├── frontend/                       # Next.js frontend (pnpm) — see frontend/AGENTS.md
 ├── docker/                         # docker-compose files, nginx config, provisioner
 ├── skills/                         # Agent skills: public/ (committed), custom/ (gitignored)
+│                                    # Managed integration skill packs are installed per user under .deer-flow/users/{user_id}/skills/integrations/
 ├── contracts/                      # Cross-component JSON contracts (e.g. subagent status, skill review)
 ├── scripts/                        # Root orchestration scripts invoked by the Makefile (check, configure, doctor, support_bundle, serve, nginx, docker, deploy, setup_wizard)
 ├── tests/                          # Root-level tests (currently tests/skills/ — public skill tests)

CHANGELOG.md

@@ -198,6 +198,13 @@ This section accumulates work toward the **2.1.0** milestone
 
 ### Fixed
 
+- **runtime:** Thread metadata now switches to `running` only after the run passes
+  the startup barrier, so pending-cancelled runs no longer briefly project
+  `running`; clients may observe the prior thread status during worker startup.
+  ([#4450])
+- **runtime:** Re-check orphan candidates through an atomic, lease-aware takeover
+  claim so a successful heartbeat after the scan keeps the run active and only
+  one reconciler reports recovery. ([#4424])
 - **skills:** Apply `allowed-tools` only to slash-activated or actually loaded
   lead-agent skills, preventing passive enabled skills and evaluation fixtures
   from removing MCP, web, file, and delegation tools from every run. ([#4095],
@@ -243,10 +250,12 @@ This section accumulates work toward the **2.1.0** milestone
   facts list is empty; stop the busy-spin in the debounced update queue; and
   flush the memory queue on graceful shutdown to prevent loss. ([#3719], [#4074],
   [#4076], [#4034], [#4217], [#3993], [#3992], [#4073], [#4181])
-- **runs:** Close multi-worker ownership gaps in run atomicity; degrade cancel
-  to lease takeover for multi-worker; keep `create_thread` idempotent when the
-  insert loses a race; read `stop_reason` from runtime context; and persist run
-  duration in checkpoints for history reads. ([#4003], [#4064], [#3800], [#4188],
+- **runs:** Close multi-worker ownership gaps in run atomicity; fail-stop local
+  execution when lease renewal cannot be confirmed before its deadline and
+  fence late completion writes after peer takeover; degrade cancel to lease
+  takeover for multi-worker; keep `create_thread` idempotent when the insert
+  loses a race; read `stop_reason` from runtime context; and persist run duration
+  in checkpoints for history reads. ([#4003], [#4064], [#4414], [#3800], [#4188],
   [#4118])
 - **runtime:** Serialize SQLite event-store writes to prevent per-thread
   sequence collisions; skip hidden human messages in the journal; and drop the
@@ -1114,3 +1123,5 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
 [#4287]: https://github.com/bytedance/deer-flow/pull/4287
 [#4288]: https://github.com/bytedance/deer-flow/pull/4288
 [#4324]: https://github.com/bytedance/deer-flow/issues/4324
+[#4414]: https://github.com/bytedance/deer-flow/issues/4414
+[#4424]: https://github.com/bytedance/deer-flow/issues/4424

README.md

@@ -249,6 +249,11 @@ make docker-start   # Start services (auto-detects sandbox mode from config.yaml
 
 Docker builds use the upstream `uv` registry by default. If you need faster mirrors in restricted networks, export `UV_INDEX_URL=https://pypi.tuna.tsinghua.edu.cn/simple` and `NPM_REGISTRY=https://registry.npmmirror.com` before running `make docker-init` or `make docker-start`.
 
+Local AIO sandbox control traffic is always direct: loopback/private addresses,
+single-label cluster hosts, and Docker/Podman internal hostnames do not inherit
+`HTTP_PROXY` or `HTTPS_PROXY`. External sandbox FQDNs and public IPs still
+honor environment proxy settings.
+
 Backend processes automatically pick up `config.yaml` changes on the next config access, so model metadata updates do not require a manual restart during development.
 
 > [!TIP]
@@ -275,7 +280,11 @@ Browser login uses `HttpOnly` session cookies. The login page offers a "keep me
 DeerFlow still uses `Forwarded` / `X-Forwarded-*` headers to recover the browser-facing scheme and origin behind a proxy. The bundled nginx sets `X-Forwarded-Proto`, but preserves an upstream HTTPS value and does not overwrite every forwarded header. Configure the outer trusted proxy to replace or strip client-supplied forwarding headers before traffic reaches DeerFlow.
 
 > [!IMPORTANT]
-> The Gateway still owns active run tasks in process, so production defaults to a single Gateway worker (`GATEWAY_WORKERS=1`). Multi-worker deployments require Postgres, the Redis stream bridge (`stream_bridge.type: redis`), and `run_ownership.heartbeat_enabled: true`. The bridge shares SSE delivery and `Last-Event-ID` replay across workers, while lease reconciliation marks runs from dead workers as errors, publishes the terminal stream marker, schedules retained-stream cleanup, and updates the affected thread status. SSE and `/wait` consumers also refresh durable status on heartbeats as a fallback if terminal publication fails. The rolling retained-buffer TTL (`stream_ttl_seconds`) remains a cleanup safety net rather than a run timeout. IM channel state and other process-local services still need their own multi-worker coordination.
+> The Gateway still owns active run tasks in process, so production defaults to a single Gateway worker (`GATEWAY_WORKERS=1`). Multi-worker deployments require Postgres, the Redis stream bridge (`stream_bridge.type: redis`), `run_ownership.heartbeat_enabled: true`, and `run_events.backend: db`; process-local memory/JSONL event stores cannot enforce singleton delivery receipts across workers. The bridge shares SSE delivery and bounded `Last-Event-ID` replay across workers. When a valid reconnect cursor has been trimmed, or a subscriber that already established an empty-stream wait falls behind before its first delivery, Memory and Redis emit a machine-readable SSE `gap` event instead of silently returning a partial replay; the Web UI reloads durable thread/event state and resumes from the retained tail. Lease reconciliation marks runs from dead workers as errors, persists their delivery receipts, publishes the terminal stream marker, schedules retained-stream cleanup, and updates the affected thread status. SSE and `/wait` consumers also refresh durable status on heartbeats as a fallback if terminal publication fails. Malformed Redis reconnect IDs live-tail new events instead of replaying the retained buffer, and the rolling retained-buffer TTL (`stream_ttl_seconds`) remains a cleanup safety net rather than a run timeout. IM channel state and other process-local services still need their own multi-worker coordination.
+>
+> With lease heartbeat enabled, a transient RunStore renewal error is retried only until the last confirmed lease expires; the stale worker then cancels local execution and suppresses checkpoint, completion-hook, delivery-receipt, and thread-status finalization. A remote tool side effect already in flight may still be outside local cancellation.
+>
+> Reconciliation uses an atomic takeover claim that re-checks the lease after candidate selection, so a successful owner renewal wins over orphan recovery and only one reconciler can report a run as recovered.
 
 See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed Docker development guide.
 
@@ -502,6 +511,7 @@ DINGTALK_CLIENT_SECRET=your_client_secret
 
 1. Chat with [@BotFather](https://t.me/BotFather), send `/newbot`, and copy the HTTP API token.
 2. Set `TELEGRAM_BOT_TOKEN` in `.env` and enable the channel in `config.yaml`.
+3. The bot accepts inbound text, photos, and documents (with or without captions). Hosted Bot API downloads are limited to 20 MB per attachment.
 
 **Slack Setup**
 
@@ -571,6 +581,17 @@ logging:
 
 When enabled, every Gateway HTTP response includes `X-Trace-Id`, logs include `trace_id`, and Langfuse traces created by that request include `metadata.deerflow_trace_id` with the same value.
 
+Gateway run history also records one terminal `run.delivery` receipt per run,
+including zero-output and crash-recovered runs. The receipt is persisted before
+the durable terminal run status during normal execution. Orphan recovery first
+atomically claims an expired lease and then idempotently backfills the receipt,
+so a stale recovery scan cannot overwrite a live run's detailed delivery facts.
+Receipt persistence remains best-effort during an event-store outage. Runs that
+fail checkpoint preflight (or are cancelled while waiting for prior
+finalization) keep the existing completion-data behavior: they receive the
+zero-delivery receipt but do not overwrite RunStore completion fields with an
+empty snapshot.
+
 #### LangSmith Tracing
 
 DeerFlow has built-in [LangSmith](https://smith.langchain.com) integration for observability. When enabled, all LLM calls, agent runs, and tool executions are traced and visible in the LangSmith dashboard.
@@ -661,6 +682,70 @@ An enabled skill's `allowed-tools` policy applies only after that skill is expli
 
 When you install `.skill` archives through the Gateway, DeerFlow accepts standard optional frontmatter metadata such as `version`, `author`, and `compatibility` instead of rejecting otherwise valid external skills.
 
+Managed integrations install shared read-only skill packs without mixing them
+into custom skills. The Lark/Feishu CLI integration is available under
+`Settings → Integrations → Lark / Feishu CLI`; an administrator installs or
+upgrades the official `lark-*` pack once under
+`{DEER_FLOW_HOME}/integrations/skills/lark-cli`, and every user discovers that
+same pack with an independent enabled state. Each user's app configuration and
+OAuth data remain isolated under
+`{DEER_FLOW_HOME}/users/{user_id}/integrations/lark-cli/{config,data}`. These
+secret directories are restricted to `0700`, regular credential files to
+`0600`, and symlinks are rejected.
+
+After installation, users can click **Connect Lark** to open a browser
+authorization link; no terminal authorization is required. The same UI can
+request additional permission domains such as Calendar, Docs, or Drive, or a
+specific OAuth scope reported by `lark-cli`. A cheap status refresh only
+inspects the local credential tree, so the UI reports **Credentials configured
+(not live-verified)** until an explicit browser completion performs live token
+verification. The action then remains **Reconnect Lark** so users can replace
+or extend authorization. If an agent hits missing Lark authorization during a
+conversation, the managed `lark-shared` guidance points the user back to the
+same settings entry with `?settings=integrations`.
+
+Installing the Lark skill pack resolves the latest official `larksuite/cli`
+release from GitHub and downloads that version's skills at install time, so the
+Gateway needs outbound internet access for that step (it falls back to a
+bottom-line pinned version if the release lookup fails). The settings page shows
+the installed version and, when available, the newest published version so an
+admin can reinstall to upgrade. Air-gapped deployments can pre-stage the archive
+and point `DEER_FLOW_LARK_CLI_SKILLS_ARCHIVE` at the local file. Integrity does
+not depend on a pinned archive byte hash (GitHub does not guarantee stable
+source-archive bytes); instead the download is restricted to the official GitHub
+host, every archive member passes structural safety guards, and a content hash
+of the effective installed skill tree (including DeerFlow's injected shared
+guidance) is recorded so content changes are auditable across reinstalls.
+
+When `sandbox.use` selects the AIO provider, the same install also downloads the
+official Linux amd64 and arm64 CLI release archives, verifies their published
+SHA-256 checksums, safely extracts one executable per architecture, and mounts
+the resulting runtime read-only at `/mnt/integrations/lark-cli/runtime`. An
+architecture-selecting launcher in that mount makes `lark-cli` available in the
+sandbox `PATH`. Air-gapped AIO deployments can pre-stage a symlink-free runtime
+tree containing `bin/lark-cli` plus both `linux-{amd64,arm64}/lark-cli` files and
+set `DEER_FLOW_LARK_CLI_SANDBOX_RUNTIME_DIR` to that directory.
+
+> **Sandbox trust boundary:** the browser never receives the Lark app secret, but
+> agent conversations run `lark-cli` inside the sandbox, so the per-user
+> credential directories are mounted into it: `config` (holding the long-lived
+> `appSecret`) is mounted **read-only** and `data` (refreshable OAuth tokens)
+> writable. Both remain *readable* by any process the agent runs there, so code
+> reached via prompt injection in a tool result could read them. Treat the
+> sandbox as inside the Lark credential trust boundary until the sidecar
+> credential-broker follow-up removes these mounts from sandbox execution.
+
+For remote/Kubernetes deployments (the provisioner backend), the sandbox
+`lark-cli` runtime can instead be supplied by an optional init container that
+copies the binaries into a shared `emptyDir` — no install-time GitHub download and
+no hostPath/PVC runtime mount. Publish the image under
+[`docker/lark-cli-init`](docker/lark-cli-init/README.md) and set
+`LARK_CLI_INIT_IMAGE` on the provisioner; it stays off (legacy behavior) when
+unset. The Lark integration status (`GET /api/integrations/lark/status`) reports
+`sandbox_runtime_mode` and `sandbox_runtime_ready` so the Settings UI shows
+whether `lark-cli` will actually be present in the sandbox at chat time, rather
+than a green status hiding a later `command not found`.
+
 If a trusted operator manages the configured skills directory through an external mount such as MinIO, NFS, or CSI, an administrator can call `POST /api/skills/reload` after changing files. This invalidates skill prompt caches for the current Gateway process and waits up to the bounded refresh timeout so subsequent runs rescan the latest files; running tasks are unchanged. A loader-level filesystem failure returns a generic server error and preserves the last successfully loaded process cache rather than publishing an empty catalog. Uvicorn workers and Kubernetes Pods must each be targeted separately. Direct mount writes bypass the validation, SkillScan, and history applied by DeerFlow's install/edit APIs, so only operator-controlled systems should have write access.
 
 Skill installs and agent-managed skill edits run through **SkillScan**, a native deterministic safety scanner before the LLM-based skill scanner. Phase 1 runs offline with no Semgrep/OpenGrep dependency, blocks high-confidence `CRITICAL` findings such as private keys or shell execution, and passes warning findings to the LLM scanner for contextual review. Python instance-client exfiltration checks follow a minimal same-scope evidence chain: a simple name bound to a known client constructor, optional name-to-name aliases, and an actual outbound method or context-manager use supported by that constructor. Constructor roots must be proven imports; bare canonical-looking names are not inferred as modules. Nested scopes do not inherit client handles and inherit only constructor import aliases that are never rebound in the enclosing scope. Comprehensions, walrus-bearing statements, annotations, complex binding targets, unsupported operations, and ambiguous branch flows produce no finding from this signal; skipped constructs conservatively invalidate every name they may bind so stale client state cannot create a finding. A deterministic work budget or recursion limit reached by this best-effort analysis does not discard findings already collected for the file. Set `skill_scan.enabled: false` in `config.yaml` to disable only the deterministic analyzers; safe archive extraction and the LLM scanner still run.
@@ -710,6 +795,9 @@ Web UI chat links percent-encode custom thread identifiers before placing them i
 
 /mnt/skills/custom
 └── your-custom-skill/SKILL.md      ← yours
+
+/mnt/skills/integrations
+└── lark-cli/lark-doc/SKILL.md      ← managed, read-only
 ```
 
 #### Claude Code Integration
@@ -755,11 +843,11 @@ Supported commands:
 
 After each Gateway-backed run, DeerFlow evaluates the visible conversation against the active goal with a non-thinking evaluator model. The evaluator must return a typed blocker (`missing_evidence`, `needs_user_input`, `run_failed`, `external_wait`, or `goal_not_met_yet`) plus visible evidence. DeerFlow only injects a hidden continuation when the latest assistant turn is durably checkpointed, the blocker is `goal_not_met_yet`, the thread did not change during evaluation, and the no-progress breaker has not fired. The safety cap defaults to 8 hidden continuations, and repeated identical non-progress evaluations stop after 2 attempts. `/goal clear` and any user-authored new input win over queued continuations. When the goal is satisfied, DeerFlow clears it automatically and publishes the updated thread state.
 
-The Web UI shows the active goal above the composer. The same command is available from the TUI and supported IM channels. In the Web UI and supported IM channels, setting `/goal <completion condition>` also starts a run with the condition as the task; status and clear commands only manage goal state.
+The Web UI shows the active goal above the composer. The same command is available from the TUI and supported IM channels. In the Web UI and supported IM channels, setting `/goal <completion condition>` also starts a run with the condition as the task; status and clear commands only manage goal state. Setting or clearing a goal is rejected while that thread has a run in flight, including a run owned by another Gateway worker, so the goal checkpoint cannot branch away from an active run's checkpoint lineage.
 
 ### Manual Context Compaction
 
-Use `/compact` in the Web UI composer to summarize older context for the current thread. DeerFlow keeps the full chat visible, but future model calls use the compacted summary plus recent messages. The command is ignored when there is not enough history to compact, and it is blocked while the thread has a run in flight.
+Use `/compact` in the Web UI composer to summarize older context for the current thread. DeerFlow keeps the full chat visible, but future model calls use the compacted summary plus recent messages. The command is ignored when there is not enough history to compact, and it is blocked while the thread has a run in flight, including when that run is owned by another Gateway worker. If a multi-worker reservation loses its lease, DeerFlow cancels the checkpoint writer before the replacing run proceeds and returns a retryable conflict after cleanup. Thread-title edits are serialized through the same state-write boundary and show a conflict without closing the rename dialog when a run is active.
 
 ### Sub-Agents
 

README_zh.md

@@ -421,6 +421,7 @@ DINGTALK_CLIENT_SECRET=your_client_secret
 
 1. 打开 [@BotFather](https://t.me/BotFather)，发送 `/newbot`，复制生成的 HTTP API token。
 2. 在 `.env` 中设置 `TELEGRAM_BOT_TOKEN`，并在 `config.yaml` 里启用该渠道。
+3. 机器人支持接收入站文本、图片和文档（可带说明文字，也可不带）；托管版 Bot API 的单个附件下载上限为 20 MB。
 
 **Slack 配置**
 

backend/.gitignore

@@ -31,3 +31,7 @@ config.yaml
 
 # Claude Code settings
 .claude/settings.local.json
+
+# pytest --basetemp workaround dirs (sandbox tmp_path permission)
+.pytest_tmp/
+.pytest_tmp_run/

backend/AGENTS.md

@@ -48,6 +48,7 @@ deer-flow/
 │   │           │   └── registry.py    # Agent registry
 │   │           ├── tools/builtins/    # Built-in tools (present_files, ask_clarification, view_image, review_skill_package)
 │   │           ├── mcp/               # MCP integration (tools, cache, client)
+│   │           ├── integrations/      # Managed first-party integration installers (e.g. Lark CLI skill pack)
 │   │           ├── models/            # Model factory with thinking/vision support
 │   │           ├── skills/            # Skills discovery, loading, parsing
 │   │           ├── config/            # Configuration system (app, model, sandbox, tool, etc.)
@@ -218,7 +219,12 @@ Blocking-IO runtime gate (`tests/blocking_io/`):
   for #1917); `test_sqlite_lifespan.py` (locks the offload around
   SQLite path resolution plus `ensure_sqlite_parent_dir`, fix for #1912);
   `test_jsonl_run_event_store.py` (locks `JsonlRunEventStore`'s async
-  API offloading its file IO via `asyncio.to_thread`);
+  API — including idempotent singleton-event writes — offloading its file IO
+  via `asyncio.to_thread`); `test_run_journal_callbacks.py` (locks
+  `RunJournal.run_inline` tool callbacks to in-memory/event-loop-safe work);
+  `test_integrations_router.py` (locks Lark integration install and auth
+  completion route handlers offloading archive filesystem work and `lark-cli`
+  subprocesses);
   `test_uploads_middleware.py` (locks `UploadsMiddleware.abefore_agent`
   offloading the uploads-directory scan off the event loop);
   `test_uploads_router.py` (locks Gateway upload/list/delete endpoints
@@ -291,6 +297,15 @@ tool graph or subagent executor during state/schema imports.
 **ThreadState** (`packages/harness/deerflow/agents/thread_state.py`):
 - Extends `AgentState` with: `sandbox`, `thread_data`, `title`, `artifacts`, `todos`, `uploaded_files`, `viewed_images`, `goal`, `promoted`, `delegations`, `skill_context`, `summary_text`
 - Uses custom reducers: `merge_artifacts` (deduplicate), `merge_viewed_images` (merge/clear), `merge_goal` (preserve the active goal across ordinary state updates unless the goal writer replaces it), `merge_promoted` (catalog-hash-scoped deferred tool promotions), `merge_delegations` (append task delegation entries, same id latest wins, terminal status never downgraded, capped to the most recent entries), and `merge_skill_context` (dedupe active-skill references by path, keep the most recently read entries; entries store a name/path/description reference, not the SKILL.md body). `summary_text` is a LastValue channel updated by summarization and projected into model requests as durable context data instead of being stored as a `messages` item.
+- Delta-mode `merge_message_writes` normalizes the current message state once,
+  then folds normalized writes in order with message-ID position indexes and
+  deferred tombstone compaction. It preserves public `add_messages` behavior,
+  including duplicate IDs, replacement position, removal errors,
+  `REMOVE_ALL_MESSAGES`, null-write errors, and missing-ID allocation order,
+  without rescanning the accumulated state for every write. Keep this
+  full-parity contract covered by differential tests: LangGraph's private
+  `_messages_delta_reducer` is also linear, but intentionally omits some of
+  those public `add_messages` semantics and cannot be substituted directly.
 
 **Runtime Configuration** (via `config.configurable`):
 - `thinking_enabled` - Enable model's extended thinking
@@ -348,8 +363,9 @@ Before changing a later authorization phase, read the [authorization RFC](../doc
 30. **Custom middlewares** - *(optional)* Any `custom_middlewares` passed to `build_middlewares` are injected here, before config-declared extensions and the terminal-response/safety/clarification tail
 31. **Configured extension middlewares** - *(optional, if `extensions.middlewares` is set in `config.yaml` or `extensions_config.json`)* Zero-argument `AgentMiddleware` classes loaded from `module.path:ClassName` entries via `deerflow.reflection.resolve_class`. Missing packages, invalid classes, and broken modules fail loudly at agent creation. These run after built-ins/programmatic custom middleware and after the lead/subagent loop/token guards, but before the terminal-response/safety/clarification tail; subagents receive the same configured extension middleware class list before their safety tail. Treat these files as trusted operator config because middleware paths instantiate arbitrary code. Gateway skill/MCP toggle endpoints preserve this field through `to_file_dict()` but must not add a write path for `extensions.middlewares` without an explicit trust-boundary review. Lead-only vs subagent-only middleware lists and per-context constructor parameters are not expressible in this MVP.
 32. **TerminalResponseMiddleware** - When a provider returns an empty terminal `AIMessage` after tool execution, injects a hidden recovery prompt and retries the model once; a second empty response is replaced in checkpoint state by a visible error fallback marked for the run worker, so the run finishes as an error instead of a silent success
-33. **SafetyFinishReasonMiddleware** - *(optional, if `safety_finish_reason.enabled`)* Repairs AIMessages the provider safety-terminated (e.g. `finish_reason=content_filter`): strips truncated tool calls so they are not executed (#3028), and — when the response is otherwise blank (no tool calls, no visible content) — backfills a user-facing explanation so the empty message is not persisted and then rejected by strict OpenAI-compatible providers on the next request (`message ... with role 'assistant' must not be empty`), which would otherwise strand the whole thread (#4393). A safety-terminated response that still carries visible text is left untouched. Registered after terminal-response/custom/configured middlewares so LangChain's reverse-order `after_model` dispatch runs it first
-34. **ClarificationMiddleware** - Intercepts `ask_clarification` tool calls, writes a readable `ToolMessage.content` fallback plus structured `ToolMessage.artifact.human_input` request payload, and interrupts via `Command(goto=END)` (must be last). Because this middleware can short-circuit tool execution before LangChain emits `on_tool_end`, `RunJournal` performs a root-run final reconciliation for allowlisted clarification `ToolMessage`s whose `tool_call_id` was produced by the current run, so human-input request cards remain recoverable from `run_events` after checkpoint compaction. Human Input Card replies are submitted as `hide_from_ui` `HumanMessage`s with `additional_kwargs.human_input_response`; `RunJournal` persists only allowlisted hidden response sources (currently `ask_clarification`) as `llm.human.input`, which preserves answered-card state after compaction without exposing generic internal hidden context.
+33. **ModelLengthFinishReasonMiddleware** - Records `stop_reason=model_length_capped` when provider-specific length detectors match a terminal `AIMessage` without tool-call intent (`finish_reason=length` / `MAX_TOKENS`, or `stop_reason=max_tokens`), preserving the original assistant content and never reparsing textual tool-call-like envelopes
+34. **SafetyFinishReasonMiddleware** - *(optional, if `safety_finish_reason.enabled`)* Suppresses tool execution when the provider safety-terminated the response (e.g. `finish_reason=content_filter`); registered after terminal-response/custom/configured middlewares so LangChain's reverse-order `after_model` dispatch runs it first
+35. **ClarificationMiddleware** - Intercepts `ask_clarification` tool calls, writes a readable `ToolMessage.content` fallback plus structured `ToolMessage.artifact.human_input` request payload, and interrupts via `Command(goto=END)` (must be last). Because this middleware can short-circuit tool execution before LangChain emits `on_tool_end`, `RunJournal` performs a root-run final reconciliation for allowlisted clarification `ToolMessage`s whose `tool_call_id` was produced by the current run, so human-input request cards remain recoverable from `run_events` after checkpoint compaction. Human Input Card replies are submitted as `hide_from_ui` `HumanMessage`s with `additional_kwargs.human_input_response`; `RunJournal` persists only allowlisted hidden response sources (currently `ask_clarification`) as `llm.human.input`, which preserves answered-card state after compaction without exposing generic internal hidden context.
 
 ### Configuration System
 
@@ -416,9 +432,10 @@ Localhost persistence deliberately reads the direct request `Host` and ignores `
 | **Console** (`/api/console`) | Read-only cross-thread observability for the current user (the data layer for an operations dashboard or external monitoring): `GET /stats` - headline counters (runs/threads/agents/tokens/cost); `GET /runs` - paginated run history joined with thread titles (per-run cost); `GET /usage` - zero-filled daily token series + per-model breakdown with spend. Queries `runs`/`threads_meta` directly as a reporting layer (no new `RunStore` methods); requires a SQL database backend — returns 503 on `database.backend: memory`. Real-cost estimation reads optional `models[*].pricing` (`currency`, `input_per_million`, `output_per_million`, `input_cache_hit_per_million`; `ModelConfig` is `extra="allow"`, so no schema change) and prices each run from its `token_usage_by_model` input/output split. Pricing is **cache-aware**: `RunJournal` accumulates prompt-cache hits from `usage_metadata.input_token_details.cache_read` into a sparse `cache_read_tokens` bucket key (also threaded through `SubagentTokenCollector` → `record_external_llm_usage_records`), and cache-hit input tokens are billed at `input_cache_hit_per_million` (omitted → billed at the miss price, a conservative upper bound). Legacy rows fall back to run-level totals at `model_name`; unpriced models yield `cost: null` and cost fields are null when no pricing is configured |
 | **MCP** (`/api/mcp`) | `GET /config` - get config; `PUT /config` - update config (saves to extensions_config.json) |
 | **Skills** (`/api/skills`) | `GET /` - list skills; `GET /{name}` - details; `PUT /{name}` - update enabled; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`); `POST /reload` - admin-only process-local prompt-cache invalidation after trusted external filesystem changes |
+| **Integrations** (`/api/integrations`) | `GET /lark/status` - inspect managed Lark/Feishu CLI integration state, including `sandbox_runtime_mode` / `sandbox_runtime_ready` (whether `lark-cli` will actually be present in the sandbox at chat time); `POST /lark/install` - admin-only install of the official `lark-*` managed skill pack; `POST /lark/config/start` and `/lark/config/complete` - internal first-time Lark connection setup; `POST /lark/auth/start` and `/lark/auth/complete` - browser device-flow user authorization without terminal access, with optional `domains` / exact `scope` for incremental permission grants |
 | **Memory** (`/api/memory`) | `GET /` - memory data; `POST /reload` - force reload; `GET /config` - config; `GET /status` - config + data |
 | **Uploads** (`/api/threads/{id}/uploads`) | `POST /` - upload files (auto-converts PDF/PPT/Excel/Word); `GET /list` - list; `DELETE /{filename}` - delete |
-| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; `POST /branches` - create a new main-thread branch from a completed assistant turn checkpoint and, when an addressable pre-user replay checkpoint exists, materialize it into the branch namespace so the inherited response remains regeneratable. Workspace files are not checkpointed, so the branch only best-effort copies the current workspace when branching from the **latest** turn (`workspace_clone_mode="current_thread_best_effort"`); branching from an older/historical turn skips the copy (`workspace_clone_mode="skipped_historical_turn"`) so the branch never inherits files that only exist in a later timeline. Branch creation also seeds the new thread's run-event feed from the branch checkpoint's visible messages (`history_seed_mode` in the response): the thread feed reads run_events, not checkpoints, so without the seed the inherited history disappears from the UI after the branch's first run (#4380); `GET /goal`, `PUT /goal`, `DELETE /goal` - read, set, and clear the active thread goal; `POST /compact` - manually summarize older active context into `summary_text` and retain the recent message window, blocked while a run is in flight; unexpected failures are logged server-side and return a generic 500 detail |
+| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; `POST /branches` - create a new main-thread branch from a completed assistant turn checkpoint and, when an addressable pre-user replay checkpoint exists, materialize it into the branch namespace so the inherited response remains regeneratable. Workspace files are not checkpointed, so the branch only best-effort copies the current workspace when branching from the **latest** turn (`workspace_clone_mode="current_thread_best_effort"`); branching from an older/historical turn skips the copy (`workspace_clone_mode="skipped_historical_turn"`) so the branch never inherits files that only exist in a later timeline. Thread-scoped runtime channels (`sandbox`, `thread_data`) are not copied onto the branch: the parent's `sandbox_id` binds path mappings and the release lifecycle to the parent's workspace, so the branch lazily acquires its own sandbox instead. Branch creation also seeds the new thread's run-event feed from the branch checkpoint's visible messages (`history_seed_mode` in the response): the thread feed reads run_events, not checkpoints, so without the seed the inherited history disappears from the UI after the branch's first run (#4380). Seeded rows are grouped into one synthetic run per inherited turn (`branch-seed-{thread_id}-{n}`, a new turn opening at every persisted human message, including an allowlisted hidden `ask_clarification` reply) because `run_id` is a turn identity to the feed's consumers, not a provenance tag: regenerating an inherited answer supersedes that row's whole `run_id` in `GET /messages/page`, so one shared id for the entire seed deleted the complete inherited history on a branch's first regenerate (#4458); `GET /goal`, `PUT /goal`, `DELETE /goal` - read, set, and clear the active thread goal; `POST /compact` - manually summarize older active context into `summary_text` and retain the recent message window, blocked while a run is in flight; unexpected failures are logged server-side and return a generic 500 detail |
 | **Artifacts** (`/api/threads/{id}/artifacts`) | `GET /{path}` - serve artifacts; active content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) are always forced as download attachments to reduce XSS risk; `?download=true` still forces download for other file types |
 | **Suggestions** (`/api/suggestions`) | `GET /config` - returns global suggestions config boolean; `POST /threads/{id}/suggestions` - generate follow-up questions; rich list/block model content is normalized and inline reasoning (`<think>...</think>`, including unclosed/truncated blocks from reasoning models like MiniMax-M3) is stripped before JSON parsing |
 | **Input Polish** (`/api/input-polish`) | `POST /` - rewrite a composer draft before it is sent. This is a short authenticated `runs:create` LLM request using `input_polish` config; it does not create a LangGraph run, persist a message, or modify thread state. Shares the non-graph one-shot LLM path (`deerflow.utils.oneshot_llm.run_oneshot_llm`) with the suggestions route so model build + Langfuse metadata + invoke stay in one place; validates the same stripped view of the draft it sends to the model, and preserves literal `<think>` substrings in the rewrite (`strip_think_blocks(truncate_unclosed=False)`) |
@@ -436,17 +453,61 @@ captures a pre-run and post-run snapshot of the thread-owned `workspace` and
 are size-limited; binary, large, and sensitive-looking paths are persisted as
 metadata only.
 
+**Run delivery receipts**: `RunJournal` records each non-empty artifact update
+once per tool `Command` for the terminal `run.delivery` event. When a command
+contains multiple messages, a unique tool name resolved from matching
+`ToolMessage` entries supplies attribution; additional command messages do not
+duplicate artifact paths or counts. If multiple different tool names resolve
+for one flat artifact update, the paths remain counted but unattributed because
+the command does not carry a per-path mapping. `RunJournal` callbacks set
+`run_inline=True`: they do only in-memory bookkeeping or schedule async writes,
+and staying on the run's event-loop thread serializes parallel tool callbacks
+before terminal delivery recording and flushing. Each worker creates a separate
+journal per run before cancellable/fallible preflight work, so checkpoint
+compatibility failures and cancellation while waiting for prior finalization
+still emit a zero-delivery receipt. The worker flushes ordinary journal events,
+idempotently persists the run-scoped receipt, and only then persists the staged
+terminal run status. A receipt failure is retried on a short bounded schedule
+while the owning worker still knows the real outcome and holds the lease. If
+all attempts fail, the worker persists that real terminal status instead of
+leaving a successful run inflight for orphan recovery to rewrite as an error;
+the receipt remains best-effort in that outage case. Orphan recovery first
+atomically claims an expired lease, then uses the same singleton write to
+backfill a zero-delivery receipt. This ordering prevents a stale recovery scan
+from overwriting a live run's later detailed receipt; an event-store outage
+does not undo the terminal takeover. An existing detailed receipt is preserved
+when a worker crashed after writing it. Event stores
+serialize `put_if_absent` with ordinary thread writers: memory and JSONL provide
+the documented single-process guarantee, while the DB store adds per-thread
+in-process locks and PostgreSQL advisory locks for cross-process writers.
+Moving journal construction ahead of preflight is receipt-only on early failure
+paths: a separate boundary flag preserves the previous completion-data
+semantics, so checkpoint incompatibility or cancellation while waiting for an
+older finalizing run does not persist an empty completion snapshot. Worker tests
+pin one accumulated receipt across multiple goal-continuation `_stream_once`
+calls; journal tests drive LangChain's real async callback dispatcher against a
+single journal to pin serialized, deduplicated parallel tool callbacks.
+Multi-worker deployments therefore require `run_events.backend: db` for shared,
+ordered delivery events; the startup gate rejects process-local memory and
+JSONL event stores when `GATEWAY_WORKERS > 1`.
+
 **RunManager / RunStore contract**:
-- LangGraph-compatible run requests validate their supported subset before creating a run. `runtime/stream_modes.py` is the shared backend contract for public stream modes and the worker's `graph.astream` mapping; the public `messages-tuple` mode maps to LangGraph's internal `messages` mode, while public `messages`, `events`, and other unsupported modes are rejected instead of being dropped or replaced with `values`. `app/gateway/run_models.py::RunCreateRequest` is shared by HTTP and internal scheduled launch paths, retains only truthful compatibility defaults for unimplemented options (`if_not_exists="create"` plus `None` placeholders), returns 422 for unsupported values including `on_completion="complete"`, `on_completion="continue"`, and `multitask_strategy="enqueue"`, and forbids undeclared SDK options so fields such as `checkpoint_during` and `durability` cannot be silently discarded.
+- LangGraph-compatible run requests validate their supported subset before creating a run. `runtime/stream_modes.py` is the shared backend contract for public stream modes and the worker's `graph.astream` mapping; the public `messages-tuple` mode maps to LangGraph's internal `messages` mode, while public `messages`, `events`, and other unsupported modes are rejected instead of being dropped or replaced with `values`. `app/gateway/run_models.py::RunCreateRequest` is shared by HTTP and internal scheduled launch paths, retains only truthful compatibility defaults for unimplemented options (`if_not_exists="create"` plus `None` placeholders), returns 422 for unsupported values including `on_completion="complete"`, `on_completion="continue"`, and `multitask_strategy="enqueue"`, and forbids undeclared SDK options so fields such as `checkpoint_during` and `durability` cannot be silently discarded. A placeholder must still accept the stock SDK's own default: `langgraph_sdk` drops only `None` from its run payload, so `stream_resumable=False` reaches every request and means "non-resumable", which is what DeerFlow serves — rejecting it 422'd every IM channel run (#4466). `tests/test_run_request_validation.py::test_gateway_accepts_langgraph_sdk_default_payload` pins the real SDK payload against this boundary; channel tests mock the SDK client and cannot catch this class of drift.
 - `RunManager.get()` is async; direct callers must `await` it.
 - The history batch helpers `list_successful_regenerate_sources()` and `get_many_by_thread()` default to `user_id=AUTO`: they resolve the request user and fail closed when no user context exists. Migration/admin callers that intentionally need an unscoped read must pass `user_id=None` explicitly.
 - When a persistent `RunStore` is configured, `get()` and `list_by_thread()` hydrate historical runs from the store. In-memory records win for the same `run_id` so task, abort, and stream-control state stays attached to active local runs.
+- Thread metadata status switches to `running` only after `RunManager.try_start()` succeeds. Pending-cancelled runs therefore skip the old `running` projection, while clients may observe the prior thread status during the short worker-startup window.
 - `cancel()` returns a :class:`~deerflow.runtime.CancelOutcome` enum: `cancelled` (local cancel), `taken_over` (non-owning worker claimed the run because the owner's lease expired — marks it as `error`), `lease_valid_elsewhere` (owner's lease is still alive — caller should return 409 + `Retry-After`), `not_active_locally` (heartbeat disabled, preserving the old 409 path), `not_cancellable` (terminal state), or `unknown` (not found in memory or store). `create_or_reject(..., multitask_strategy="interrupt"|"rollback")` persists interrupted status through `RunStore.update_status()`, matching normal `set_status()` transitions.
+- Interrupt/rollback admission registers the replacement before its best-effort persistence of locally interrupted predecessors. If the admitting caller is cancelled during that post-registration await, `RunManager` drains a shielded replacement cleanup before propagating `CancelledError`, including across repeated cancellation. The cleanup normally persists `interrupted`; if that best-effort transition fails, it retries the active-to-`interrupted` store transition strictly and verifies the result with the replacement's captured owner identity. A concurrent peer terminal transition wins and is synchronized back into the local record rather than being overwritten or deleted.
 - Store-only hydrated runs are readable history. In multi-worker mode with heartbeat enabled, cancel on a store-only run can take over (mark `error`) when the owner's lease has expired past the grace window; otherwise it fails with 409 + `Retry-After`. In single-worker mode (heartbeat off), store-only runs still return 409.
+- A local worker's `RunRecord.lease_expires_at` is the last durably confirmed ownership deadline. `_renew_leases()` bounds each renewal attempt by that deadline: transient store exceptions remain retryable while it is valid, but an exception or blocked call that reaches expiry sets the process-local `ownership_lost` fence, raises `abort_event`, and cancels the run task. Fenced workers do not perform subsequent journal/delivery-receipt, progress/completion/status, checkpoint/thread-metadata, or `on_run_completed` writes; the peer recovery path owns the terminal receipt. `RunStore.update_run_completion()` also refuses to replace a different terminal status, closing the peer-takeover/late-finalization race. `grace_seconds` delays peer reclamation for clock skew but is not extra execution time for an owner that can no longer confirm its lease. Already-committed remote tool side effects remain outside this local cancellation boundary.
 - Startup/orphan reconciliation must claim stale active rows with `RunStore.claim_for_takeover()`, not a plain `update_status()`. The final claim re-checks `status` and lease expiry atomically, so a heartbeat renewal between the candidate scan and the recovery write keeps the run active.
+- Run admission and independent checkpoint writes are first-class thread operations. `runs.operation_kind` distinguishes user-visible `run` rows from internal `checkpoint_write` reservations, while every active kind shares the existing durable active-thread uniqueness constraint. New operation kinds must go through `RunStore.create_thread_operation_atomic()` and `RunManager.reserve_thread_operation()` rather than adding another lock or metadata marker. Live and lease-less reservations are non-interruptible; an expired leased reservation can be reclaimed immediately by interrupt/rollback admission without waiting for orphan reconciliation. Lease-less rows stay fail-closed because the store cannot distinguish a stale row from a live checkpoint writer in another heartbeat-disabled worker; a rare failed delete therefore requires startup reconciliation, and heartbeat-disabled multi-worker deployment remains unsupported. Reservation bodies are attached to their caller task so loss detected by lease renewal cancels the checkpoint writer before it can continue after takeover; the context manager translates that lease-loss cancellation to `ConflictError` after cleanup so Gateway mutation routes return a retryable 409 instead of dropping the HTTP request. The cleanup scope begins immediately after durable admission, including the await that attaches the caller task, so cancellation cannot strand a locally renewed pending reservation. A failed renewal is revalidated under the manager lock before cancellation; if the reservation completed and unregistered while the store update was in flight, its request task must not be cancelled after the checkpoint write. Reservations are excluded from run history/reporting and from run-only helpers such as `list_by_thread()` and `has_inflight()`, release uses the captured owner rather than ambient user context, and local cleanup still runs when the best-effort store delete fails. `RunStore.create_run_atomic()` remains a deprecated compatibility shim for external stores that only admit normal runs; new stores must implement `create_thread_operation_atomic()` to support internal operation kinds.
+- Gateway checkpoint mutations outside run execution must use `services.reserve_checkpoint_write()`, which composes the process-local thread lock with the durable `checkpoint_write` reservation. Manual compaction, `POST /threads/{id}/state`, and both goal mutation routes (`PUT` / `DELETE /threads/{id}/goal`, including creation of a missing goal checkpoint) use this boundary, so an existing run blocks the write and the reservation blocks new reject/interrupt/rollback runs across workers.
 - `POST /wait` (both thread-scoped and `/api/runs/wait`) drains the stream bridge via `wait_for_run_completion()` instead of bare `await record.task`, so it honours the run's `on_disconnect` setting and cancels the background run on real client disconnect rather than returning a stale checkpoint (issue #3265).
+- Memory and Redis `StreamBridge` implementations retain only `stream_bridge.queue_maxsize` data events. A syntactically valid `Last-Event-ID` older than the retained watermark, or a live subscriber that falls behind it, yields `StreamGap` before any partial replay. `sse_consumer` maps that control item to an id-less SSE `gap` payload (`stream_replay_gap`) and intentionally leaves the run active; internal `/wait` consumers resume from its latest retained ID because they only need terminal completion. Redis checks bounds plus the non-blocking read in one transaction, using blocking `XREAD` only as a wake-up before repeating the atomic snapshot. For a no-cursor subscriber that established a wait on an empty stream, the first wake response remains provisional until that next snapshot verifies its tail is still retained; this closes the pre-first-delivery trimming window without changing malformed-cursor live tailing. The correctness tradeoff is one three-command snapshot pipeline per poll plus the blocking wake round trip while idle. Malformed cursor behavior remains backend-specific. Memory treats a syntactically numeric cursor below its watermark conservatively as a gap even when the evicted timestamp can no longer be verified; unknown ids at or above the watermark retain the legacy replay-from-earliest policy.
 - Redis `StreamBridge` keys use a rolling retained-buffer TTL (`stream_bridge.stream_ttl_seconds`, refreshed on `publish()` / `publish_end()`) as a leak safety net, not as a run timeout. Startup and lease-driven periodic orphan recovery share one Gateway stream-terminalization path: after `RunManager` durably marks a run `error` with `stop_reason=orphan_recovered`, Gateway publishes `END_SENTINEL` and schedules stream cleanup. The periodic store scan, per-row status writes, and Gateway callback run as one supervised single-flight task, so a slow pass is skipped at the next interval instead of piling up or pausing the sole lease-renewal loop. Store retries have bounded attempts/backoff; an individual operation still relies on the database driver/pool timeout. `RunManager.shutdown()` gives active user runs priority within its shared deadline, then drains or cancels orphan recovery. Gateway tracks delayed recovered-stream cleanups and converts unfinished delays to immediate deletes before closing the bridge; the Redis TTL remains the outage safety net. Only startup recovery, before the runtime yields to requests, projects the latest affected thread to `error`; periodic recovery deliberately avoids that non-atomic projection because `ThreadMetaStore` has no `latest_run_id` conditional-update contract. Store-only SSE and `/wait` consumers wait for the bridge's real END marker after an ordinary durable terminal status, because status persistence can precede tail events. The explicit `orphan_recovered` signal is the only heartbeat fallback: its publisher is known to be gone, so it supplies the liveness boundary if END publication fails or the retained key expires. Malformed `Last-Event-ID` reconnect values live-tail new Redis events rather than replaying the retained buffer. Keep cross-component recovery orchestration in Gateway through the generic `RunManager.on_orphans_recovered` callback; do not introduce a harness-to-app dependency. Callback failure warnings include every recovered `run_id` so operators can identify rows whose Gateway-side terminalization needs inspection.
-- Thread-scoped run creation accepts `checkpoint` / `checkpoint_id`; Gateway validates the checkpoint belongs to the request thread before writing `checkpoint_id` / `checkpoint_ns` into `config.configurable` for LangGraph branching.
+- Thread-scoped run creation accepts `checkpoint` / `checkpoint_id`; Gateway validates the checkpoint belongs to the request thread before writing `checkpoint_id` / `checkpoint_ns` into `config.configurable` for LangGraph branching. In `delta` checkpoint mode the worker rewrites that fork into a linear head write before the graph starts (see "A delta-mode run cannot fork" under Checkpoint Channel Modes), because delta state for a fork replays the abandoned sibling's writes.
 - Thread-scoped Gateway runs evaluate an active `ThreadState.goal` after the visible turn completes. `runtime/goal.py` asks a non-thinking evaluator model to judge only visible conversation evidence and return a typed blocker; the evaluator model is created once per run and reused across hidden continuation checks. The evaluator runs after the graph root's tracing scope has already closed, so `create_goal_evaluator_model`/`evaluate_goal_completion` attach their own model-level tracing callbacks (`attach_tracing=True`) and inject Langfuse trace metadata (`thread_id`/`user_id`/`deerflow_trace_id`) directly onto the `ainvoke` call — the same standalone-caller pattern as `oneshot_llm.run_oneshot_llm` and `MemoryUpdater` (see Tracing System below). Satisfied goals are cleared; every non-satisfied evaluation — continuable or stand-down — is persisted with `last_evaluation` (the blocker, reason, and evidence summary; outcomes that stop the loop additionally record a `stand_down_reason` for observability), but only `goal_not_met_yet` evaluations are streamed as hidden `HumanMessage` continuations, and only when a durable assistant end-of-turn checkpoint exists, the run has not been aborted, the thread did not change during evaluation, and the no-progress breaker has not fired. The continuation cap is 8 — a hard maximum in the `0`–`8` range; callers requesting more are clamped (`set_goal`/TUI) or rejected with 422 (`PUT /goal`). The no-progress breaker keys on the latest visible assistant evidence (not the evaluator's free-text reason, which an LLM rewords every turn), so two consecutive continuations that add no new visible assistant output stop the loop after 2 attempts. Model-response cleanup helpers such as think-block stripping and code-fence stripping live in `deerflow.utils.llm_text` so `runtime/goal.py` and Gateway suggestion parsing share the same JSON-prep behavior.
 - Run event stream changes must keep producer code, `deerflow/constants.py`, `runtime/events/catalog.py`, `contracts/run_event_stream_contract.json`, `backend/docs/RUN_EVENT_STREAM.md`, and `tests/test_run_event_stream_contract.py` in sync. The dependency-free constants module owns the persisted envelope limits (`event_type` 32 characters, `category` 16) and cross-layer workspace event identity; the catalog owns validated runtime definitions and categories. Dynamic middleware tags are limited to 21 characters after the `middleware:` prefix. The JSON contract owns payload schemas, backend-specific storage semantics, legacy aliases, and compatibility rules; conformance tests require both views and all producer groups to agree. `run.end.content` remains opaque and may retain nested Python values in memory while JSONL/database stores stringify non-JSON nested values, so consumers must not assume backend-identical nested output representations.
 
@@ -469,7 +530,7 @@ copying a raw checkpoint because delta state is not self-contained in one tuple.
 **Environment policy** (`sandbox/env_policy.py`): `execute_command` no longer inherits the full `os.environ`. `build_sandbox_env()` scrubs secret-looking names (`*KEY*`/`*SECRET*`/`*TOKEN*`/`*PASS*`/`*CREDENTIAL*`) from the inherited environment before layering injected request secrets on top, so platform credentials (e.g. `OPENAI_API_KEY`) never leak into skill subprocesses. Benign vars (`PATH`, `HOME`, `LANG`, `VIRTUAL_ENV`, ...) are preserved.
 **Implementations**:
 - `LocalSandboxProvider` - Local filesystem execution. `acquire(thread_id)` returns a per-thread `LocalSandbox` (id `local:{thread_id}`) whose `path_mappings` resolve `/mnt/user-data/{workspace,uploads,outputs}` and `/mnt/acp-workspace` to that thread's host directories, so the public `Sandbox` API honours the `/mnt/user-data` contract uniformly with AIO. `acquire()` / `acquire(None)` keeps the legacy generic singleton (id `local`) for callers without a thread context. Per-thread sandboxes are held in an LRU cache (default 256 entries) guarded by a `threading.Lock`. Legacy global-custom mounts are gated by the same user-scoped skill discovery rule used for prompt/list visibility; providers must not infer visibility from raw directory presence alone.
-- `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker-based isolation. Active-cache and warm-pool entries are checked with the backend during acquire/reuse; definitively dead containers are dropped from all in-process maps so the thread can discover or create a fresh sandbox instead of reusing a stale client. Backend health-check failures are treated as unknown, not dead; local discovery likewise treats an unverifiable container as not adoptable and falls through to create rather than failing acquire. `get()` remains an in-memory lookup for event-loop-safe tool paths — it never touches the ownership store (that would be blocking IO on the event loop); ownership is published on acquire/reclaim and refreshed off the event loop by the dedicated renewal thread (`_renew_owned_leases`). Legacy global-custom mounts follow the same shared visibility helper as local and remote providers.
+- `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker-based isolation. Active-cache and warm-pool entries are checked with the backend during acquire/reuse; definitively dead containers are dropped from all in-process maps so the thread can discover or create a fresh sandbox instead of reusing a stale client. Backend health-check failures are treated as unknown, not dead; local discovery likewise treats an unverifiable container as not adoptable and falls through to create rather than failing acquire. `get()` remains an in-memory lookup for event-loop-safe tool paths — it never touches the ownership store (that would be blocking IO on the event loop); ownership is published on acquire/reclaim and refreshed off the event loop by the dedicated renewal thread (`_renew_owned_leases`). Legacy global-custom mounts follow the same shared visibility helper as local and remote providers. Readiness probes and `agent_sandbox` clients classify loopback/private IPs, single-label cluster hosts, and Docker/Podman internal hostnames as direct control-plane destinations and set `trust_env=False`; external FQDNs and public IPs retain environment proxy support.
 - `E2BSandboxProvider` (`packages/harness/deerflow/community/e2b_sandbox/`) provides E2B remote isolation.
   Acquire and release share a per-user and thread lock. The provider lock does
   not cover remote IO. `burst_limit` adds capacity only for the `burst` policy.
@@ -602,9 +663,9 @@ E2B output sync records remote file versions and actual host file metadata in a
 
 ### Skills System (`packages/harness/deerflow/skills/`)
 
-- **Location**: `deer-flow/skills/{public,custom}/`
+- **Location**: global public skills live under `deer-flow/skills/public/`; user-authored custom skills live under `{DEER_FLOW_HOME}/users/{user_id}/skills/custom/`; globally managed integration skills live under `{DEER_FLOW_HOME}/integrations/skills/{provider}/`; per-user integration credentials remain under `{DEER_FLOW_HOME}/users/{user_id}/integrations/{provider}/{config,data}`
 - **Format**: Directory with `SKILL.md` (YAML frontmatter: name, description, license, allowed-tools, required-secrets)
-- **Loading**: `load_skills()` recursively scans namespace directories under `skills/{public,custom}`, but stops descending once it finds a `SKILL.md`; that directory is a package boundary, so no nested `SKILL.md` is registered as a runtime skill. SkillScan has a deliberately narrower packaging rule: known eval fixtures are permitted as support data, while other nested `SKILL.md` files are reported as package defects. It parses runtime metadata and reads enabled state from extensions_config.json.
+- **Loading**: `load_skills()` recursively scans public, per-user custom, global integration, and legacy custom locations for `SKILL.md`, parses metadata, and reads enabled state from extensions_config.json plus per-user skill state for non-public categories; that directory is a package boundary, so no nested `SKILL.md` is registered as a runtime skill. SkillScan has a deliberately narrower packaging rule: known eval fixtures are permitted as support data, while other nested `SKILL.md` files are reported as package defects. It parses runtime metadata and reads enabled state from extensions_config.json.
 - **External reload**: `POST /api/skills/reload` is an admin-only, process-local invalidation hook for trusted MinIO/NFS/CSI writes. `SkillStorage` instances do not cache a catalog — `load_skills()` scans on every call — so the route clears all `(app_config, user_id)` entries and the rendered prompt-section LRU, then waits up to the shared refresh timeout for the existing off-loop single-flight refresh. Each invalidation receives a generation-bound result handle; a successful scan atomically replaces the global enabled-skills cache, while a loader-level failure propagates to the HTTP waiter and preserves the last-known-good global cache. Per-user/config scans capture the refresh version and cannot repopulate shared caches if invalidation occurs while they are loading. A timed-out HTTP wait fails generically while the daemon refresh worker continues. Subsequent runs rescan after a successful reload; active runs keep their existing snapshot. Each Uvicorn worker/Kubernetes Pod must be targeted separately. Direct mount writes bypass install/edit validation, SkillScan, and history, so mounted roots are an operator-controlled trust boundary.
 - **Tool policy**: Lead-agent `allowed-tools` declarations apply dynamically only to slash-activated skills and skills captured in `ThreadState.skill_context` through configured `read_file` loads; passive enabled skills and custom-agent skill allowlists remain discoverable without clamping the global toolset. Slash policy is dominant for its run, preventing subsequently read skills from widening explicit authority; autonomous captured skills use the existing union only when no slash source exists. `tool_search` and `describe_skill` stay available as framework discovery infrastructure, while every discovered or promoted business tool still requires active-policy permission for schema visibility and execution; `task` likewise requires an explicit declaration. Each active model call intentionally reloads the full live registry so enable/disable changes, frontmatter edits, and custom/public name-shadow winners take effect without a stale TTL or unsafe direct-path cache; all tool calls produced by that model step reuse the resulting source-and-path-signed decision. Registry failures and all-invalid active sets fail closed, while stale individual paths are skipped when another valid skill remains. This is best-effort behavioral scoping, not a hard security boundary: alternate loading paths are not captured and bounded autonomous context may evict entries. Subagents still filter statically because their configured skills are all loaded into the session at startup.
 - **Injection (legacy / default)**: Enabled skills are listed in the agent system prompt with full metadata and container paths (`<available_skills>` block). Controlled by `skills.deferred_discovery: false` (default).
@@ -613,6 +674,7 @@ E2B output sync records remote file versions and actual host file metadata in a
   - `skills/describe.py` — `build_describe_skill_tool(catalog)` builds the `describe_skill` tool as a closure; `build_skill_search_setup(skills, enabled, ...)` produces a `SkillSearchSetup(describe_skill_tool, skill_names)` that is wired into both the LangGraph agent factory (`agent.py`) and the embedded client (`client.py`).
 - **Slash activation**: `/skill-name task` loads that enabled skill's `SKILL.md` for the current model call only. The resolver rejects leading whitespace, missing separators, reserved channel commands (`/new`, `/help`, `/bootstrap`, `/status`, `/models`, `/memory`, `/goal`), disabled skills, and skills outside a custom agent's whitelist.
 - **Installation**: `POST /api/skills/install` extracts .skill ZIP archive to custom/ directory
+- **Managed integrations**: Lark/Feishu CLI support installs one global official `lark-*` pack as read-only `SkillCategory.INTEGRATION` entries under `/mnt/skills/integrations/lark-cli/...`; enabled flags, app configuration, and OAuth data remain per-user. Install resolves the newest `larksuite/cli` release from GitHub (`releases/latest`) at install time (falling back to a bottom-line pinned version if the lookup fails) rather than hard-coding the pack version; integrity relies on the official host + structural archive guards + a recorded hash of the effective installed tree after shared guidance injection (not a pinned archive-byte SHA, which GitHub does not keep stable). The Gateway image still installs a pinned `@larksuite/cli` binary, so `get_lark_integration_status` surfaces `latest_available_version` and `runtime_version_mismatch` for the UI. AIO installs additionally verify and publish official Linux amd64/arm64 binaries under `{DEER_FLOW_HOME}/integrations/lark-cli/sandbox-cli`, mounted read-only at `/mnt/integrations/lark-cli/runtime`; `/mnt/integrations/lark-cli/config` (app credentials, incl. the long-lived `appSecret`) is mounted **read-only** into the sandbox and `/mnt/integrations/lark-cli/data` (refreshable OAuth tokens) stays writable, both mapping to owner-only per-user credential directories. **Sandbox trust boundary:** those two dirs are still *readable* by arbitrary sandbox processes (the agent's `bash` tool, or code reached via prompt-injection in a tool result), so the app secret and tokens are exposed to sandbox-side code even though they never reach the browser — the read-only config mount only prevents in-sandbox tampering, not read/exfiltration. The sidecar credential-broker follow-up (issue #4338) is the planned fix that removes these plaintext mounts from sandbox execution. `DEER_FLOW_LARK_CLI_SANDBOX_RUNTIME_DIR` supplies a validated, symlink-free pre-staged runtime for air-gapped deployments. For the remote provisioner (K8s), the runtime is instead provisioned by an optional init container + shared `emptyDir` (Pattern A): set `LARK_CLI_INIT_IMAGE` on the provisioner (see `docker/lark-cli-init/`) and the Gateway sends `provision_lark_cli_runtime` on sandbox create once the pack is installed, so remote installs skip the Gateway-side GitHub download entirely. `get_lark_integration_status(check_runtime=True)` surfaces `sandbox_runtime_mode` (`none` / `gateway-download` / `init-container`) and `sandbox_runtime_ready` (init-container mode reads the provisioner `GET /api/capabilities`) so a green UI can't hide a chat-time `lark-cli: command not found`. Cheap status probes are explicitly not live-verified; users authorize or reconnect through the browser device-flow endpoints instead of running terminal commands.
 - **SkillScan**: `packages/harness/deerflow/skills/skillscan/` is the native deterministic scanner for `.skill` archives and agent-managed skill writes. It runs offline before the LLM scanner, emits structured findings (`rule_id`, `severity`, `file`, `line`, `message`, `remediation`, redacted `evidence` — category/analyzer are encoded in the `rule_id` prefix), blocks `CRITICAL`, and passes warning findings into `scan_skill_content()`. `scan_archive_preflight()` / `scan_skill_dir()` are pure sync functions (dispatch off the event loop); `enforce_static_scan()` applies the blocking policy and the `skill_scan.enabled` kill switch. The Python instance-client signal deliberately follows only a one-level, same-scope evidence chain (PR #4265 review): a proven imported constructor bound to a simple name, optional name-to-name alias propagation, rebinding invalidation, and a constructor-supported outbound method or context-manager use; bare canonical-looking names never fall back to module identity. Nested scopes never inherit client handles and inherit only constructor aliases proven stable by a binding-only enclosing-scope prepass. Comprehensions, walrus-bearing statements, annotations, executable expressions inside complex binding targets, unsupported operations, and ambiguous flows produce no finding from this signal; skipped constructs invalidate all names they may bind, while representative false negatives are pinned by `test_python_declared_false_negatives_stay_unreported`. Compound bodies are walked from isolated copies so wrapping code in `if True:` is not a bypass, while copied scope entries, binding-only prepasses, and AST visits consume a deterministic work budget and the walk stops after its first sink. Budget or recursion exhaustion skips only this best-effort signal and retains deterministic findings already collected for the file. Do not add Semgrep/OpenGrep or YAML rule-engine dependencies to the core path; Phase 1 rule specs live in Python constants next to their analyzers in `skillscan/orchestrator.py`.
 - **Skill Review Core**: `packages/harness/deerflow/skills/review/` provides read-only package snapshots, deterministic facts, resource/eval analysis, report rendering, and the CLI (`python -m deerflow.skills.review.cli`). It reuses the shared frontmatter helper and SkillScan; it must not import `app.*`, execute target scripts, install dependencies, or call networks. JSON contracts live in `contracts/skill_review/`. The `review_skill_package` built-in tool labels results with `review_subject_entry` and never `skill_context_entry`, so reviewing a target does not activate it, bind its `required-secrets`, or apply its `allowed-tools`. Its model-visible `ToolMessage.content` is a compact JSON payload with untrusted control tags neutralized; the full raw review payload, including Markdown renders, stays in `ToolMessage.artifact`. CI should run the CLI with `--fail-on error --fail-on-incomplete` so blocker/error findings and truncated/not-assessed packages fail the gate. The public `skills/public/skill-reviewer` skill owns semantic readiness review and suggestions only; mutation and runtime experiments remain owned by `skill-creator`.
 
@@ -657,7 +719,7 @@ Bridges external messaging platforms (Feishu, Slack, Telegram, Discord, DingTalk
   A swallowed streaming failure publishes its final outbound before releasing the inbound dedupe key, so a provider redelivery can retry without overtaking the terminal reply.
 - `base.py` - Abstract `Channel` base class (start/stop/send lifecycle)
 - `service.py` - Manages lifecycle of all configured channels from `config.yaml`
-- `slack.py` / `feishu.py` / `telegram.py` / `discord.py` / `dingtalk.py` - Platform-specific implementations (`feishu.py` tracks the running card `message_id` in memory and patches the same card in place; `telegram.py` registers the "Working on it..." placeholder as the stream target and edits it in place via `editMessageText`; `dingtalk.py` optionally uses AI Card streaming for in-place updates when `card_template_id` is configured)
+- `slack.py` / `feishu.py` / `telegram.py` / `discord.py` / `dingtalk.py` - Platform-specific implementations (`feishu.py` tracks the running card `message_id` in memory and patches the same card in place; `telegram.py` accepts inbound text/photos/documents, preserves media captions, hands token-free attachment bytes to the shared upload pipeline, and edits the "Working on it..." stream target in place via `editMessageText`; `dingtalk.py` optionally uses AI Card streaming for in-place updates when `card_template_id` is configured)
 - `github.py` - Webhook-driven GitHub channel. Inbound messages come from `POST /api/webhooks/github`; outbound is log-only because GitHub agents post explicitly with `gh` from their sandbox when they choose to comment or create a PR
 - `app/gateway/routers/channel_connections.py` - Browser-facing user connection and disconnect APIs
 - `deerflow.persistence.channel_connections` - SQL-backed user-owned connection, optional credential, connect state, and conversation store
@@ -665,6 +727,7 @@ Bridges external messaging platforms (Feishu, Slack, Telegram, Discord, DingTalk
 **Message Flow**:
 1. External platform -> Channel impl -> `MessageBus.publish_inbound()`
    - For GitHub, the webhook router verifies the delivery then calls `fanout_event(bus, ...)`; matching agent bindings publish one `InboundMessage` each instead of a long-polling channel worker.
+   - Telegram photo/document updates use the largest photo size or document metadata, preserve `message.caption`, enforce the hosted Bot API's 20,000,000-byte download ceiling before and after download, and never expose the token-bearing Bot API file URL. Downloaded bytes cross the adapter/manager boundary only through `message_bus.INBOUND_FILE_CONTENT_KEY`; the manager consumes that transient field before persisting safe upload metadata.
 2. `ChannelManager._dispatch_loop()` consumes from queue
 3. For user-owned channel connections, incoming messages carry `connection_id`, `owner_user_id`, and `workspace_id`; `owner_user_id` becomes the DeerFlow run `user_id`, while the raw platform user id remains `channel_user_id`. The Gateway accepts `channel_user_id` only from an internally authenticated channel caller's top-level `body.context`, clears it from both free-form `body.config` sections, and writes it into runtime context only (never `configurable`, which is checkpointed). `bash_tool` exposes it to sandbox commands as the fixed env var `DEERFLOW_CHANNEL_USER_ID` — via a shell-quoted command-string prefix, NOT the `execute_command(env=...)` channel, which is reserved for request-scoped secrets and would switch `AioSandbox` onto the `bash.exec` path (image >= 1.9.3, fresh session per call). Per-call injection keeps group-chat identity correct (one thread/sandbox, many senders) **without depending on the AIO shell's session semantics**: every IM-channel command carries an explicit `export VAR=<id>; ` (valid id) or `unset VAR; ` (empty / non-str / over the 256-char cap). The AIO no-env path reuses a persistent shell session (the reason for the class lock, #1433), so a bare command could otherwise resolve a stale id an earlier sender exported; the `unset` closes the window the length/type guard would open (a dropped id would inherit the previous sender's value). Non-IM runs (no `channel_user_id` in context) are left untouched. Not injected on the Windows local sandbox (its PowerShell/cmd.exe fallback has no `export`/`unset`). Propagates across `task` delegation: `task_tool` captures the dispatching turn's id and the subagent executor forwards it into the subagent's runtime context, same as the guardrail attribution fields. The runtime-context value is authorization-grade at the Gateway/guardrail boundary, but the exported shell variable remains informational because any bash command can overwrite its own environment; skills must not treat the shell variable itself as authenticated identity. Tests: `tests/test_gateway_services.py`, `tests/test_channel_user_id_env.py`
 4. For chat: look up/create thread through Gateway's LangGraph-compatible API
@@ -790,6 +853,7 @@ Focused regression coverage for the updater lives in `backend/tests/test_memory_
 - `consolidation_min_facts` - Minimum facts in a category to trigger consolidation review (default: 8; range: 3–30)
 - `consolidation_max_groups_per_cycle` - Maximum categories the LLM can merge in one cycle (default: 3; range: 1–10; also controls the LLM's prompt instruction)
 - `consolidation_max_sources` - Maximum source facts per merge group; prevents over-merging (default: 8; range: 2–20)
+- `watermark_max_keys` - Soft cap on the in-memory conversation-watermark cache (one entry per distinct thread/user/agent). A bounded LRU: when over capacity the least-recently-used entry is dropped, and a dropped key re-extracts one batch on that thread's next turn (same as a restart). Bounds memory in long-lived gateways handling many threads (default: 4096; 0 = unbounded)
 
 ### Reflection System (`packages/harness/deerflow/reflection/`)
 
@@ -830,6 +894,7 @@ This invokes `alembic revision --autogenerate` against the live ORM models. Revi
 - `migrations/versions/0002_runs_token_usage.py` — fixes issue #3682
 - `migrations/versions/0004_run_ownership.py` — `runs` multi-worker ownership + the `uq_runs_thread_active` partial unique index, with a `_dedupe_active_runs_per_thread()` pre-step so `CREATE UNIQUE INDEX` cannot fail on a field DB that already has duplicate active rows per thread
 - `migrations/versions/0007_scheduled_run_active_index.py` — the `uq_scheduled_task_run_active` partial unique index (at most one queued/running `scheduled_task_runs` row per `task_id`), with a `_dedupe_active_scheduled_runs_per_task()` pre-step (keeps the newest active row per task, supersedes the rest to `interrupted` with an explanatory `error` + `finished_at`) mirroring 0004; chains after `0006_agents`
+- `migrations/versions/0008_thread_operation_kind.py` — adds `runs.operation_kind` for durable non-run thread reservations; chains after `0007_scheduled_run_active_index`
 - `persistence/bootstrap.py` — `bootstrap_schema(engine, backend=...)`, the three-branch decision + locking
 - Tests: `tests/test_persistence_bootstrap.py` (branches), `tests/test_persistence_bootstrap_concurrency.py` (concurrency), `tests/test_persistence_bootstrap_regression.py` (issue #3682), `tests/test_persistence_migrations_env.py` (filter), `tests/blocking_io/test_persistence_bootstrap.py` (asyncio.to_thread anchor), `tests/test_migration_0004_run_ownership_dedupe.py` + `tests/test_migration_0007_scheduled_run_active_dedupe.py` (dedupe-before-unique-index pre-steps)
 
@@ -845,14 +910,16 @@ Checkpointer storage runs in one of two channel modes, selected by `checkpoint_c
 
 **Replay checkpoint lookup prefers lineage and degrades only for an explicitly missing legacy parent link.** Branch and regenerate paths first walk `parent_config`, which prevents a global chronological scan from selecting a sibling created by regeneration. `CheckpointParentMissingError` alone enables the bounded newest-first history fallback in `app/gateway/checkpoint_lineage.py`; cycles, dangling/non-addressable parents, target mismatches, and depth exhaustion raise `CheckpointLineageIntegrityError` and fail closed instead of selecting a sibling. The compatibility scans request 400 raw checkpoints so up to 200 duration-only entries do not consume the effective branch-history budget; the fallback scans oldest-to-newest internally, skips duration-only checkpoints, and accepts only checkpoints with an addressable id as the replay base. A source history with no discoverable pre-user checkpoint preserves the historical single-checkpoint branch behavior instead of rejecting the branch; regeneration remains unavailable for that inherited response. Existing single-checkpoint branches are not mutated by regenerate preparation, and no raw checkpoint tuple is copied across threads because delta state depends on ancestry and pending writes. Regenerate source-run lookup uses the current thread's exact event, then the server-stamped `run_id` on the copied human message, then verified RunManager content matching; it does not read parent-thread events. Storage or checkpoint-mode failures are not treated as a missing base and still fail closed.
 
-**Wholesale message replacement uses a state-only mutation graph + `Overwrite`.** `update_state` values pass through channel reducers (`add_messages` merge in full, append in delta), so replacing `messages` wholesale (run rollback, context compaction) requires `{"messages": Overwrite([...])}`. The write goes through `build_state_mutation_graph(as_node, mode, state_schema)` — when the write carries materialized state, `state_schema` MUST be the thread's effective schema (`graph_state_schema(assistant_graph)`), because the base-ThreadState fallback silently discards written channels contributed by custom `AgentMiddleware.state_schema`. Channels absent from the write are unaffected: forked checkpoints clone the parent's channel blobs, so middleware channels survive rollback/compaction regardless of schema (locked by `test_rollback_preserves_middleware_contributed_channels` and `test_compact_thread_context_preserves_middleware_contributed_channels`) — a compiled graph with one no-op node (entry = finish) whose checkpoint machinery (channels/versions/metadata) is identical to the agent graph's but schedules no pending tasks, so the restored/compacted head stays idle instead of re-triggering the agent. Never hand-write checkpoints via `checkpointer.aput` for this; raw writers elsewhere must preserve checkpoint parentage — severed ancestry breaks delta replay (see `runtime/runs/worker.py` writer parenting and `checkpoint_patches.py`).
+**A delta-mode run cannot fork; `runtime/runs/worker.py` linearizes the resume instead.** Resuming from an older checkpoint (regenerate, or any client-supplied `checkpoint`) forks the lineage, and delta state for a fork is not materializable: `BaseCheckpointSaver.get_delta_channel_history` — and the bespoke overrides in `InMemorySaver`/`PostgresSaver` — collect **every** `pending_writes` entry stored on each on-path ancestor, but a shared parent also carries the writes of the sibling child that was abandoned. Those writes replay into the fork, so the run starts from a message list still containing the answer it was supposed to replace (#4458: regenerating in a branched thread showed the superseded assistant message beside the new one after a reload; reproduced on postgres, sqlite, and the in-memory saver). Write-to-child ownership belongs to the upstream delta contract, so DeerFlow does not reimplement the walk: `_linearize_delta_checkpoint_resume` materializes the requested checkpoint's complete state and writes every channel onto the **current head** (which has no siblings) through the state mutation graph, using `Overwrite` for reducer channels and resetting newer head-only channels to their schema default (or `None` when no constructible default exists); it then drops the `checkpoint_id` selector and lets the run proceed linearly, while the abandoned turn stays in history as the rewritten head's ancestry. The worker holds `_checkpoint_thread_lock` across `_capture_rollback_point` and the optional linear rewrite, making the rollback snapshot and rewrite atomic with graph streaming and the preceding run's duration-metadata checkpoint write. Capture preserves the complete real pre-run state; cancel-with-rollback then linearly replaces the current delta head with that captured state rather than forking the now-shared pre-run checkpoint, so the abandoned turn is restored without replaying the resume sibling's writes. The worker also recomputes the current-run message boundary from the rewritten state and fails closed (an unreadable resume checkpoint raises rather than falling back to the corrupt fork). `full` mode keeps forking — its checkpoints carry complete `channel_values` and need no replay — so LangGraph branching semantics are unchanged there. Root namespace only; subgraph namespaces are left alone.
 
-**Run rollback flow** (`runtime/runs/worker.py`): `_capture_rollback_point` materializes pre-run state (messages via accessor, raw `pending_writes` via `aget_tuple`) into an immutable `RollbackPoint` before the run starts — capture failure disables rollback (fail-closed), never restores partial state. Cancel-with-rollback then forks from the pre-run checkpoint via the mutation graph and replays the captured pending writes.
+**Wholesale state replacement uses a state-only mutation graph + `Overwrite`.** `update_state` values pass through channel reducers (`add_messages` merge in full, append in delta), so replacing reducer values requires `Overwrite` rather than an ordinary update. Full-mode rollback and context compaction replace `messages`; delta resume and delta rollback replace every materialized channel and reset current-head-only channels to their schema default (or `None`). These writes go through `build_state_mutation_graph(as_node, mode, state_schema)`, and `state_schema` MUST be the thread's effective schema (`graph_state_schema(assistant_graph)`), because the base-ThreadState fallback silently discards written channels contributed by custom `AgentMiddleware.state_schema`. Channels absent from a full-mode fork write inherit the parent's channel blobs, so middleware channels survive rollback/compaction (locked by `test_rollback_preserves_middleware_contributed_channels` and `test_compact_thread_context_preserves_middleware_contributed_channels`). The compiled mutation graph has one no-op node (entry = finish) whose checkpoint machinery (channels/versions/metadata) is identical to the agent graph's but schedules no pending tasks, so the restored/compacted head stays idle instead of re-triggering the agent. Never hand-write checkpoints via `checkpointer.aput` for this; raw writers elsewhere must preserve checkpoint parentage — severed ancestry breaks delta replay (see `runtime/runs/worker.py` writer parenting and `checkpoint_patches.py`).
+
+**Run rollback flow** (`runtime/runs/worker.py`): `_capture_rollback_point` materializes the complete pre-run state via the accessor and captures raw `pending_writes` via `aget_tuple` into an immutable `RollbackPoint` before the run starts — capture failure disables rollback (fail-closed), never restores partial state. In `full` mode, cancel-with-rollback forks from the pre-run checkpoint via the mutation graph and inherits non-message channels from that parent. In `delta` mode, forking is unsafe once the cancelled path has attached sibling writes to the pre-run checkpoint, so rollback replaces every captured channel on the current head, using `Overwrite` for reducers and schema defaults for current-head-only channels. Both modes reattach only the captured pre-run pending writes to the restored checkpoint.
 
 **Where things live**:
 - `runtime/checkpoint_mode.py` — mode freeze, marker injection, delta detection, compatibility gate, both error types
 - `runtime/checkpoint_state.py` — `CheckpointStateAccessor`, `build_state_mutation_graph`, `RollbackPoint`
-- `checkpoint_patches.py` (package root) — saver patches: delta-history folding for `InMemorySaver` (delegating to the base walk), stable message IDs across materialization, upstream first-write drop fix
+- `checkpoint_patches.py` (package root) — checkpoint-machinery patches: delta-history folding for `InMemorySaver` (delegating to the base walk), stable message IDs across materialization, upstream first-write drop fix, and `BinaryOperatorAggregate` unwrapping an `Overwrite` first write into an empty (MISSING) channel — Union-typed reducer channels (`sandbox`/`goal`/`todos`/`promoted`) have no constructible default, so a replace-style write into a fresh branch thread or a never-written channel stored the wrapper literally and crashed the next consumer (#4380; probe-guarded, stands down if upstream fixes it)
 - `agents/thread_state.py` — `ThreadState`/`DeltaThreadState`, `DELTA_MESSAGES_FIELD` (`DeltaChannel` with `snapshot_frequency=1000`), schema adaptation helpers
 - `runtime/context_compaction.py` — compaction via accessor + mutation graph (reference consumer)
 - Tests: `tests/test_checkpoint_mode.py` (freeze/detect/gate), `tests/test_checkpoint_state.py` (accessor/mutation graph), `tests/test_delta_channel_checkpointers.py` (saver parity), `tests/test_threads_checkpoint_mode.py`, `tests/test_gateway_checkpoint_mode.py` (dual-mode e2e parity), `tests/test_context_compaction.py` (mutation-graph write, no scheduling), `tests/test_run_worker_rollback.py`
@@ -1125,9 +1192,10 @@ Automatic conversation summarization when approaching token limits:
 - Manual compaction uses `POST /api/threads/{id}/compact`, reuses the same
   `DeerFlowSummarizationMiddleware`, writes a new checkpoint with updated
   `messages` and `summary_text`, and bumps only those channel versions.
-  The route shares the per-thread serialization gate used by `/goal` writes
-  and run admission so compaction cannot race with goal updates or runs that
-  read/write checkpoints.
+  The route uses the shared `reserve_checkpoint_write()` boundary (also used by
+  manual state updates). Its short-lived `checkpoint_write` thread operation
+  shares the durable active-thread uniqueness constraint with run admission,
+  preventing either worker-local or cross-worker checkpoint-write races.
 
 See [docs/summarization.md](docs/summarization.md) for details.
 

backend/Dockerfile

@@ -13,6 +13,8 @@ FROM python:3.12-slim-bookworm AS builder
 ARG NODE_MAJOR=22
 ARG APT_MIRROR
 ARG UV_INDEX_URL
+ARG NPM_REGISTRY
+ARG LARK_CLI_NPM_VERSION=1.0.65
 # Optional extras to install (e.g. "postgres" for PostgreSQL support)
 # Usage: docker build --build-arg UV_EXTRAS=postgres,discord ...
 ARG UV_EXTRAS
@@ -36,6 +38,11 @@ RUN apt-get update && apt-get install -y \
     && apt-get install -y nodejs \
     && rm -rf /var/lib/apt/lists/*
 
+# Install the official Lark/Feishu CLI used by the managed Lark integration.
+RUN if [ -n "${NPM_REGISTRY}" ]; then npm config set registry "${NPM_REGISTRY}"; fi \
+    && npm install -g @larksuite/cli@${LARK_CLI_NPM_VERSION} \
+    && lark-cli --version
+
 # Install uv (source image overridable via UV_IMAGE build arg)
 COPY --from=uv-source /uv /uvx /usr/local/bin/
 
@@ -95,7 +102,10 @@ ENV PYTHONIOENCODING=utf-8
 COPY --from=builder /usr/bin/node /usr/bin/node
 COPY --from=builder /usr/lib/node_modules /usr/lib/node_modules
 RUN ln -s ../lib/node_modules/npm/bin/npm-cli.js /usr/bin/npm \
-    && ln -s ../lib/node_modules/npm/bin/npx-cli.js /usr/bin/npx
+    && ln -s ../lib/node_modules/npm/bin/npx-cli.js /usr/bin/npx \
+    && LARK_CLI_BIN="$(node -p 'const bin=require("/usr/lib/node_modules/@larksuite/cli/package.json").bin; typeof bin === "string" ? bin : (bin["lark-cli"] || Object.values(bin)[0])')" \
+    && ln -s "../lib/node_modules/@larksuite/cli/${LARK_CLI_BIN#./}" /usr/bin/lark-cli \
+    && lark-cli --version
 
 # Install Docker CLI (for DooD: allows starting sandbox containers via host Docker socket)
 COPY --from=docker:cli /usr/local/bin/docker /usr/local/bin/docker

backend/app/channels/manager.py

@@ -21,6 +21,7 @@ from langgraph_sdk.errors import ConflictError
 from app.channels import feishu_run_policy as _feishu_run_policy  # noqa: F401
 from app.channels.commands import KNOWN_CHANNEL_COMMANDS
 from app.channels.message_bus import (
+    INBOUND_FILE_CONTENT_KEY,
     PENDING_CLARIFICATION_METADATA_KEY,
     InboundMessage,
     InboundMessageType,
@@ -833,15 +834,21 @@ async def _ingest_inbound_files(thread_id: str, msg: InboundMessage, *, user_id:
             ftype = f.get("type") if isinstance(f.get("type"), str) else "file"
             filename = f.get("filename") if isinstance(f.get("filename"), str) else ""
 
-            try:
-                data = await file_reader(f, client)
-            except Exception:
-                logger.exception(
-                    "[Manager] failed to read inbound file: channel=%s, file=%s",
-                    msg.channel_name,
-                    f.get("url") or filename or idx,
-                )
-                continue
+            inline_content = f.pop(INBOUND_FILE_CONTENT_KEY, None)
+            if isinstance(inline_content, bytes):
+                data = inline_content
+            elif isinstance(inline_content, (bytearray, memoryview)):
+                data = bytes(inline_content)
+            else:
+                try:
+                    data = await file_reader(f, client)
+                except Exception:
+                    logger.exception(
+                        "[Manager] failed to read inbound file: channel=%s, file=%s",
+                        msg.channel_name,
+                        f.get("url") or filename or idx,
+                    )
+                    continue
 
             if data is None:
                 logger.warning(

backend/app/channels/message_bus.py

@@ -15,6 +15,9 @@ logger = logging.getLogger(__name__)
 
 PENDING_CLARIFICATION_METADATA_KEY = "pending_clarification"
 RESOLVED_FROM_PENDING_CLARIFICATION_METADATA_KEY = "resolved_from_pending_clarification"
+# Adapter-owned bytes may use this transient key while crossing the channel
+# boundary. ChannelManager consumes and removes it before persisting metadata.
+INBOUND_FILE_CONTENT_KEY = "_content"
 
 
 # ---------------------------------------------------------------------------

backend/app/channels/telegram.py

@@ -6,15 +6,29 @@ import asyncio
 import logging
 import threading
 import time
+from collections.abc import Coroutine
 from typing import Any
 
 from app.channels.base import Channel
 from app.channels.connection_identity import attach_connection_identity
-from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
+from app.channels.message_bus import (
+    INBOUND_FILE_CONTENT_KEY,
+    InboundMessage,
+    InboundMessageType,
+    MessageBus,
+    OutboundMessage,
+    ResolvedAttachment,
+)
+from deerflow.uploads.manager import is_upload_staging_file, normalize_filename
 
 logger = logging.getLogger(__name__)
 
 TELEGRAM_MAX_MESSAGE_LENGTH = 4096
+# Telegram's hosted Bot API documents this as 20 MB (decimal bytes).
+TELEGRAM_MAX_INBOUND_FILE_BYTES = 20_000_000
+# Keep Telegram cleanup inside the Gateway's five-second shutdown-hook budget.
+TELEGRAM_SHUTDOWN_TIMEOUT_SECONDS = 4.0
+TELEGRAM_BRIDGE_DRAIN_TIMEOUT_SECONDS = 1.0
 STREAM_EDIT_MIN_INTERVAL_SECONDS = 1.0
 # Groups (negative chat_id) are capped at 20 messages/minute by Telegram,
 # so stream edits there must pace well below the private-chat 1 msg/s guideline.
@@ -41,6 +55,9 @@ class TelegramChannel(Channel):
         self._thread: threading.Thread | None = None
         self._tg_loop: asyncio.AbstractEventLoop | None = None
         self._main_loop: asyncio.AbstractEventLoop | None = None
+        # Tasks submitted from the main dispatcher loop back to PTB's loop.
+        # Only the Telegram loop mutates this set.
+        self._tg_bridge_tasks: set[asyncio.Task[Any]] = set()
         self._allowed_users: set[int] = set()
         for uid in config.get("allowed_users", []):
             try:
@@ -96,6 +113,10 @@ class TelegramChannel(Channel):
         # General message handler
         app.add_handler(MessageHandler(filters.TEXT & ~filters.COMMAND, self._on_text))
 
+        # Telegram keeps attachment captions separate from message.text, and
+        # photo/document updates do not match filters.TEXT.
+        app.add_handler(MessageHandler(filters.PHOTO | filters.Document.ALL, self._on_text))
+
         self._application = app
 
         # Run polling in a dedicated thread with its own event loop
@@ -106,12 +127,47 @@ class TelegramChannel(Channel):
     async def stop(self) -> None:
         self._running = False
         self.bus.unsubscribe_outbound(self._on_outbound)
-        if self._tg_loop and self._tg_loop.is_running():
-            self._tg_loop.call_soon_threadsafe(self._tg_loop.stop)
-        if self._thread:
-            self._thread.join(timeout=10)
-            self._thread = None
-        self._application = None
+        shutdown_loop = asyncio.get_running_loop()
+        deadline = shutdown_loop.time() + TELEGRAM_SHUTDOWN_TIMEOUT_SECONDS
+        telegram_loop = self._tg_loop
+        worker_thread = self._thread
+
+        try:
+            if telegram_loop and telegram_loop.is_running():
+                drain_future = asyncio.run_coroutine_threadsafe(self._cancel_telegram_bridge_tasks(), telegram_loop)
+                try:
+                    remaining = max(0.0, deadline - shutdown_loop.time())
+                    await asyncio.wait_for(asyncio.wrap_future(drain_future), timeout=remaining)
+                except asyncio.CancelledError:
+                    drain_future.cancel()
+                    raise
+                except TimeoutError:
+                    drain_future.cancel()
+                    logger.warning("[Telegram] timed out cancelling inbound file downloads during shutdown")
+                except Exception as exc:
+                    logger.warning("[Telegram] failed to cancel inbound file downloads during shutdown: %s", type(exc).__name__)
+        finally:
+            if telegram_loop and telegram_loop.is_running():
+                try:
+                    telegram_loop.call_soon_threadsafe(telegram_loop.stop)
+                except RuntimeError:
+                    pass
+            try:
+                if worker_thread is not None and worker_thread.is_alive():
+                    remaining = max(0.0, deadline - shutdown_loop.time())
+                    if remaining:
+                        try:
+                            await asyncio.wait_for(
+                                asyncio.to_thread(worker_thread.join, remaining),
+                                timeout=remaining,
+                            )
+                        except TimeoutError:
+                            logger.warning("[Telegram] polling thread did not stop within the shutdown budget")
+            finally:
+                if worker_thread is not None and worker_thread.is_alive():
+                    logger.warning("[Telegram] polling thread is still exiting after bounded shutdown")
+                self._thread = None
+                self._application = None
         logger.info("Telegram channel stopped")
 
     async def send(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
@@ -293,8 +349,137 @@ class TelegramChannel(Channel):
             logger.exception("[Telegram] failed to send file: %s", attachment.filename)
             return False
 
+    async def receive_file(self, msg: InboundMessage, thread_id: str, *, user_id: str | None = None) -> InboundMessage:
+        """Download inbound Telegram attachments for the shared upload pipeline.
+
+        The Bot API download URL contains the bot token, so this adapter never
+        exposes that URL to ``InboundMessage`` or logs. Instead it hands the
+        downloaded bytes to ``ChannelManager`` through a short-lived private
+        field that the manager consumes before persisting safe upload metadata.
+        """
+        # Owner/thread scoping is intentionally applied by ChannelManager when
+        # it persists these bytes through _ingest_inbound_files().
+        del thread_id, user_id
+        if not msg.files:
+            return msg
+
+        bot = self._application.bot if self._application is not None else None
+        materialized: list[dict[str, Any]] = []
+        unavailable: list[str] = []
+
+        for file_info in msg.files:
+            if not isinstance(file_info, dict):
+                continue
+
+            filename = self._safe_inbound_filename(file_info.get("filename"), "attachment")
+            file_id = file_info.get("file_id") if isinstance(file_info.get("file_id"), str) else ""
+            declared_size = self._file_size(file_info.get("size"))
+            if declared_size is not None and declared_size > TELEGRAM_MAX_INBOUND_FILE_BYTES:
+                logger.warning("[Telegram] inbound file exceeds 20 MB download limit, skipping: %s", filename)
+                unavailable.append(f"{filename} (exceeds the 20 MB download limit)")
+                continue
+
+            if bot is None or not file_id:
+                logger.error("[Telegram] cannot download inbound file: %s", filename)
+                unavailable.append(f"{filename} (download unavailable)")
+                continue
+
+            try:
+                resolved_size, content = await self._run_on_telegram_loop(self._download_inbound_file(bot, file_id))
+            except Exception as exc:
+                # Exception strings from HTTP clients can contain request URLs.
+                # Log only the class name so a Bot API token can never leak.
+                logger.error("[Telegram] failed to download inbound file %s: %s", filename, type(exc).__name__)
+                unavailable.append(f"{filename} (download failed)")
+                continue
+
+            if content is None:
+                logger.warning("[Telegram] resolved inbound file exceeds 20 MB download limit, skipping: %s", filename)
+                unavailable.append(f"{filename} (exceeds the 20 MB download limit)")
+                continue
+
+            if len(content) > TELEGRAM_MAX_INBOUND_FILE_BYTES:
+                logger.warning("[Telegram] downloaded inbound file exceeds 20 MB limit, skipping: %s", filename)
+                unavailable.append(f"{filename} (exceeds the 20 MB download limit)")
+                continue
+
+            materialized.append(
+                {
+                    "type": "image" if file_info.get("type") == "image" else "file",
+                    "filename": filename,
+                    "mime_type": file_info.get("mime_type") if isinstance(file_info.get("mime_type"), str) else "application/octet-stream",
+                    "size": len(content),
+                    INBOUND_FILE_CONTENT_KEY: content,
+                }
+            )
+
+        msg.files = materialized
+        if unavailable:
+            notice = f"[Telegram attachment unavailable: {', '.join(unavailable)}]"
+            msg.text = f"{msg.text}\n\n{notice}" if msg.text else notice
+        return msg
+
     # -- helpers -----------------------------------------------------------
 
+    async def _download_inbound_file(self, bot: Any, file_id: str) -> tuple[int | None, bytearray | None]:
+        """Fetch one file entirely on the event loop that owns PTB's HTTP client."""
+        telegram_file = await bot.get_file(file_id)
+        resolved_size = self._file_size(getattr(telegram_file, "file_size", None))
+        if resolved_size is not None and resolved_size > TELEGRAM_MAX_INBOUND_FILE_BYTES:
+            return resolved_size, None
+        return resolved_size, await telegram_file.download_as_bytearray()
+
+    async def _track_telegram_bridge_task(self, coroutine: Coroutine[Any, Any, Any]) -> Any:
+        """Keep a main-loop submission visible to Telegram-loop shutdown."""
+        task = asyncio.current_task()
+        if task is None:
+            coroutine.close()
+            raise RuntimeError("Telegram bridge task is unavailable")
+        self._tg_bridge_tasks.add(task)
+        try:
+            return await coroutine
+        finally:
+            self._tg_bridge_tasks.discard(task)
+
+    async def _cancel_telegram_bridge_tasks(self) -> None:
+        """Cancel and drain cross-loop PTB work before its event loop exits."""
+        current = asyncio.current_task()
+        pending = [task for task in self._tg_bridge_tasks if task is not current and not task.done()]
+        for task in pending:
+            task.cancel()
+        if pending:
+            done, still_pending = await asyncio.wait(pending, timeout=TELEGRAM_BRIDGE_DRAIN_TIMEOUT_SECONDS)
+            if done:
+                await asyncio.gather(*done, return_exceptions=True)
+            if still_pending:
+                logger.warning("[Telegram] %d inbound file download task(s) did not cancel promptly", len(still_pending))
+
+    async def _run_on_telegram_loop(self, coroutine: Coroutine[Any, Any, Any]) -> Any:
+        """Await a PTB coroutine without using its HTTP client across event loops."""
+        telegram_loop = self._tg_loop
+        current_loop = asyncio.get_running_loop()
+        if telegram_loop is None or telegram_loop is current_loop:
+            return await coroutine
+
+        if not telegram_loop.is_running():
+            coroutine.close()
+            raise RuntimeError("Telegram event loop is not running")
+
+        tracked_coroutine = self._track_telegram_bridge_task(coroutine)
+        try:
+            future = asyncio.run_coroutine_threadsafe(tracked_coroutine, telegram_loop)
+        except BaseException:
+            # A concurrent shutdown can close the loop between the running
+            # check and scheduling. Close the unscheduled coroutine explicitly.
+            tracked_coroutine.close()
+            coroutine.close()
+            raise
+        try:
+            return await asyncio.wrap_future(future)
+        except asyncio.CancelledError:
+            future.cancel()
+            raise
+
     @staticmethod
     def _stream_key(chat_id: str, thread_ts: str | None) -> str:
         return f"{chat_id}:{thread_ts or ''}"
@@ -303,8 +488,83 @@ class TelegramChannel(Channel):
     def _parse_message_id(value: str | None) -> int | None:
         try:
             return int(value) if value else None
+        except (OverflowError, TypeError, ValueError):
+            return None
+
+    @staticmethod
+    def _file_size(value: Any) -> int | None:
+        if isinstance(value, bool):
+            return None
+        try:
+            size = int(value)
         except (TypeError, ValueError):
             return None
+        return size if size >= 0 else None
+
+    @staticmethod
+    def _safe_inbound_filename(value: Any, fallback: str) -> str:
+        if not isinstance(value, str):
+            return fallback
+        try:
+            candidate = normalize_filename(value.strip())
+        except (UnicodeError, ValueError):
+            return fallback
+        if is_upload_staging_file(candidate) or any(ord(char) < 32 for char in candidate):
+            return fallback
+        return candidate
+
+    @classmethod
+    def _extract_inbound_files(cls, message: Any) -> list[dict[str, Any]]:
+        files: list[dict[str, Any]] = []
+        message_id = str(getattr(message, "message_id", "message"))
+
+        # Materialize the PTB sequence once. Test doubles and partially shaped
+        # update objects can expose a truthy but empty iterable here.
+        photo_sizes = tuple(getattr(message, "photo", None) or ())
+        if photo_sizes:
+            photo = max(
+                photo_sizes,
+                key=lambda item: (
+                    (cls._file_size(getattr(item, "width", None)) or 0) * (cls._file_size(getattr(item, "height", None)) or 0),
+                    cls._file_size(getattr(item, "file_size", None)) or 0,
+                ),
+            )
+            file_id = getattr(photo, "file_id", None)
+            if isinstance(file_id, str) and file_id:
+                file_unique_id = getattr(photo, "file_unique_id", None)
+                files.append(
+                    {
+                        "type": "image",
+                        "file_id": file_id,
+                        "file_unique_id": file_unique_id if isinstance(file_unique_id, str) else None,
+                        "filename": f"telegram-photo-{message_id}.jpg",
+                        "mime_type": "image/jpeg",
+                        "size": cls._file_size(getattr(photo, "file_size", None)),
+                    }
+                )
+
+        document = getattr(message, "document", None)
+        document_id = getattr(document, "file_id", None) if document is not None else None
+        if isinstance(document_id, str) and document_id:
+            file_unique_id = getattr(document, "file_unique_id", None)
+            mime_type = getattr(document, "mime_type", None)
+            if not isinstance(mime_type, str) or not mime_type:
+                mime_type = "application/octet-stream"
+            # Avoid the lazy system MIME database lookup on the Telegram event
+            # loop. The original name is preferred; an opaque extension is safe.
+            fallback = f"telegram-document-{message_id}.bin"
+            files.append(
+                {
+                    "type": "file",
+                    "file_id": document_id,
+                    "file_unique_id": file_unique_id if isinstance(file_unique_id, str) else None,
+                    "filename": cls._safe_inbound_filename(getattr(document, "file_name", None), fallback),
+                    "mime_type": mime_type,
+                    "size": cls._file_size(getattr(document, "file_size", None)),
+                }
+            )
+
+        return files
 
     def _register_stream_message(self, key: str, *, message_id: int, last_text: str, last_edit_at: float) -> None:
         self._stream_messages.pop(key, None)
@@ -358,15 +618,18 @@ class TelegramChannel(Channel):
 
     def _run_polling(self) -> None:
         """Run telegram polling in a dedicated thread."""
+        application = self._application
+        if application is None:
+            return
         self._tg_loop = asyncio.new_event_loop()
         asyncio.set_event_loop(self._tg_loop)
         try:
             # Cannot use run_polling() because it calls add_signal_handler(),
             # which only works in the main thread.  Instead, manually
             # initialize the application and start the updater.
-            self._tg_loop.run_until_complete(self._application.initialize())
-            self._tg_loop.run_until_complete(self._application.start())
-            self._tg_loop.run_until_complete(self._application.updater.start_polling())
+            self._tg_loop.run_until_complete(application.initialize())
+            self._tg_loop.run_until_complete(application.start())
+            self._tg_loop.run_until_complete(application.updater.start_polling())
             self._tg_loop.run_forever()
         except Exception:
             if self._running:
@@ -374,10 +637,11 @@ class TelegramChannel(Channel):
         finally:
             # Graceful shutdown
             try:
-                if self._application.updater.running:
-                    self._tg_loop.run_until_complete(self._application.updater.stop())
-                self._tg_loop.run_until_complete(self._application.stop())
-                self._tg_loop.run_until_complete(self._application.shutdown())
+                self._tg_loop.run_until_complete(self._cancel_telegram_bridge_tasks())
+                if application.updater.running:
+                    self._tg_loop.run_until_complete(application.updater.stop())
+                self._tg_loop.run_until_complete(application.stop())
+                self._tg_loop.run_until_complete(application.shutdown())
             except Exception:
                 logger.exception("Error during Telegram shutdown")
 
@@ -517,12 +781,19 @@ class TelegramChannel(Channel):
             logger.warning("[Telegram] Main loop not running. Cannot publish inbound message.")
 
     async def _on_text(self, update, context) -> None:
-        """Handle regular text messages."""
+        """Handle regular text, photo, and document messages."""
         if not self._check_user(update.effective_user.id):
             return
 
-        text = self._strip_bot_username_from_leading_command(update.message.text.strip(), self._get_bot_username(context))
-        if not text:
+        message = update.message
+        message_text = getattr(message, "text", None)
+        caption = getattr(message, "caption", None)
+        raw_text = message_text if isinstance(message_text, str) else caption if isinstance(caption, str) else ""
+        text = raw_text.strip()
+        if text:
+            text = self._strip_bot_username_from_leading_command(text, self._get_bot_username(context))
+        files = self._extract_inbound_files(message)
+        if not text and not files:
             return
 
         chat_id = str(update.effective_chat.id)
@@ -549,6 +820,7 @@ class TelegramChannel(Channel):
             text=text,
             msg_type=InboundMessageType.CHAT,
             thread_ts=msg_id,
+            files=files,
             metadata={"message_id": msg_id},
         )
         inbound.topic_id = topic_id

backend/app/gateway/app.py

@@ -25,6 +25,7 @@ from app.gateway.routers import (
     feedback,
     github_webhooks,
     input_polish,
+    integrations,
     mcp,
     memory,
     models,
@@ -516,6 +517,9 @@ This gateway provides runtime endpoints for agent runs plus custom endpoints for
     # Skills API is mounted at /api/skills
     app.include_router(skills.router)
 
+    # First-party integrations API is mounted at /api/integrations
+    app.include_router(integrations.router)
+
     # Artifacts API is mounted at /api/threads/{thread_id}/artifacts
     app.include_router(artifacts.router)
 

backend/app/gateway/auth/repositories/base.py

@@ -30,7 +30,10 @@ class UserRepository(ABC):
             user: User object to create
 
         Returns:
-            Created User with ID assigned
+            Created User with ID assigned. ``email`` is the canonical
+            (lowercase) stored form, which may differ in case from what was
+            passed in -- implementations mutate the input ``user`` in place
+            to reflect this rather than returning a fresh object.
 
         Raises:
             ValueError: If email already exists
@@ -69,7 +72,9 @@ class UserRepository(ABC):
             user: User object with updated fields
 
         Returns:
-            Updated User
+            Updated User. ``email`` is the canonical (lowercase) stored
+            form -- implementations mutate the input ``user`` in place to
+            reflect this rather than returning a fresh object.
 
         Raises:
             UserNotFoundError: If no row exists for ``user.id``. This is

backend/app/gateway/auth/repositories/sqlite.py

@@ -24,6 +24,25 @@ from app.gateway.auth.repositories.base import UserNotFoundError, UserRepository
 from deerflow.persistence.user.model import UserRow
 
 
+def _normalize_email(email: str) -> str:
+    """Canonicalise an email address for storage and lookup.
+
+    An email identifies exactly one account regardless of the case the client
+    sends. The two write paths would otherwise disagree: local registration
+    normalises through ``EmailStr``, which lowercases only the *domain* and
+    keeps the local-part case (``Victim@X.COM`` -> ``Victim@x.com``), while OIDC
+    provisioning lowercases the whole address (``-> victim@x.com``). Combined
+    with the previous case-sensitive lookup, ``Victim@x.com`` and
+    ``victim@x.com`` resolved to two separate rows, defeating the invariant
+    that a local account blocks an SSO login on the same email.
+
+    Canonicalising to lowercase at every write site and matching
+    case-insensitively on read closes that gap for new accounts while letting
+    existing mixed-case rows keep resolving, without a destructive bulk rewrite.
+    """
+    return email.lower()
+
+
 class SQLiteUserRepository(UserRepository):
     """Async user repository backed by the shared SQLAlchemy engine."""
 
@@ -65,9 +84,20 @@ class SQLiteUserRepository(UserRepository):
     # ── CRUD ──────────────────────────────────────────────────────────
 
     async def create_user(self, user: User) -> User:
-        """Insert a new user. Raises ``ValueError`` on duplicate email."""
+        """Insert a new user. Raises ``ValueError`` on duplicate email.
+
+        The email is canonicalised to lowercase before insert so the existing
+        unique constraint enforces case-insensitive uniqueness for new rows and
+        the returned ``User`` reflects the stored form.
+        """
+        user.email = _normalize_email(user.email)
         row = self._user_to_row(user)
         async with self._sf() as session:
+            # The unique constraint is case-sensitive, so it cannot catch a
+            # canonical address colliding with a mixed-case legacy row.
+            existing = select(UserRow.id).where(func.lower(UserRow.email) == user.email).limit(1)
+            if await session.scalar(existing) is not None:
+                raise ValueError(f"Email already registered: {user.email}")
             session.add(row)
             try:
                 await session.commit()
@@ -82,10 +112,17 @@ class SQLiteUserRepository(UserRepository):
             return self._row_to_user(row) if row is not None else None
 
     async def get_user_by_email(self, email: str) -> User | None:
-        stmt = select(UserRow).where(UserRow.email == email)
+        # Case-insensitive match: an account is keyed by its email regardless of
+        # the case the caller supplies (see ``_normalize_email``). ``.first()``
+        # with a deterministic ``created_at`` ordering resolves to the oldest
+        # account instead of raising if a pre-fix database already holds two
+        # rows differing only in case, so the fix never turns a legacy duplicate
+        # pair into a 500. ``id`` is a secondary tiebreaker so the choice stays
+        # deterministic even if two legacy rows share the same ``created_at``.
+        stmt = select(UserRow).where(func.lower(UserRow.email) == _normalize_email(email)).order_by(UserRow.created_at, UserRow.id).limit(1)
         async with self._sf() as session:
             result = await session.execute(stmt)
-            row = result.scalar_one_or_none()
+            row = result.scalars().first()
             return self._row_to_user(row) if row is not None else None
 
     async def update_user(self, user: User) -> User:
@@ -99,7 +136,24 @@ class SQLiteUserRepository(UserRepository):
                 # success would let the caller log "password reset" for
                 # a row that no longer exists.
                 raise UserNotFoundError(f"User {user.id} no longer exists")
-            row.email = user.email
+            # Canonicalise the email only when it actually changes, comparing
+            # case-insensitively against the stored value, then mirror the
+            # persisted value back onto the returned object. Re-lowercasing an
+            # *unchanged* legacy mixed-case email — e.g. a password-only update
+            # on a pre-fix ``Victim@x.com`` row while a canonical ``victim@x.com``
+            # row also exists — would rewrite it onto the other row's unique
+            # email and raise IntegrityError, surfacing as a 500 on the
+            # change-password / reset-admin paths that do not catch it. Guarding
+            # against the *raw* stored value (``canonical != row.email``) is not
+            # enough: the mixed-case row's canonical form still differs from its
+            # own stored casing, so it would rewrite and collide anyway. A genuine
+            # change still normalises, so the unique constraint keeps enforcing
+            # case-insensitive uniqueness for updated rows; get_user_by_email
+            # already resolves legacy mixed-case rows case-insensitively on read.
+            canonical_email = _normalize_email(user.email)
+            if canonical_email != _normalize_email(row.email):
+                row.email = canonical_email
+            user.email = row.email
             row.password_hash = user.password_hash
             row.system_role = user.system_role
             row.oauth_provider = user.oauth_provider

backend/app/gateway/deps.py

@@ -58,14 +58,16 @@ def _browser_tools_enabled_in_config(config: AppConfig) -> bool:
 def _enforce_postgres_for_multi_worker(config: AppConfig) -> None:
     """Refuse unsafe multi-worker configurations before persistence starts.
 
-    Three checks (all must pass for multi-worker):
+    Four checks (all must pass for multi-worker):
 
     1. Process-local browser sessions must be disabled. Browser tools keep
        Chromium and Playwright objects in one worker's memory, while ordinary
        uvicorn dispatch provides no thread-id affinity.
     2. The DB backend must be Postgres — SQLite write-locks cannot support
        concurrent multi-process access.
-    3. ``run_ownership.heartbeat_enabled`` must be True — without heartbeat,
+    3. ``run_events.backend`` must be ``db``. Memory and JSONL stores are
+       process-local, so workers cannot enforce a shared singleton receipt.
+    4. ``run_ownership.heartbeat_enabled`` must be True — without heartbeat,
        every run has a NULL lease, so reconciliation treats all inflight
        runs as orphans and Worker B would kill Worker A's live runs on
        every rolling update or scale-up.
@@ -89,6 +91,14 @@ def _enforce_postgres_for_multi_worker(config: AppConfig) -> None:
     if backend != "postgres":
         raise SystemExit(f"GATEWAY_WORKERS={workers} requires database.backend='postgres', but database.backend is '{backend}'. SQLite cannot support concurrent multi-process access. Set GATEWAY_WORKERS=1 or switch to Postgres.")
 
+    run_events_backend = getattr(getattr(config, "run_events", None), "backend", None)
+    if run_events_backend != "db":
+        raise SystemExit(
+            f"GATEWAY_WORKERS={workers} requires run_events.backend='db', but run_events.backend is '{run_events_backend}'. "
+            "Memory and JSONL event stores are process-local, so delivery receipt singleton guarantees cannot hold across workers. "
+            "Set GATEWAY_WORKERS=1 or configure run_events.backend: db."
+        )
+
     run_ownership = getattr(config, "run_ownership", None)
     if run_ownership is None or not run_ownership.heartbeat_enabled:
         raise SystemExit(
@@ -457,6 +467,7 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen
         app.state.run_manager = RunManager(
             store=app.state.run_store,
             run_ownership_config=run_ownership_config,
+            event_store=app.state.run_event_store,
             on_orphans_recovered=terminalize_recovered_runs,
         )
         # Startup recovery: mark inflight runs whose lease has expired as error.

backend/app/gateway/routers/console.py

@@ -272,7 +272,9 @@ async def console_stats(request: Request) -> ConsoleStatsResponse:
     """Return the dashboard's headline counters."""
     sf = _session_factory_or_503()
     user_id = await get_current_user(request)
-    run_where = (RunRow.user_id == user_id,) if user_id else ()
+    run_where = (RunRow.operation_kind == "run",)
+    if user_id:
+        run_where += (RunRow.user_id == user_id,)
     thread_where = (ThreadMetaRow.user_id == user_id,) if user_id else ()
 
     pricing = _build_pricing_map()
@@ -347,7 +349,14 @@ async def console_runs(
     sf = _session_factory_or_503()
     user_id = await get_current_user(request)
 
-    stmt = select(RunRow, ThreadMetaRow.display_name).join(ThreadMetaRow, ThreadMetaRow.thread_id == RunRow.thread_id, isouter=True).order_by(RunRow.created_at.desc(), RunRow.run_id.desc()).limit(limit + 1).offset(offset)
+    stmt = (
+        select(RunRow, ThreadMetaRow.display_name)
+        .join(ThreadMetaRow, ThreadMetaRow.thread_id == RunRow.thread_id, isouter=True)
+        .where(RunRow.operation_kind == "run")
+        .order_by(RunRow.created_at.desc(), RunRow.run_id.desc())
+        .limit(limit + 1)
+        .offset(offset)
+    )
     if user_id:
         stmt = stmt.where(RunRow.user_id == user_id)
     if status:
@@ -415,7 +424,7 @@ async def console_usage(
     start_local = today_local - timedelta(days=days - 1)
     window_start_utc = datetime.combine(start_local, time.min, tzinfo=UTC) - tz_delta
 
-    stmt = select(RunRow).where(RunRow.created_at >= window_start_utc)
+    stmt = select(RunRow).where(RunRow.operation_kind == "run", RunRow.created_at >= window_start_utc)
     if user_id:
         stmt = stmt.where(RunRow.user_id == user_id)
 

backend/app/gateway/routers/integrations.py

@@ -0,0 +1,354 @@
+import asyncio
+import logging
+
+from fastapi import APIRouter, Depends, HTTPException, Request
+from pydantic import BaseModel, Field
+
+from app.gateway.deps import get_config, require_admin_user
+from deerflow.agents.lead_agent.prompt import refresh_skills_system_prompt_cache_async
+from deerflow.config.app_config import AppConfig
+from deerflow.integrations.lark_cli import (
+    LARK_AUTH_COMPLETE_DEFAULT_WAIT_SECONDS,
+    LARK_AUTH_COMPLETE_MAX_WAIT_SECONDS,
+    LARK_AUTH_COMPLETE_MIN_WAIT_SECONDS,
+    LarkAuthCompleteResult,
+    LarkAuthProbe,
+    LarkAuthStartResult,
+    LarkCliProbe,
+    LarkConfigCompleteResult,
+    LarkConfigStartResult,
+    LarkInstallResult,
+    LarkIntegrationStatus,
+    complete_lark_auth,
+    complete_lark_config,
+    get_lark_integration_status,
+    install_lark_integration,
+    start_lark_auth,
+    start_lark_config,
+)
+from deerflow.runtime.user_context import get_effective_user_id
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/api/integrations", tags=["integrations"])
+
+_ADMIN_REQUIRED_DETAIL = "Admin privileges required to install integrations."
+
+
+async def _is_admin_user(request: Request) -> bool:
+    """Non-raising admin check used to gate host-path disclosure in responses.
+
+    Fails closed: any error (missing middleware state, auth failure) is treated
+    as non-admin so host paths are redacted rather than accidentally exposed.
+    """
+    try:
+        await require_admin_user(request, detail=_ADMIN_REQUIRED_DETAIL)
+    except Exception:
+        return False
+    return True
+
+
+class LarkCliProbeResponse(BaseModel):
+    available: bool = Field(..., description="Whether lark-cli is available to the Gateway, either managed by DeerFlow or on PATH")
+    path: str | None = Field(None, description="Resolved lark-cli executable path")
+    version: str | None = Field(None, description="lark-cli --version output")
+    error: str | None = Field(None, description="Probe failure message")
+
+
+class LarkAuthProbeResponse(BaseModel):
+    status: str = Field(..., description="Auth status: authenticated, not_configured, unavailable, or error")
+    message: str | None = Field(None, description="Human-readable status detail")
+    user: str | None = Field(None, description="Authenticated Lark/Feishu user display value when available")
+    verified: bool = Field(False, description="Whether this status came from a live token verification")
+
+
+class LarkIntegrationStatusResponse(BaseModel):
+    installed: bool = Field(..., description="Whether the managed Lark skill pack is installed")
+    version: str = Field(..., description="Installed Lark CLI skill-pack version (from manifest, resolved at install time)")
+    manifest_version: str | None = Field(None, description="Installed manifest version")
+    latest_available_version: str | None = Field(None, description="Newest larksuite/cli release version available on GitHub, when known")
+    runtime_version_mismatch: bool = Field(False, description="Whether the installed skill-pack version differs from the Gateway runtime lark-cli binary")
+    app_configured: bool = Field(..., description="Whether lark-cli has app_id/app_secret configured for this user")
+    app_id: str | None = Field(None, description="Configured Lark app ID")
+    app_brand: str | None = Field(None, description="Configured Lark brand: feishu or lark")
+    skills_expected: int = Field(..., description="Number of skills expected in the official pack")
+    skills_installed: int = Field(..., description="Number of installed managed Lark skills")
+    installed_skills: list[str] = Field(default_factory=list, description="Installed managed Lark skill names")
+    enabled_skills: list[str] = Field(default_factory=list, description="Installed Lark skills currently enabled for this user")
+    install_path: str = Field(..., description="Host path of the managed Lark skill pack")
+    cli: LarkCliProbeResponse
+    auth: LarkAuthProbeResponse
+    sandbox_runtime_mode: str = Field("none", description="How lark-cli is provisioned into the sandbox: none, gateway-download, or init-container")
+    sandbox_runtime_ready: bool = Field(False, description="Whether the sandbox lark-cli runtime is provisioned and usable at chat time")
+    sandbox_runtime_detail: str | None = Field(None, description="Human-readable reason when the sandbox runtime is not ready")
+
+
+class LarkInstallResponse(BaseModel):
+    success: bool
+    installed_skills: list[str]
+    message: str
+    status: LarkIntegrationStatusResponse
+
+
+class LarkAuthStartRequest(BaseModel):
+    recommend: bool = Field(default=False, description="Request the official recommended auto-approve scopes")
+    domains: list[str] = Field(default_factory=list, description="Optional Lark auth domains, e.g. calendar or docs")
+    scope: str | None = Field(default=None, description="Optional explicit OAuth scope string")
+
+
+class LarkConfigStartRequest(BaseModel):
+    brand: str = Field(default="feishu", description="Lark brand to start app registration for: feishu or lark")
+
+
+class LarkConfigStartResponse(BaseModel):
+    verification_url: str = Field(..., description="URL the user should open in a browser to configure the Lark app")
+    device_code: str = Field(..., description="Device code used by config/complete after browser approval")
+    expires_in: int | None = Field(None, description="Seconds before the configuration URL expires")
+    interval: int | None = Field(None, description="Suggested polling interval from Lark")
+    user_code: str | None = Field(None, description="Optional user code shown by Lark")
+    brand: str = Field(..., description="Brand used for this app registration flow")
+
+
+class LarkConfigCompleteRequest(BaseModel):
+    device_code: str = Field(..., description="Device code returned by config/start")
+    brand: str = Field(default="feishu", description="Brand returned by config/start")
+    interval: int | None = Field(default=None, description="Polling interval returned by config/start")
+    expires_in: int | None = Field(default=None, description="Expiration returned by config/start")
+
+
+class LarkConfigCompleteResponse(BaseModel):
+    success: bool
+    message: str
+    status: LarkIntegrationStatusResponse
+
+
+class LarkAuthStartResponse(BaseModel):
+    verification_url: str = Field(..., description="URL the user should open in a browser to authorize")
+    device_code: str = Field(..., description="Device code used by the complete endpoint after browser approval")
+    expires_in: int | None = Field(None, description="Seconds before the authorization URL expires")
+    user_code: str | None = Field(None, description="Optional user code shown by Lark")
+    hint: str | None = Field(None, description="Optional guidance returned by lark-cli")
+
+
+class LarkAuthCompleteRequest(BaseModel):
+    device_code: str = Field(..., description="Device code returned by auth/start")
+    wait_timeout_seconds: int = Field(
+        default=LARK_AUTH_COMPLETE_DEFAULT_WAIT_SECONDS,
+        ge=LARK_AUTH_COMPLETE_MIN_WAIT_SECONDS,
+        le=LARK_AUTH_COMPLETE_MAX_WAIT_SECONDS,
+        description="Maximum seconds for this device-code poll; automatic UI polling uses a shorter wait",
+    )
+
+
+class LarkAuthCompleteResponse(BaseModel):
+    success: bool
+    message: str
+    status: LarkIntegrationStatusResponse
+
+
+def _cli_probe_to_response(probe: LarkCliProbe) -> LarkCliProbeResponse:
+    return LarkCliProbeResponse(
+        available=probe.available,
+        path=probe.path,
+        version=probe.version,
+        error=probe.error,
+    )
+
+
+def _auth_probe_to_response(probe: LarkAuthProbe) -> LarkAuthProbeResponse:
+    return LarkAuthProbeResponse(
+        status=probe.status,
+        message=probe.message,
+        user=probe.user,
+        verified=probe.verified,
+    )
+
+
+def _status_to_response(status: LarkIntegrationStatus, *, include_host_paths: bool = True) -> LarkIntegrationStatusResponse:
+    cli = _cli_probe_to_response(status.cli)
+    if not include_host_paths:
+        # Host filesystem paths (Gateway layout) are admin-only info; redact them
+        # for non-admin callers of the otherwise non-gated status/complete routes.
+        cli = cli.model_copy(update={"path": None})
+    return LarkIntegrationStatusResponse(
+        installed=status.installed,
+        version=status.version,
+        manifest_version=status.manifest_version,
+        latest_available_version=status.latest_available_version,
+        runtime_version_mismatch=status.runtime_version_mismatch,
+        app_configured=status.app_configured,
+        app_id=status.app_id,
+        app_brand=status.app_brand,
+        skills_expected=status.skills_expected,
+        skills_installed=status.skills_installed,
+        installed_skills=list(status.installed_skills),
+        enabled_skills=list(status.enabled_skills),
+        install_path=status.install_path if include_host_paths else "",
+        cli=cli,
+        auth=_auth_probe_to_response(status.auth),
+        sandbox_runtime_mode=status.sandbox_runtime_mode,
+        sandbox_runtime_ready=status.sandbox_runtime_ready,
+        sandbox_runtime_detail=status.sandbox_runtime_detail,
+    )
+
+
+def _install_to_response(result: LarkInstallResult) -> LarkInstallResponse:
+    return LarkInstallResponse(
+        success=result.success,
+        installed_skills=list(result.installed_skills),
+        message=result.message,
+        status=_status_to_response(result.status),
+    )
+
+
+def _config_start_to_response(result: LarkConfigStartResult) -> LarkConfigStartResponse:
+    return LarkConfigStartResponse(
+        verification_url=result.verification_url,
+        device_code=result.device_code,
+        expires_in=result.expires_in,
+        interval=result.interval,
+        user_code=result.user_code,
+        brand=result.brand,
+    )
+
+
+def _config_complete_to_response(result: LarkConfigCompleteResult, *, include_host_paths: bool = True) -> LarkConfigCompleteResponse:
+    return LarkConfigCompleteResponse(
+        success=result.success,
+        message=result.message,
+        status=_status_to_response(result.status, include_host_paths=include_host_paths),
+    )
+
+
+def _auth_start_to_response(result: LarkAuthStartResult) -> LarkAuthStartResponse:
+    return LarkAuthStartResponse(
+        verification_url=result.verification_url,
+        device_code=result.device_code,
+        expires_in=result.expires_in,
+        user_code=result.user_code,
+        hint=result.hint,
+    )
+
+
+def _auth_complete_to_response(result: LarkAuthCompleteResult, *, include_host_paths: bool = True) -> LarkAuthCompleteResponse:
+    return LarkAuthCompleteResponse(
+        success=result.success,
+        message=result.message,
+        status=_status_to_response(result.status, include_host_paths=include_host_paths),
+    )
+
+
+@router.get("/lark/status", response_model=LarkIntegrationStatusResponse, summary="Get Lark/Feishu Integration Status")
+async def get_lark_status(request: Request, config: AppConfig = Depends(get_config)) -> LarkIntegrationStatusResponse:
+    try:
+        status = await asyncio.to_thread(get_lark_integration_status, get_effective_user_id(), config, check_latest=True, check_runtime=True)
+        return _status_to_response(status, include_host_paths=await _is_admin_user(request))
+    except Exception as e:
+        logger.error("Failed to get Lark integration status: %s", e, exc_info=True)
+        raise HTTPException(status_code=500, detail="Failed to get Lark integration status.")
+
+
+@router.post("/lark/install", response_model=LarkInstallResponse, summary="Install Lark/Feishu Skill Pack")
+async def install_lark(request: Request, config: AppConfig = Depends(get_config)) -> LarkInstallResponse:
+    await require_admin_user(request, detail=_ADMIN_REQUIRED_DETAIL)
+    try:
+        result = await asyncio.to_thread(install_lark_integration, get_effective_user_id(), config)
+        await refresh_skills_system_prompt_cache_async()
+        return _install_to_response(result)
+    except FileNotFoundError as e:
+        raise HTTPException(status_code=404, detail=str(e))
+    except ValueError as e:
+        raise HTTPException(status_code=400, detail=str(e))
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error("Failed to install Lark integration: %s", e, exc_info=True)
+        raise HTTPException(status_code=500, detail="Failed to install Lark integration.")
+
+
+@router.post("/lark/config/start", response_model=LarkConfigStartResponse, summary="Start Lark/Feishu App Configuration")
+async def start_lark_app_config(body: LarkConfigStartRequest) -> LarkConfigStartResponse:
+    try:
+        result = await asyncio.to_thread(
+            start_lark_config,
+            get_effective_user_id(),
+            brand=body.brand,
+        )
+        return _config_start_to_response(result)
+    except FileNotFoundError as e:
+        raise HTTPException(status_code=404, detail=str(e))
+    except ValueError as e:
+        raise HTTPException(status_code=400, detail=str(e))
+    except TimeoutError as e:
+        raise HTTPException(status_code=504, detail=str(e))
+    except Exception as e:
+        logger.error("Failed to start Lark connection setup: %s", e, exc_info=True)
+        raise HTTPException(status_code=500, detail="Failed to start Lark connection setup.")
+
+
+@router.post("/lark/config/complete", response_model=LarkConfigCompleteResponse, summary="Complete Lark/Feishu App Configuration")
+async def complete_lark_app_config(request: Request, body: LarkConfigCompleteRequest, config: AppConfig = Depends(get_config)) -> LarkConfigCompleteResponse:
+    try:
+        result = await asyncio.to_thread(
+            complete_lark_config,
+            get_effective_user_id(),
+            config,
+            device_code=body.device_code,
+            brand=body.brand,
+            interval=body.interval,
+            expires_in=body.expires_in,
+        )
+        return _config_complete_to_response(result, include_host_paths=await _is_admin_user(request))
+    except FileNotFoundError as e:
+        raise HTTPException(status_code=404, detail=str(e))
+    except ValueError as e:
+        raise HTTPException(status_code=400, detail=str(e))
+    except TimeoutError as e:
+        raise HTTPException(status_code=504, detail=str(e))
+    except Exception as e:
+        logger.error("Failed to complete Lark connection setup: %s", e, exc_info=True)
+        raise HTTPException(status_code=500, detail="Failed to complete Lark connection setup.")
+
+
+@router.post("/lark/auth/start", response_model=LarkAuthStartResponse, summary="Start Lark/Feishu Browser Authorization")
+async def start_lark_browser_auth(body: LarkAuthStartRequest) -> LarkAuthStartResponse:
+    try:
+        result = await asyncio.to_thread(
+            start_lark_auth,
+            get_effective_user_id(),
+            domains=tuple(body.domains),
+            scope=body.scope,
+            recommend=body.recommend,
+        )
+        return _auth_start_to_response(result)
+    except FileNotFoundError as e:
+        raise HTTPException(status_code=404, detail=str(e))
+    except ValueError as e:
+        raise HTTPException(status_code=400, detail=str(e))
+    except TimeoutError as e:
+        raise HTTPException(status_code=504, detail=str(e))
+    except Exception as e:
+        logger.error("Failed to start Lark authorization: %s", e, exc_info=True)
+        raise HTTPException(status_code=500, detail="Failed to start Lark authorization.")
+
+
+@router.post("/lark/auth/complete", response_model=LarkAuthCompleteResponse, summary="Complete Lark/Feishu Browser Authorization")
+async def complete_lark_browser_auth(request: Request, body: LarkAuthCompleteRequest, config: AppConfig = Depends(get_config)) -> LarkAuthCompleteResponse:
+    try:
+        result = await asyncio.to_thread(
+            complete_lark_auth,
+            get_effective_user_id(),
+            config,
+            device_code=body.device_code,
+            wait_timeout_seconds=body.wait_timeout_seconds,
+        )
+        return _auth_complete_to_response(result, include_host_paths=await _is_admin_user(request))
+    except FileNotFoundError as e:
+        raise HTTPException(status_code=404, detail=str(e))
+    except ValueError as e:
+        raise HTTPException(status_code=400, detail=str(e))
+    except TimeoutError as e:
+        raise HTTPException(status_code=504, detail=str(e))
+    except Exception as e:
+        logger.error("Failed to complete Lark authorization: %s", e, exc_info=True)
+        raise HTTPException(status_code=500, detail="Failed to complete Lark authorization.")

backend/app/gateway/routers/threads.py

@@ -32,18 +32,20 @@ from app.gateway.checkpoint_lineage import (
     find_checkpoint_before_message_chronologically,
     is_duration_only_checkpoint,
 )
-from app.gateway.deps import get_checkpointer, get_run_event_store, get_run_manager
+from app.gateway.deps import get_checkpointer, get_run_event_store
 from app.gateway.internal_auth import get_trusted_internal_owner_user_id
 from app.gateway.services import (
     build_checkpoint_state_accessor,
     build_checkpoint_state_mutation_accessor,
     build_thread_checkpoint_state_accessor,
     build_thread_checkpoint_state_mutation_accessor,
+    reserve_checkpoint_write,
 )
 from app.gateway.utils import sanitize_log_param
 from deerflow.agents.thread_state import THREAD_STATE_REDUCER_FIELDS
 from deerflow.config.paths import Paths, get_paths
 from deerflow.config.summarization_config import ContextSize
+from deerflow.persistence.thread_meta import THREAD_PINNED_METADATA_KEY
 from deerflow.runtime import serialize_channel_values_for_api
 from deerflow.runtime.checkpoint_mode import CheckpointModeMismatchError, CheckpointModeReconfigurationError
 from deerflow.runtime.checkpoint_state import graph_reducer_channels, graph_state_schema, graph_writable_channels
@@ -57,11 +59,11 @@ from deerflow.runtime.goal import (
     DEFAULT_MAX_GOAL_CONTINUATIONS,
     build_goal_state,
     ensure_thread_checkpoint,
-    goal_thread_lock,
     read_thread_goal,
     write_thread_goal,
 )
 from deerflow.runtime.journal import build_branch_history_seed_events
+from deerflow.runtime.runs.manager import ConflictError
 from deerflow.runtime.runs.worker import valid_duration_entry
 from deerflow.runtime.user_context import get_effective_user_id
 from deerflow.utils.file_io import run_file_io
@@ -96,6 +98,14 @@ def _checkpoint_mode_http_error(exc: Exception, thread_id: str) -> HTTPException
 _SERVER_RESERVED_METADATA_KEYS: frozenset[str] = frozenset({"owner_id", "user_id"})
 _SIDECAR_METADATA_KEY = "deerflow_sidecar"
 _BRANCH_METADATA_KEY = "deerflow_branch"
+# Thread-scoped runtime channels a branch must NOT inherit from its parent:
+# ``sandbox.sandbox_id`` binds path mappings and the release lifecycle to the
+# *parent* thread, so copying it would make the branch read/write the parent's
+# workspace (bypassing the per-branch user-data clone) and release the
+# parent's sandbox after its first run; the branch lazily acquires its own
+# sandbox keyed by its own thread_id instead. ``thread_data`` is recomputed
+# from the branch's thread_id by ThreadDataMiddleware on every run.
+_BRANCH_EXCLUDED_CHANNELS = frozenset({"sandbox", "thread_data"})
 _BRANCH_HISTORY_SCAN_LIMIT = 200
 _BRANCH_HISTORY_RAW_SCAN_LIMIT = _BRANCH_HISTORY_SCAN_LIMIT * 2
 
@@ -107,6 +117,11 @@ def _strip_reserved_metadata(metadata: dict[str, Any] | None) -> dict[str, Any]:
     return {k: v for k, v in metadata.items() if k not in _SERVER_RESERVED_METADATA_KEYS}
 
 
+def _is_pin_metadata_patch(metadata: dict[str, Any]) -> bool:
+    """Return True for the narrow pin/unpin PATCH shape."""
+    return set(metadata) == {THREAD_PINNED_METADATA_KEY} and isinstance(metadata.get(THREAD_PINNED_METADATA_KEY), bool)
+
+
 def _message_id(message: Any) -> str | None:
     if isinstance(message, dict):
         raw = message.get("id")
@@ -441,6 +456,7 @@ class ThreadCompactRequest(BaseModel):
     force: bool = Field(default=True, description="Run compaction even if automatic summarization thresholds are not met")
     keep: ContextSize | None = Field(default=None, description="Optional retention policy for this compaction only")
     agent_name: str | None = Field(default=None, max_length=128, description="Optional custom agent name for memory attribution")
+    model_name: str | None = Field(default=None, max_length=128, description="Optional model to summarize with; resolved request override -> custom-agent model -> default, mirroring run model selection")
 
 
 class ThreadCompactResponse(BaseModel):
@@ -828,6 +844,8 @@ async def branch_thread(thread_id: str, body: ThreadBranchRequest, request: Requ
     def branch_values(source_snapshot: Any) -> dict[str, Any]:
         values: dict[str, Any] = {}
         for key, value in dict(source_snapshot.values).items():
+            if key in _BRANCH_EXCLUDED_CHANNELS:
+                continue
             if key in branch_reducer_fields:
                 values[key] = Overwrite(list(value) if key == "messages" and isinstance(value, list) else value)
             else:
@@ -886,7 +904,7 @@ async def branch_thread(thread_id: str, body: ThreadBranchRequest, request: Requ
         seed_events = build_branch_history_seed_events(
             _checkpoint_messages(snapshot),
             thread_id=new_thread_id,
-            run_id=f"branch-seed-{new_thread_id}",
+            run_id_prefix=f"branch-seed-{new_thread_id}",
             parent_thread_id=thread_id,
         )
         if seed_events:
@@ -961,13 +979,17 @@ async def patch_thread(thread_id: str, body: ThreadPatchRequest, request: Reques
         raise HTTPException(status_code=404, detail=f"Thread {thread_id} not found")
 
     # ``body.metadata`` already stripped by ``ThreadPatchRequest._strip_reserved``.
+    # Pin/unpin is not conversation activity, so it must not bump ``updated_at``.
+    # Other metadata PATCH callers keep the public endpoint's existing recency
+    # contract unless they get their own explicit no-touch API surface.
+    touch = not _is_pin_metadata_patch(body.metadata)
     try:
-        await thread_store.update_metadata(thread_id, body.metadata)
+        await thread_store.update_metadata(thread_id, body.metadata, touch=touch)
     except Exception:
         logger.exception("Failed to patch thread %s", sanitize_log_param(thread_id))
         raise HTTPException(status_code=500, detail="Failed to update thread")
 
-    # Re-read to get the merged metadata + refreshed updated_at
+    # Re-read to get the merged metadata and the store's timestamp decision.
     record = await thread_store.get(thread_id) or record
     return ThreadResponse(
         thread_id=thread_id,
@@ -1053,13 +1075,17 @@ async def set_thread_goal(thread_id: str, body: ThreadGoalRequest, request: Requ
     this endpoint creates the missing thread checkpoint on demand.
     """
     checkpointer = get_checkpointer(request)
-    await _ensure_thread_for_goal(thread_id, request)
     try:
         goal = build_goal_state(body.objective, max_continuations=body.max_continuations)
-        async with goal_thread_lock(thread_id):
+        async with reserve_checkpoint_write(request, thread_id, user_id=get_effective_user_id()):
+            await _ensure_thread_for_goal(thread_id, request)
             await write_thread_goal(checkpointer, thread_id, goal, as_node="goal", create_if_missing=True)
     except ValueError as exc:
         raise HTTPException(status_code=422, detail=str(exc)) from exc
+    except ConflictError:
+        raise HTTPException(status_code=409, detail="Thread has a run in flight. Set the goal after the run finishes.") from None
+    except HTTPException:
+        raise
     except Exception:
         logger.exception("Failed to set goal for thread %s", sanitize_log_param(thread_id))
         raise HTTPException(status_code=500, detail="Failed to set thread goal") from None
@@ -1072,8 +1098,10 @@ async def clear_thread_goal(thread_id: str, request: Request) -> ThreadGoalRespo
     """Clear the active goal for a thread."""
     checkpointer = get_checkpointer(request)
     try:
-        async with goal_thread_lock(thread_id):
+        async with reserve_checkpoint_write(request, thread_id, user_id=get_effective_user_id()):
             await write_thread_goal(checkpointer, thread_id, None, as_node="goal")
+    except ConflictError:
+        raise HTTPException(status_code=409, detail="Thread has a run in flight. Clear the goal after the run finishes.") from None
     except LookupError:
         return ThreadGoalResponse(goal=None)
     except Exception:
@@ -1099,7 +1127,6 @@ def _thread_compact_response(result: ThreadCompactionResult) -> ThreadCompactRes
 @require_permission("threads", "write", owner_check=True, require_existing=True)
 async def compact_thread(thread_id: str, body: ThreadCompactRequest, request: Request) -> ThreadCompactResponse:
     """Manually summarize old thread context while preserving the visible history."""
-    run_manager = get_run_manager(request)
     # Compaction writes only base-schema channels (messages + summary_text);
     # every other channel — including middleware-contributed ones — is carried
     # forward by checkpoint fork inheritance, so the base-schema mutation
@@ -1114,9 +1141,7 @@ async def compact_thread(thread_id: str, body: ThreadCompactRequest, request: Re
         raise _checkpoint_mode_http_error(exc, thread_id) from exc
     keep = body.keep.to_tuple() if body.keep is not None else None
     try:
-        async with goal_thread_lock(thread_id):
-            if await run_manager.has_inflight(thread_id):
-                raise HTTPException(status_code=409, detail="Thread has a run in flight. Compact after the run finishes.")
+        async with reserve_checkpoint_write(request, thread_id, user_id=get_effective_user_id()):
             result = await compact_thread_context(
                 accessor,
                 thread_id,
@@ -1124,7 +1149,10 @@ async def compact_thread(thread_id: str, body: ThreadCompactRequest, request: Re
                 force=body.force,
                 user_id=get_effective_user_id(),
                 agent_name=body.agent_name,
+                model_name=body.model_name,
             )
+    except ConflictError:
+        raise HTTPException(status_code=409, detail="Thread has a run in flight. Compact after the run finishes.") from None
     except _CHECKPOINT_MODE_ERRORS as exc:
         raise _checkpoint_mode_http_error(exc, thread_id) from exc
     except ContextCompactionDisabled:
@@ -1232,12 +1260,15 @@ async def update_thread_state(thread_id: str, body: ThreadStateUpdateRequest, re
         reducer_fields = THREAD_STATE_REDUCER_FIELDS
     updates = {key: Overwrite(value) if key in reducer_fields else value for key, value in values.items()}
     try:
-        updated_config = await accessor.aupdate(
-            read_config,
-            updates,
-            as_node=mutation_node,
-        )
-        snapshot = await accessor.aget(updated_config)
+        async with reserve_checkpoint_write(request, thread_id, user_id=get_effective_user_id()):
+            updated_config = await accessor.aupdate(
+                read_config,
+                updates,
+                as_node=mutation_node,
+            )
+            snapshot = await accessor.aget(updated_config)
+    except ConflictError:
+        raise HTTPException(status_code=409, detail="Thread has a run in flight. Update state after the run finishes.") from None
     except _CHECKPOINT_MODE_ERRORS as exc:
         raise _checkpoint_mode_http_error(exc, thread_id) from exc
     except Exception:

backend/app/gateway/run_models.py

@@ -28,7 +28,7 @@ class RunCreateRequest(BaseModel):
     interrupt_after: list[str] | Literal["*"] | None = Field(default=None, description="Nodes to interrupt after")
     stream_mode: list[RunStreamMode] | RunStreamMode | None = Field(default=None, description="Supported stream mode(s)")
     stream_subgraphs: bool = Field(default=False, description="Include subgraph events")
-    stream_resumable: None = Field(default=None, description="Compatibility placeholder; resumable SSE is not supported")
+    stream_resumable: Literal[False] | None = Field(default=None, description="Compatibility placeholder; only the SDK's non-resumable default (null/false) is accepted")
     on_disconnect: Literal["cancel", "continue"] = Field(default="cancel", description="Behaviour on SSE disconnect")
     on_completion: None = Field(default=None, description="Compatibility placeholder; completion behavior is not supported")
     multitask_strategy: Literal["reject", "rollback", "interrupt"] = Field(default="reject", description="Concurrency strategy")
@@ -38,7 +38,6 @@ class RunCreateRequest(BaseModel):
 
     @field_validator(
         "webhook",
-        "stream_resumable",
         "on_completion",
         "multitask_strategy",
         "after_seconds",
@@ -53,7 +52,6 @@ class RunCreateRequest(BaseModel):
 
         supported_defaults = {
             "webhook": None,
-            "stream_resumable": None,
             "on_completion": None,
             "multitask_strategy": {"reject", "rollback", "interrupt"},
             "after_seconds": None,
@@ -73,6 +71,20 @@ class RunCreateRequest(BaseModel):
             )
         return value
 
+    @field_validator("stream_resumable", mode="before")
+    @classmethod
+    def reject_resumable_streams(cls, value: Any) -> Any:
+        # LangGraph SDK clients always send this field (its default is ``False``, which the
+        # payload's ``None`` filter keeps). ``False`` asks for the non-resumable stream
+        # DeerFlow already serves, so only an explicit ``True`` requests the unsupported feature.
+        if value is None or value is False:
+            return value
+        raise PydanticCustomError(
+            "unsupported_run_option",
+            "Run option '{option}' is not supported by DeerFlow",
+            {"option": "stream_resumable"},
+        )
+
     @field_validator("stream_mode", mode="before")
     @classmethod
     def reject_unsupported_stream_modes(cls, value: Any) -> Any:

backend/app/gateway/services.py

@@ -11,7 +11,8 @@ import asyncio
 import json
 import logging
 import re
-from collections.abc import Mapping
+from collections.abc import AsyncIterator, Mapping
+from contextlib import asynccontextmanager
 from types import SimpleNamespace
 from typing import Any
 
@@ -40,10 +41,13 @@ from deerflow.runtime import (
     CheckpointStateAccessor,
     ConflictError,
     DisconnectMode,
+    RunContext,
     RunManager,
     RunRecord,
     RunStatus,
     StreamBridge,
+    StreamGap,
+    ThreadOperationKind,
     UnsupportedStrategyError,
     build_state_mutation_graph,
     run_agent,
@@ -64,6 +68,25 @@ from deerflow.utils.messages import ORIGINAL_USER_CONTENT_KEY
 
 logger = logging.getLogger(__name__)
 
+
+@asynccontextmanager
+async def reserve_checkpoint_write(
+    request: Request,
+    thread_id: str,
+    *,
+    user_id: str | None = None,
+) -> AsyncIterator[None]:
+    """Serialize an out-of-run checkpoint writer against all thread operations."""
+    run_manager = get_run_manager(request)
+    async with goal_thread_lock(thread_id):
+        async with run_manager.reserve_thread_operation(
+            thread_id,
+            kind=ThreadOperationKind.checkpoint_write,
+            user_id=user_id,
+        ):
+            yield
+
+
 _TERMINAL_RUN_STATUSES = {
     RunStatus.success,
     RunStatus.error,
@@ -71,6 +94,8 @@ _TERMINAL_RUN_STATUSES = {
     RunStatus.interrupted,
 }
 
+_THREAD_METADATA_SETUP_TIMEOUT_SECONDS = 5.0
+
 _SERVER_OWNED_MESSAGE_METADATA_KEYS = frozenset(
     {
         _DYNAMIC_CONTEXT_REMINDER_KEY,
@@ -105,6 +130,51 @@ def _run_is_terminal(record: RunRecord) -> bool:
     return record.status in _TERMINAL_RUN_STATUSES
 
 
+def _consume_task_result(task: asyncio.Task) -> None:
+    """Retrieve a detached task's exception without propagating cancellation."""
+    if not task.cancelled():
+        task.exception()
+
+
+def _log_thread_metadata_task_result(task: asyncio.Task, *, thread_id: str) -> None:
+    """Log detached metadata setup failures while ignoring cancellation."""
+    if task.cancelled():
+        return
+    try:
+        task.result()
+    except asyncio.CancelledError:
+        return
+    except Exception:
+        logger.warning(
+            "Failed to ensure thread_meta for %s after worker detached (non-fatal)",
+            sanitize_log_param(thread_id),
+            exc_info=True,
+        )
+
+
+async def _ensure_thread_metadata(
+    run_ctx: RunContext,
+    record: RunRecord,
+    *,
+    owner_user_id: str | None,
+) -> None:
+    """Ensure an admitted run's thread exists without delaying task attachment."""
+    thread_store = run_ctx.thread_store
+    existing = await thread_store.get(record.thread_id)
+    if existing is None and owner_user_id:
+        unscoped = await thread_store.get(record.thread_id, user_id=None)
+        if unscoped is not None:
+            if unscoped.get("user_id") != owner_user_id:
+                await thread_store.update_owner(record.thread_id, owner_user_id, user_id=None)
+            existing = await thread_store.get(record.thread_id)
+    if existing is None:
+        await thread_store.create(
+            record.thread_id,
+            assistant_id=record.assistant_id,
+            metadata=record.metadata,
+        )
+
+
 async def _terminal_record_stream_missing(bridge: StreamBridge, record: RunRecord) -> bool:
     """True when a terminal run has no retained stream on bridges that can tell."""
     if not _run_is_terminal(record):
@@ -958,49 +1028,6 @@ async def start_run(
 
     owner_context_token = set_current_user(SimpleNamespace(id=owner_user_id)) if owner_user_id else None
     try:
-        try:
-            async with goal_thread_lock(thread_id):
-                record = await run_mgr.create_or_reject(
-                    thread_id,
-                    body.assistant_id,
-                    on_disconnect=disconnect,
-                    metadata=body.metadata or {},
-                    # Persist a secret-redacted copy of the config: the run record is
-                    # written to runs.kwargs_json and echoed by the run API, so a
-                    # request-scoped secret (#3861) must not ride along. The live
-                    # config built below keeps the secrets for the actual run.
-                    kwargs={"input": body.input, "config": redact_config_secrets(body.config)},
-                    multitask_strategy=body.multitask_strategy,
-                    model_name=model_name,
-                    user_id=owner_user_id,
-                )
-        except ConflictError as exc:
-            raise HTTPException(status_code=409, detail=str(exc)) from exc
-        except UnsupportedStrategyError as exc:
-            raise HTTPException(status_code=501, detail=str(exc)) from exc
-
-        # Upsert thread metadata so the thread appears in /threads/search,
-        # even for threads that were never explicitly created via POST /threads
-        # (e.g. stateless runs).
-        try:
-            existing = await run_ctx.thread_store.get(thread_id)
-            if existing is None and owner_user_id:
-                unscoped_existing = await run_ctx.thread_store.get(thread_id, user_id=None)
-                if unscoped_existing is not None:
-                    if unscoped_existing.get("user_id") != owner_user_id:
-                        await run_ctx.thread_store.update_owner(thread_id, owner_user_id, user_id=None)
-                    existing = await run_ctx.thread_store.get(thread_id)
-            if existing is None:
-                await run_ctx.thread_store.create(
-                    thread_id,
-                    assistant_id=body.assistant_id,
-                    metadata=body.metadata,
-                )
-            else:
-                await run_ctx.thread_store.update_status(thread_id, "running")
-        except Exception:
-            logger.warning("Failed to upsert thread_meta for %s (non-fatal)", sanitize_log_param(thread_id))
-
         agent_factory = resolve_agent_factory(body.assistant_id)
         is_internal_caller = getattr(getattr(request, "state", None), "auth_source", None) == AUTH_SOURCE_INTERNAL
         command = getattr(body, "command", None)
@@ -1028,8 +1055,59 @@ async def start_run(
             request_context=getattr(body, "context", None),
         )
 
-        task = asyncio.create_task(
-            run_agent(
+        async def run_after_metadata(record: RunRecord) -> None:
+            metadata_task = asyncio.create_task(
+                _ensure_thread_metadata(
+                    run_ctx,
+                    record,
+                    owner_user_id=owner_user_id,
+                )
+            )
+            abort_task = asyncio.create_task(record.abort_event.wait())
+            metadata_failure_logged = False
+            try:
+                done, _ = await asyncio.wait(
+                    (metadata_task, abort_task),
+                    timeout=_THREAD_METADATA_SETUP_TIMEOUT_SECONDS,
+                    return_when=asyncio.FIRST_COMPLETED,
+                )
+                if metadata_task in done:
+                    try:
+                        metadata_task.result()
+                    except asyncio.CancelledError:
+                        pass
+                    except Exception:
+                        metadata_failure_logged = True
+                        logger.warning(
+                            "Failed to ensure thread_meta for %s (non-fatal)",
+                            sanitize_log_param(thread_id),
+                            exc_info=True,
+                        )
+                elif abort_task not in done:
+                    logger.warning(
+                        "Timed out ensuring thread_meta for %s after %.1fs",
+                        sanitize_log_param(thread_id),
+                        _THREAD_METADATA_SETUP_TIMEOUT_SECONDS,
+                    )
+            finally:
+                if metadata_task.done():
+                    if not metadata_failure_logged:
+                        _log_thread_metadata_task_result(metadata_task, thread_id=thread_id)
+                else:
+                    metadata_task.cancel()
+                    metadata_task.add_done_callback(
+                        lambda task: _log_thread_metadata_task_result(
+                            task,
+                            thread_id=thread_id,
+                        )
+                    )
+                if not abort_task.done():
+                    abort_task.cancel()
+                    abort_task.add_done_callback(_consume_task_result)
+            # Continue through run_agent even after metadata abort/timeout:
+            # its startup barrier is the single path that turns pending
+            # cancellation into no-agent-construction plus publish_end.
+            await run_agent(
                 bridge,
                 run_mgr,
                 record,
@@ -1042,8 +1120,43 @@ async def start_run(
                 interrupt_before=body.interrupt_before,
                 interrupt_after=body.interrupt_after,
             )
-        )
-        record.task = task
+
+        try:
+            async with goal_thread_lock(thread_id):
+                record = await run_mgr.create_or_reject(
+                    thread_id,
+                    body.assistant_id,
+                    on_disconnect=disconnect,
+                    metadata=body.metadata or {},
+                    # Persist a secret-redacted copy of the config: the run record is
+                    # written to runs.kwargs_json and echoed by the run API, so a
+                    # request-scoped secret (#3861) must not ride along. The live
+                    # config built above keeps the secrets for the actual run.
+                    kwargs={"input": body.input, "config": redact_config_secrets(body.config)},
+                    multitask_strategy=body.multitask_strategy,
+                    model_name=model_name,
+                    user_id=owner_user_id,
+                )
+
+                worker = run_after_metadata(record)
+                try:
+                    # No await is allowed between durable admission and task
+                    # attachment. Metadata setup runs inside the attached
+                    # worker so a pending cancellation can bypass stalled
+                    # thread-store IO and still reach run_agent's startup
+                    # barrier / stream finalization.
+                    record.task = asyncio.create_task(worker)
+                except Exception as exc:
+                    worker.close()
+                    await run_mgr.fail_start_if_pending(
+                        record.run_id,
+                        error=f"Failed to attach run worker: {exc}",
+                    )
+                    raise
+        except ConflictError as exc:
+            raise HTTPException(status_code=409, detail=str(exc)) from exc
+        except UnsupportedStrategyError as exc:
+            raise HTTPException(status_code=501, detail=str(exc)) from exc
 
         # Title sync is handled by worker.py's finally block which reads the
         # title from the checkpoint and calls thread_store.update_display_name
@@ -1124,11 +1237,27 @@ async def sse_consumer(
         yield format_sse("end", None)
         return
 
+    gap_emitted = False
     try:
         async for entry in bridge.subscribe(record.run_id, last_event_id=last_event_id):
             if await request.is_disconnected():
                 break
 
+            if isinstance(entry, StreamGap):
+                gap_emitted = True
+                yield format_sse(
+                    "gap",
+                    {
+                        "code": "stream_replay_gap",
+                        "run_id": record.run_id,
+                        "requested_event_id": entry.requested_event_id,
+                        "earliest_available_event_id": entry.earliest_available_event_id,
+                        "latest_available_event_id": entry.latest_available_event_id,
+                        "recovery": "reload_durable_state",
+                    },
+                )
+                return
+
             if entry is HEARTBEAT_SENTINEL:
                 if await _orphan_recovery_observed_after_heartbeat(record, run_mgr):
                     yield format_sse("end", None)
@@ -1147,7 +1276,7 @@ async def sse_consumer(
         # worker holds no in-memory task/abort state for them, so run_mgr.cancel()
         # cannot stop the task (it would 409). Skip on_disconnect cancellation for
         # those and only act on runs this worker actually owns.
-        if not record.store_only and record.status in (RunStatus.pending, RunStatus.running):
+        if not gap_emitted and not record.store_only and record.status in (RunStatus.pending, RunStatus.running):
             if record.on_disconnect == DisconnectMode.cancel:
                 await run_mgr.cancel(record.run_id)
 
@@ -1185,21 +1314,32 @@ async def wait_for_run_completion(
     if await _terminal_record_stream_missing(bridge, record):
         return True
 
+    resume_from_event_id: str | None = None
     try:
-        async for entry in bridge.subscribe(record.run_id):
-            # END_SENTINEL means the run reached a terminal state; honour it
-            # even if the client just disconnected so the caller still serializes
-            # the real final checkpoint.
-            if entry is END_SENTINEL:
-                completed = True
-                return True
-            if entry is HEARTBEAT_SENTINEL and await _orphan_recovery_observed_after_heartbeat(record, run_mgr):
-                completed = True
-                return True
-            if await request.is_disconnected():
-                break
-            # Heartbeats and regular events: keep waiting for END_SENTINEL.
-        return completed
+        while True:
+            gap_seen = False
+            async for entry in bridge.subscribe(record.run_id, last_event_id=resume_from_event_id):
+                # END_SENTINEL means the run reached a terminal state; honour it
+                # even if the client just disconnected so the caller still serializes
+                # the real final checkpoint.
+                if entry is END_SENTINEL:
+                    completed = True
+                    return True
+                if isinstance(entry, StreamGap):
+                    # The wait API only needs terminal completion, not a complete
+                    # event replay. Resume at the retained tail rather than
+                    # treating a bridge gap as a client disconnect.
+                    resume_from_event_id = entry.latest_available_event_id
+                    gap_seen = True
+                    break
+                if entry is HEARTBEAT_SENTINEL and await _orphan_recovery_observed_after_heartbeat(record, run_mgr):
+                    completed = True
+                    return True
+                if await request.is_disconnected():
+                    return False
+                # Heartbeats and regular events: keep waiting for END_SENTINEL.
+            if not gap_seen:
+                return completed
     finally:
         if not completed and record.status in (RunStatus.pending, RunStatus.running):
             if record.on_disconnect == DisconnectMode.cancel:

backend/docs/API.md

@@ -109,7 +109,8 @@ Content-Type: application/json
 **Run Option Compatibility:**
 - Supported concurrency strategies: `reject`, `rollback`, and `interrupt`
 - Compatibility default: `if_not_exists="create"`; this matches DeerFlow's current behavior
-- Unsupported options return `422`: `webhook`, `stream_resumable`, `after_seconds`, `feedback_keys`, any non-null `on_completion` value (including the SDK values `"complete"` and `"continue"`), `if_not_exists="reject"`, and `multitask_strategy="enqueue"`
+- Unsupported options return `422`: `webhook`, `stream_resumable=true`, `after_seconds`, `feedback_keys`, any non-null `on_completion` value (including the SDK values `"complete"` and `"continue"`), `if_not_exists="reject"`, and `multitask_strategy="enqueue"`
+- `stream_resumable=false` is accepted: it is the LangGraph SDK's default and requests the non-resumable stream DeerFlow already serves
 - Undeclared SDK options, including `checkpoint_during` and `durability`, also return `422` instead of being silently discarded
 
 **Recursion Limit:**
@@ -789,6 +790,31 @@ Both endpoints return `Content-Location: /api/threads/{thread_id}/runs/{run_id}`
 The DeerFlow web UI and LangGraph SDK clients rely on this header to discover the
 assigned `thread_id` and `run_id` on the first message of a new chat.
 
+### SSE replay retention and gaps
+
+Clients may reconnect to a run stream with `Last-Event-ID`. Replay history is
+bounded by `stream_bridge.queue_maxsize` (default `256`) and, for Redis, by the
+rolling `stream_ttl_seconds`. A retained cursor resumes after that event with no
+additional control frame.
+
+When a syntactically valid cursor is older than the retained watermark, the
+server sends exactly one `gap` event before any retained data and closes that
+subscription without an `end` event:
+
+```text
+event: gap
+data: {"code":"stream_replay_gap","run_id":"run-123","requested_event_id":"1718000000000-1","earliest_available_event_id":"1718000000100-42","latest_available_event_id":"1718000000200-84","recovery":"reload_durable_state"}
+
+```
+
+The frame deliberately has no SSE `id:`. Consumers must reload durable thread
+state and persisted run events/messages, then may reconnect from
+`latest_available_event_id` to follow newer live events. A gap does not cancel
+the active run. The same signal applies when a no-cursor subscriber has already
+established an empty-stream wait but the first Redis wake-up falls behind before
+delivery; in that case `requested_event_id` is `null`. Malformed cursor handling
+is backend-specific and is not the same as a valid cursor that was evicted.
+
 ---
 
 ## SDK Usage

backend/docs/CONFIGURATION.md

@@ -248,7 +248,7 @@ Notes:
 - `enabled: false` keeps background polling off by default.
 - `max_concurrent_runs` is a global cap on active scheduled runs (queued/running run rows); each poll cycle claims only into the remaining budget, so long runs accumulating across cycles cannot exceed it.
 - All scheduler fields are restart-required; edits need a Gateway restart.
-- Multi-worker deployments (`GATEWAY_WORKERS > 1`) must use the Postgres database backend. SQLite silently ignores row-level locks, so multiple workers can double-fire the same task. The process-local agentic browser tool group is incompatible with multiple Gateway workers; keep `GATEWAY_WORKERS=1` while `browser_navigate` is enabled. Browser control also requires the backend `browser` extra (`cd backend && uv sync --extra browser && uv run playwright install chromium`); startup detects enabled browser config and fails fast when Playwright is missing, and `/api/features` reports `browser_control.enabled=false` until the runtime is available.
+- Multi-worker deployments (`GATEWAY_WORKERS > 1`) must use the Postgres database backend, enable run ownership heartbeats, and set `run_events.backend: db`. SQLite silently ignores row-level locks, while memory and JSONL run-event stores are process-local and cannot enforce singleton delivery receipts across workers; startup rejects these combinations. The process-local agentic browser tool group is incompatible with multiple Gateway workers; keep `GATEWAY_WORKERS=1` while `browser_navigate` is enabled. Browser control also requires the backend `browser` extra (`cd backend && uv sync --extra browser && uv run playwright install chromium`); startup detects enabled browser config and fails fast when Playwright is missing, and `/api/features` reports `browser_control.enabled=false` until the runtime is available.
 - The MVP supports thread reuse and fresh-thread-per-run execution modes.
 - The MVP supports only `once` and `cron`.
 - Manual trigger uses the same scheduled-task resource and run lifecycle.
@@ -505,6 +505,12 @@ When you configure `sandbox.mounts`, DeerFlow exposes those `container_path` val
 
 For bare-metal Docker sandbox runs that use localhost, DeerFlow binds the sandbox HTTP port to `127.0.0.1` by default so it is not exposed on every host interface. Docker-outside-of-Docker deployments that connect through `host.docker.internal` keep the broad legacy bind for compatibility. Set `DEER_FLOW_SANDBOX_BIND_HOST` explicitly if your deployment needs a different bind address.
 
+Sandbox control-plane HTTP calls to loopback/private IPs, single-label cluster
+hosts, and Docker/Podman internal hostnames bypass `HTTP_PROXY`/`HTTPS_PROXY`
+inside the client. This prevents an inherited proxy from returning a misleading
+502 for a healthy local sandbox. Externally hosted sandbox FQDNs and public IPs
+continue to use the normal environment proxy configuration.
+
 ### Building a Custom AIO Sandbox Image
 
 `AioSandboxProvider` talks to the sandbox container through the `agent-sandbox` SDK. The Dockerfile for the default `enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest` image is not part of this repository; DeerFlow treats that image as an upstream AIO sandbox runtime.

backend/docs/STREAMING.md

@@ -143,12 +143,28 @@ sequenceDiagram
 关键组件：
 
 - `runtime/runs/worker.py::run_agent` — 在 `asyncio.Task` 里跑 `agent.astream()`，把每个 chunk 通过 `serialize(chunk, mode=mode)` 转成 JSON，再 `bridge.publish()`。
-- `runtime/stream_bridge` — 抽象 Queue。`publish/subscribe` 解耦生产者和消费者，支持 `Last-Event-ID` 重连、心跳、多订阅者 fan-out。Redis backend 会在每次 `publish()` / `publish_end()` 刷新 retained stream key TTL；启动恢复与基于 worker lease 的周期恢复共用 Gateway stream terminalization 路径：`RunManager` 先将 orphan run 持久化为 `error` 并写入显式的 `stop_reason=orphan_recovered`，随后 Gateway 发布 `END_SENTINEL` 并安排 stream cleanup。周期扫描、逐行状态写入和 Gateway callback 作为一个受监督的 single-flight 后台 task 执行；慢任务不会堆积，也不会阻塞唯一的 lease heartbeat。shutdown 优先收敛活跃 run，再处理恢复 task；尚未执行的延迟 stream cleanup 会改为立即删除。只有 runtime `yield` 前、无并发请求的启动恢复会把最新受影响 thread 标记为 error；周期恢复不做非原子的 thread 投影。store-only SSE 与 `/wait` consumer 不能把普通 durable terminal status 当成流已完成，否则可能跳过延迟发布的 error 等尾部事件；只有 `orphan_recovered` 信号能在 heartbeat 时触发 END fallback，因为此时 producer 已被确认失联。TTL 仍是 Redis 内存和故障安全网，不是正常的 subscriber 终止机制。
+- `runtime/stream_bridge` — 抽象 Queue。`publish/subscribe` 解耦生产者和消费者，支持 `Last-Event-ID` 重连、心跳、多订阅者 fan-out。Memory 和 Redis 都只保留 `queue_maxsize` 条数据事件；游标早于保留水位线时返回 `StreamGap`，不会从当前最早事件静默部分重放。Redis backend 会在每次 `publish()` / `publish_end()` 刷新 retained stream key TTL；启动恢复与基于 worker lease 的周期恢复共用 Gateway stream terminalization 路径：`RunManager` 先将 orphan run 持久化为 `error` 并写入显式的 `stop_reason=orphan_recovered`，随后 Gateway 发布 `END_SENTINEL` 并安排 stream cleanup。周期扫描、逐行状态写入和 Gateway callback 作为一个受监督的 single-flight 后台 task 执行；慢任务不会堆积，也不会阻塞唯一的 lease heartbeat。shutdown 优先收敛活跃 run，再处理恢复 task；尚未执行的延迟 stream cleanup 会改为立即删除。只有 runtime `yield` 前、无并发请求的启动恢复会把最新受影响 thread 标记为 error；周期恢复不做非原子的 thread 投影。store-only SSE 与 `/wait` consumer 不能把普通 durable terminal status 当成流已完成，否则可能跳过延迟发布的 error 等尾部事件；只有 `orphan_recovered` 信号能在 heartbeat 时触发 END fallback，因为此时 producer 已被确认失联。TTL 仍是 Redis 内存和故障安全网，不是正常的 subscriber 终止机制。
 - `app/gateway/services.py::sse_consumer` — 从 bridge 订阅，格式化为 SSE wire 帧。
 - `runtime/serialization.py::serialize` — mode-aware 序列化；`messages` mode 下 `serialize_messages_tuple` 把 `(chunk, metadata)` 转成 `[chunk.model_dump(), metadata]`。
 
 **`StreamBridge` 的存在价值**：当生产者（`run_agent` 任务）和消费者（HTTP 连接）在不同的 asyncio task 里运行时，需要一个可以跨 task 传递事件的中介。Queue 同时还承担断连重连的 buffer 和多订阅者的 fan-out。
 
+### 有界历史与 `gap` 恢复
+
+`Last-Event-ID` 只保证在保留窗口内完整重放，不代表无限历史。有效游标仍在窗口内时，bridge 从该 ID 的下一条事件正常恢复；有效游标已经被 `queue_maxsize` 裁剪，或在线消费者慢到落后水位线时，Gateway 会在任何部分重放之前发送：
+
+```text
+event: gap
+data: {"code":"stream_replay_gap","run_id":"...","requested_event_id":"...","earliest_available_event_id":"...","latest_available_event_id":"...","recovery":"reload_durable_state"}
+
+```
+
+`gap` 帧没有 SSE `id:`，后面也没有正常 `end`；当前订阅随即关闭。它是恢复边界而不是客户端断开，因此不会触发 `on_disconnect=cancel`。客户端必须丢弃不再可信的瞬时状态，重新读取 thread checkpoint 和持久化 run-event/message history，再以 `latest_available_event_id` 为游标跟随新事件。DeerFlow Web UI 自动执行此流程并最多连续恢复五次。
+
+Redis 对无游标、空 stream 上已经建立的阻塞等待也遵循相同契约：第一次 `XREAD` 唤醒的数据在交付前仍是 provisional baseline，bridge 会用下一次事务快照确认其尾 ID 仍在保留窗口。若生产者已经裁剪了该基线，订阅直接返回 `requested_event_id: null` 的 `gap`，不会先交付 retained tail。这个检查有明确的性能代价：每轮订阅需要一个包含 `XRANGE`、`XREVRANGE`、非阻塞 `XREAD` 的事务快照；空闲时还需要单独的阻塞 `XREAD` 来唤醒。
+
+“有效但已淘汰”和 malformed cursor 是不同策略：本契约只要求前者产生 `gap`。Redis 的 malformed ID 仍从 live tail 等待；Memory 对 malformed ID 及序号不低于水位线的未知 ID 仍采用既有的最早保留事件策略。对于序号已经低于水位线的数字格式 foreign ID，Memory 无法再校验已淘汰的 timestamp，因此保守返回 `gap`，优先保证客户端不会把不完整重放误认为完整。
+
 ---
 
 ## DeerFlowClient 路径：sync + in-process

backend/docs/summarization.md

@@ -21,7 +21,7 @@ Summarization is configured in `config.yaml` under the `summarization` key:
 ```yaml
 summarization:
   enabled: true
-  model_name: null  # Use default model or specify a lightweight model
+  model_name: null  # null = summarize with the run's own model (see below); or name a lightweight model
 
   # Trigger conditions (OR logic - any condition triggers summarization)
   trigger:
@@ -61,8 +61,11 @@ summarization:
 
 #### `model_name`
 - **Type**: String or null
-- **Default**: `null` (uses default model)
-- **Description**: Model to use for generating summaries. Recommended to use a lightweight, cost-effective model like `gpt-4o-mini` or equivalent.
+- **Default**: `null`
+- **Description**: Model to use for generating summaries.
+  - **`null` (model ownership)**: summarize with the model the run actually executes with — the lead run's resolved model, a subagent's own model, or a thread's custom-agent model — **not** `config.models[0]`. This keeps compaction working on a run whose model is healthy even when `models[0]`'s provider is broken (expired key, quota, outage).
+  - **Set to a model name**: that model generates summaries. If its provider fails, compaction **falls back to the run's own model** so a broken summary provider cannot disable compaction while a working model is available. Recommended to use a lightweight, cost-effective model like `gpt-4o-mini` or equivalent.
+  - Ownership applies to all three paths — automatic lead compaction, subagent compaction, and manual `/compact`. Manual `/compact` resolves the run model with the same precedence as a normal run: the model selected for the request (`POST /api/threads/{id}/compact` body `model_name`, sent by the frontend from the composer's current model) → the thread's custom-agent model → the default. A whitespace-only summary response is treated as a generation failure (it is never committed as a valid empty summary).
 
 #### `trigger`
 - **Type**: Single `ContextSize` or list of `ContextSize` objects
@@ -223,9 +226,9 @@ The middleware intelligently preserves message context:
   - Summaries don't require the most powerful models
   - Significant cost savings on high-volume applications
 
-- **Default**: If `model_name` is `null`, uses the default model
-  - May be more expensive but ensures consistency
-  - Good for simple setups
+- **Default**: If `model_name` is `null`, summarizes with the run's own model (not `models[0]`)
+  - Keeps compaction working when `models[0]`'s provider is broken but the run's model is healthy
+  - Good for simple setups; no separate summary provider to keep credentialed
 
 ### Optimization Tips
 

backend/packages/harness/deerflow/agents/lead_agent/agent.py

@@ -36,6 +36,7 @@ from deerflow.agents.middlewares.clarification_middleware import ClarificationMi
 from deerflow.agents.middlewares.configured_extensions import load_configured_extension_middlewares
 from deerflow.agents.middlewares.loop_detection_middleware import LoopDetectionMiddleware
 from deerflow.agents.middlewares.memory_middleware import MemoryMiddleware
+from deerflow.agents.middlewares.model_length_finish_reason_middleware import ModelLengthFinishReasonMiddleware
 from deerflow.agents.middlewares.safety_finish_reason_middleware import SafetyFinishReasonMiddleware
 from deerflow.agents.middlewares.subagent_limit_middleware import SubagentLimitMiddleware
 from deerflow.agents.middlewares.summarization_middleware import DeerFlowSummarizationMiddleware, create_summarization_middleware
@@ -133,9 +134,14 @@ def _resolve_model_name(requested_model_name: str | None = None, *, app_config:
     return default_model_name
 
 
-def _create_summarization_middleware(*, app_config: AppConfig | None = None) -> DeerFlowSummarizationMiddleware | None:
-    """Create and configure the summarization middleware from config."""
-    return create_summarization_middleware(app_config=app_config)
+def _create_summarization_middleware(*, app_config: AppConfig | None = None, run_model_name: str | None = None) -> DeerFlowSummarizationMiddleware | None:
+    """Create and configure the summarization middleware from config.
+
+    ``run_model_name`` is the resolved run model; it is the source of truth for
+    ``model_name: null`` summarization and the explicit-summary-model fallback, so a
+    custom agent's model is used instead of ``config.models[0]``.
+    """
+    return create_summarization_middleware(app_config=app_config, run_model_name=run_model_name)
 
 
 def _create_todo_list_middleware(is_plan_mode: bool) -> TodoMiddleware | None:
@@ -359,7 +365,7 @@ def build_middlewares(
     )
 
     # Add summarization middleware if enabled
-    summarization_middleware = _create_summarization_middleware(app_config=resolved_app_config)
+    summarization_middleware = _create_summarization_middleware(app_config=resolved_app_config, run_model_name=model_name)
     if summarization_middleware is not None:
         middlewares.append(summarization_middleware)
 
@@ -447,6 +453,12 @@ def build_middlewares(
     # allowing LangChain's no-tool-call router to end a silent successful run.
     middlewares.append(TerminalResponseMiddleware())
 
+    # A provider may also cap the final assistant response at the model output
+    # limit. Preserve the assistant content unchanged, but stamp a run-level
+    # stop_reason so Gateway consumers can tell a length-capped completion from
+    # a clean one.
+    middlewares.append(ModelLengthFinishReasonMiddleware())
+
     # SafetyFinishReasonMiddleware — suppress tool execution when the provider
     # safety-terminated the response. Registered after the terminal-response
     # and custom/configured middlewares so LangChain's reverse-order after_model

backend/packages/harness/deerflow/agents/memory/backends/deermem/deer_mem.py

@@ -33,14 +33,15 @@ from deerflow.agents.memory.manager import MemoryConflictError, MemoryCorruption
 from .deermem.config import DeerMemConfig
 from .deermem.core.llm import build_llm
 from .deermem.core.message_processing import (
-    detect_correction,
-    detect_reinforcement,
+    SIGNAL_NAMES,
+    detect_signals,
     filter_messages_for_memory,
+    filter_trivial,
     load_patterns,
 )
 from .deermem.core.paths import DEFAULT_AGENT_BUCKET
 from .deermem.core.prompt import format_memory_for_injection, load_prompt, load_prompt_messages, warm_tiktoken_cache
-from .deermem.core.queue import MemoryUpdateQueue
+from .deermem.core.queue import MemoryUpdateQueue, QueueFull
 from .deermem.core.storage import MemoryRevisionConflict, MemoryStorageCorruption, create_storage
 from .deermem.core.updater import MemoryUpdater, _coerce_source_confidence
 
@@ -100,8 +101,7 @@ class DeerMem(MemoryManager):
     _llm: Any = PrivateAttr(default=None)
     _updater: Any = PrivateAttr(default=None)
     _queue: Any = PrivateAttr(default=None)
-    _correction_patterns: Any = PrivateAttr(default=None)
-    _reinforcement_patterns: Any = PrivateAttr(default=None)
+    _trivial_patterns: Any = PrivateAttr(default=None)
 
     # DeerMem implements search() (case-insensitive substring over stored facts),
     # so it is valid for mode="tool" (the base invariant validator requires this
@@ -121,8 +121,12 @@ class DeerMem(MemoryManager):
         # Signal-detection patterns (externalized YAML; ``patterns_dir`` override
         # or bundled defaults = pre-externalization behavior). Loaded once at
         # construction and reused by ``_prepare_update``'s detect_* calls.
-        self._correction_patterns = load_patterns("correction", patterns_dir=self._config.patterns_dir)
-        self._reinforcement_patterns = load_patterns("reinforcement", patterns_dir=self._config.patterns_dir)
+        # Pre-load trivial + signal patterns at construction so a misconfigured
+        # patterns_dir (missing / invalid yaml) surfaces at startup, not on the
+        # first update. Compiled patterns are cached by load_patterns.
+        self._trivial_patterns = load_patterns("trivial", patterns_dir=self._config.patterns_dir)
+        for _signal_name in SIGNAL_NAMES:
+            load_patterns(_signal_name, patterns_dir=self._config.patterns_dir)
         # host_llm (host-injected default model) takes precedence over build_llm(model)
         # so zero-config DeerMem (empty `model`) still extracts via the app default,
         # mirroring pre-abstraction `model_name: null`. Standalone (no factory) -> None.
@@ -168,7 +172,7 @@ class DeerMem(MemoryManager):
         ``model_post_init`` (shared with direct construction).
         """
         config_dict = dict(backend_config or {})
-        for key in ("should_keep_hidden_message", "trace_context_manager"):
+        for key in ("should_keep_hidden_message", "trace_context_manager", "extraction_callback"):
             if key not in config_dict and key in host_hooks:
                 config_dict[key] = host_hooks[key]
         if "host_llm" not in config_dict:
@@ -207,16 +211,25 @@ class DeerMem(MemoryManager):
         prepared = self._prepare_update(messages)
         if prepared is None:
             return
-        filtered, correction_detected, reinforcement_detected = prepared
-        self._queue.add(
-            thread_id=thread_id,
-            messages=filtered,
-            agent_name=_resolve_agent_name(agent_name),
-            user_id=user_id,
-            trace_id=trace_id,
-            correction_detected=correction_detected,
-            reinforcement_detected=reinforcement_detected,
-        )
+        filtered, signals = prepared
+        # DeerMem owns the queue, so it owns the backpressure degradation: a
+        # QueueFull here is logged + dropped so memory backpressure degrades to
+        # "update skipped" rather than propagating into
+        # MemoryMiddleware.after_agent and breaking the agent run (peer
+        # middlewares self-guard the same way). The dropped update is re-fed
+        # next turn (the middleware passes the full conversation each cycle, and
+        # the watermark does not advance on a non-enqueued turn).
+        try:
+            self._queue.add(
+                thread_id=thread_id,
+                messages=filtered,
+                agent_name=_resolve_agent_name(agent_name),
+                user_id=user_id,
+                trace_id=trace_id,
+                signals=signals,
+            )
+        except QueueFull as e:
+            logger.warning("Memory update rejected under backpressure (thread=%s): %s", thread_id, e)
 
     def add_nowait(
         self,
@@ -234,37 +247,44 @@ class DeerMem(MemoryManager):
         prepared = self._prepare_update(messages)
         if prepared is None:
             return
-        filtered, correction_detected, reinforcement_detected = prepared
-        self._queue.add_nowait(
-            thread_id=thread_id,
-            messages=filtered,
-            agent_name=_resolve_agent_name(agent_name),
-            user_id=user_id,
-            correction_detected=correction_detected,
-            reinforcement_detected=reinforcement_detected,
-        )
+        filtered, signals = prepared
+        # Defense-in-depth: the emergency path always admits under backpressure
+        # (see _enqueue_locked), so QueueFull is not expected here -- but the
+        # emergency flush is invoked from summarization_hook, so a propagated
+        # exception would break summarization. Catch + log to be safe.
+        try:
+            self._queue.add_nowait(
+                thread_id=thread_id,
+                messages=filtered,
+                agent_name=_resolve_agent_name(agent_name),
+                user_id=user_id,
+                signals=signals,
+            )
+        except QueueFull as e:
+            logger.warning("Memory emergency flush rejected under backpressure (thread=%s): %s", thread_id, e)
 
     def _prepare_update(
         self,
         messages: list[Any],
-    ) -> tuple[list[Any], bool, bool] | None:
+    ) -> tuple[list[Any], frozenset[str]] | None:
         """Filter to user+final-AI messages, require both, detect signals.
 
-        Returns ``(filtered, correction_detected, reinforcement_detected)``
-        or ``None`` when there is no meaningful conversation (missing a user
-        or an assistant turn).
+        Returns ``(filtered, signals)`` where ``signals`` is the set of signal
+        classes detected in the recent turns, or ``None`` when there is no
+        meaningful conversation (missing a user or an assistant turn, or every
+        turn dropped as a trivial pure-acknowledgment).
         """
         filtered = filter_messages_for_memory(
             messages,
             should_keep_hidden_message=self._config.should_keep_hidden_message,
         )
+        filtered = filter_trivial(filtered, patterns=self._trivial_patterns)
         user_messages = [m for m in filtered if getattr(m, "type", None) == "human"]
         assistant_messages = [m for m in filtered if getattr(m, "type", None) == "ai"]
         if not user_messages or not assistant_messages:
             return None
-        correction_detected = detect_correction(filtered, patterns=self._correction_patterns)
-        reinforcement_detected = not correction_detected and detect_reinforcement(filtered, patterns=self._reinforcement_patterns)
-        return filtered, correction_detected, reinforcement_detected
+        signals = detect_signals(filtered, patterns_dir=self._config.patterns_dir)
+        return filtered, frozenset(signals)
 
     # ── Read ─────────────────────────────────────────────────────────────
     def get_context(

backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/config.py

@@ -79,6 +79,11 @@ class DeerMemConfig(BaseModel):
         le=300,
         description="Seconds to wait before processing queued updates (debounce).",
     )
+    queue_max_depth: int = Field(
+        default=1000,
+        ge=0,
+        description=("Backpressure cap on pending items. 0 = unlimited. When the cap is reached, new non-signal updates are rejected (QueueFull); signal updates are always admitted so important memories are never shed."),
+    )
     # ── Facts ────────────────────────────────────────────────────────────
     max_facts: int = Field(default=100, ge=10, le=500, description="Maximum number of facts to store.")
     fact_confidence_threshold: float = Field(
@@ -199,6 +204,29 @@ class DeerMemConfig(BaseModel):
         le=20,
         description=("Maximum number of source facts per consolidation group. Prevents the LLM from merging too many facts into one and losing important details."),
     )
+    # ── Extraction quality callback (post-invoke observability) ─────────
+    extraction_callback: Any = Field(
+        default=None,
+        description=(
+            "Optional ``callback(metrics)`` invoked AFTER the extraction LLM "
+            "call (token usage, facts passing/rejected by the confidence "
+            "filter, rejection rate, prompt version). The host injects a "
+            "Langfuse-based callback to emit an extraction span; None = no "
+            "post-invoke observability. Set programmatically (not from YAML)."
+        ),
+    )
+    # ── Watermark cache (in-memory, bounded LRU) ─────────────────────────
+    watermark_max_keys: int = Field(
+        default=4096,
+        ge=0,
+        description=(
+            "Soft cap on the in-memory conversation-watermark cache (one entry "
+            "per distinct thread/user/agent). The cache is a bounded LRU: when "
+            "over capacity the least-recently-used entry is dropped, and a "
+            "dropped key re-extracts one batch on that thread's next turn (the "
+            "same as a restart). 0 = unbounded."
+        ),
+    )
     # ── Message processing (externalized patterns / prompts) ──
     patterns_dir: str | None = Field(
         default=None,

backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/message_patterns/decision.yaml

@@ -0,0 +1,27 @@
+# Decision signal patterns for detect_signals (message_processing).
+#
+# Detects that the user made a decision / chose an option -- a high-value "what
+# to remember" signal. Matched via ``search`` over the last 6 human turns. Keep
+# these narrow to avoid false positives.
+#
+# Each list entry is either:
+#   - a string                       -> compiled with no flags
+#   - a mapping {pattern, flags}     -> flags is a list of names: "ignorecase"
+#
+# In single-quoted strings backslashes are literal (regex \b stays \b); double
+# '' for a literal single quote (apostrophe).
+
+- pattern: '\blet.s (?:go with|use|pick|choose)\b'
+  flags: [ignorecase]
+- pattern: '\bI.ll (?:go with|use|pick|choose)\b'
+  flags: [ignorecase]
+- pattern: '\bwe (?:should|will) (?:go with|use|pick|choose)\b'
+  flags: [ignorecase]
+- pattern: '\bI (?:decide|chose|selected) to\b'
+  flags: [ignorecase]
+- '就用'
+- '决定用'
+- '决定采用'
+- '我们选'
+- '我选'
+- '就采用'

backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/message_patterns/goal.yaml

@@ -0,0 +1,27 @@
+# Goal signal patterns for detect_signals (message_processing).
+#
+# Detects that the user stated an objective / intent / plan -- a high-value
+# "what to remember" signal. Matched via ``search`` over the last 6 human
+# turns. Keep these narrow to avoid false positives.
+#
+# Each list entry is either:
+#   - a string                       -> compiled with no flags
+#   - a mapping {pattern, flags}     -> flags is a list of names: "ignorecase"
+#
+# In single-quoted strings backslashes are literal (regex \b stays \b); double
+# '' for a literal single quote (apostrophe).
+
+- pattern: '\bI (?:plan to|want to|am going to|aim to|intend to)\b'
+  flags: [ignorecase]
+- pattern: '\bmy goal is\b'
+  flags: [ignorecase]
+- pattern: '\bI.m going to\b'
+  flags: [ignorecase]
+- pattern: '\bnext I.ll\b'
+  flags: [ignorecase]
+- '我的目标是'
+- '我打算'
+- '我准备'
+- '接下来我要'
+- '接下来我打算'
+- '我想做'

backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/message_patterns/identity.yaml

@@ -0,0 +1,26 @@
+# Identity signal patterns for detect_signals (message_processing).
+#
+# Detects that the user stated something about who they are (role, profession,
+# background) -- a high-value "what to remember" signal. Matched via ``search``
+# over the last 6 human turns. Keep these narrow to avoid false positives.
+#
+# Each list entry is either:
+#   - a string                       -> compiled with no flags
+#   - a mapping {pattern, flags}     -> flags is a list of names: "ignorecase"
+#
+# In single-quoted strings backslashes are literal (regex \b stays \b); double
+# '' for a literal single quote (apostrophe).
+
+- pattern: '\bI am (?:a |an )?[a-z][a-z -]{2,}\b'
+  flags: [ignorecase]
+- pattern: '\bI work as\b'
+  flags: [ignorecase]
+- pattern: '\bI.m (?:a |an )?[a-z][a-z -]{2,}\b'
+  flags: [ignorecase]
+- pattern: '\bmy job is\b'
+  flags: [ignorecase]
+- '我是'
+- '我的职业是'
+- '我的工作是'
+- '我担任'
+- '我是一名'

backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/message_patterns/preference.yaml

@@ -0,0 +1,29 @@
+# Preference signal patterns for detect_signals (message_processing).
+#
+# Detects that the user stated a preference / dislike -- a high-value "what to
+# remember" signal. Matched via ``search`` over the last 6 human turns. Keep
+# these narrow to avoid false positives; extend here without touching code.
+#
+# Each list entry is either:
+#   - a string                       -> compiled with no flags
+#   - a mapping {pattern, flags}     -> flags is a list of names: "ignorecase"
+#
+# In single-quoted strings backslashes are literal (regex \b stays \b); double
+# '' for a literal single quote (apostrophe).
+
+- pattern: '\bI (?:really )?(?:prefer|like|love|enjoy|favor)\b'
+  flags: [ignorecase]
+- pattern: '\bI (?:really )?(?:hate|dislike|don.t like|cannot stand)\b'
+  flags: [ignorecase]
+- pattern: '\bmy favorite\b'
+  flags: [ignorecase]
+- pattern: '\bI.d rather\b'
+  flags: [ignorecase]
+- '我喜欢'
+- '我偏好'
+- '我更喜欢'
+- '我偏爱'
+- '我讨厌'
+- '我不喜欢'
+- '我钟爱'
+- '我最喜欢'

backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/message_patterns/trivial.yaml

@@ -0,0 +1,36 @@
+# Trivial pure-acknowledgment patterns for filter_trivial (message_processing).
+#
+# Each entry is a regex matched against the WHOLE (stripped) human message via
+# ``fullmatch`` -- a message is trivial only if it is nothing but an ack, so a
+# substantive turn that happens to contain "ok" is never dropped. Trailing
+# punctuation/whitespace is stripped before matching, so "ok." / "好的！" count.
+#
+# Each list entry is either:
+#   - a string                       -> compiled with no flags
+#   - a mapping {pattern, flags}     -> flags is a list of names: "ignorecase"
+#
+# Keep these narrow: a false positive silently drops a real user turn from
+# memory. Extend here without touching code.
+
+- pattern: '嗯+'
+- pattern: 'ok(?:ay)?'
+  flags: [ignorecase]
+- pattern: '好的[呢]?'
+- pattern: '好[的呀]?'
+- pattern: '谢谢'
+- pattern: '感谢'
+- pattern: '多谢'
+- pattern: '对'
+- pattern: '是的'
+- pattern: '收到'
+- pattern: '明白'
+- pattern: '了解'
+- pattern: '知道了'
+- pattern: 'got it'
+  flags: [ignorecase]
+- pattern: 'thanks'
+  flags: [ignorecase]
+- pattern: 'thank you'
+  flags: [ignorecase]
+- pattern: 'cheers'
+  flags: [ignorecase]

backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/message_processing.py

@@ -246,3 +246,95 @@ def detect_reinforcement(messages: list[Any], *, patterns: list[re.Pattern[str]]
             return True
 
     return False
+
+
+# Signal classes detected by :func:`detect_signals`. Names align with the fact
+# ``category`` enum (CORE_CATEGORIES) so a signal can drive an extraction
+# category hint directly, except ``reinforcement`` (no same-named category; it
+# maps to preference/behavior in the extraction hint). Keep new signal names in
+# sync with CORE_CATEGORIES before adding them here.
+SIGNAL_NAMES: tuple[str, ...] = (
+    "correction",
+    "reinforcement",
+    "preference",
+    "identity",
+    "goal",
+    "decision",
+)
+
+
+def detect_signals(
+    messages: list[Any],
+    *,
+    patterns_dir: str | None = None,
+) -> set[str]:
+    """Detect signal classes in the recent conversation turns.
+
+    Returns the set of signal names whose patterns match any of the last 6
+    human turns. This generalizes :func:`detect_correction` /
+    :func:`detect_reinforcement` (which remain for backward compatibility) to
+    the full signal set. The window stays ``messages[-6:]``.
+    """
+    recent_user_msgs = [msg for msg in messages[-6:] if getattr(msg, "type", None) == "human"]
+    if not recent_user_msgs:
+        return set()
+
+    hits: set[str] = set()
+    for name in SIGNAL_NAMES:
+        patterns = load_patterns(name, patterns_dir=patterns_dir)
+        if not patterns:
+            continue
+        for msg in recent_user_msgs:
+            content = extract_message_text(msg).strip()
+            if content and any(pattern.search(content) for pattern in patterns):
+                hits.add(name)
+                break
+    return hits
+
+
+# Trailing characters stripped before a whole-message trivial match: a pure
+# acknowledgment with trailing punctuation ("ok.", "好的！") is still trivial.
+_TRIVIAL_TRAIL = " \t\n\r.。,，!！?？;；"
+
+
+def filter_trivial(
+    messages: list[Any],
+    *,
+    patterns: list[re.Pattern[str]] | None = None,
+    patterns_dir: str | None = None,
+) -> list[Any]:
+    """Drop pure-acknowledgment human turns and their AI replies.
+
+    A human turn is "trivial" when its whole (stripped) text matches a trivial
+    pattern (e.g. "嗯", "ok", "好的", "谢谢") -- matched via ``fullmatch`` so a
+    substantive turn containing "ok" is never dropped. The matched human turn
+    and its following assistant reply are both removed (reusing the
+    ``skip_next_ai`` discipline from :func:`filter_messages_for_memory`). When
+    every turn is trivial, the result is empty, which the caller treats as "do
+    not enqueue" (saving an extraction LLM call).
+    """
+    if patterns is None:
+        patterns = load_patterns("trivial", patterns_dir=patterns_dir)
+    if not patterns:
+        return list(messages)
+
+    result: list[Any] = []
+    skip_next_ai = False
+    for msg in messages:
+        msg_type = getattr(msg, "type", None)
+        if msg_type == "human":
+            content = extract_message_text(msg).strip().rstrip(_TRIVIAL_TRAIL)
+            is_trivial = bool(content) and any(pattern.fullmatch(content) for pattern in patterns)
+            if is_trivial:
+                skip_next_ai = True
+                continue
+            result.append(msg)
+            skip_next_ai = False
+        elif msg_type == "ai":
+            tool_calls = getattr(msg, "tool_calls", None)
+            if not tool_calls:
+                if skip_next_ai:
+                    skip_next_ai = False
+                    continue
+                result.append(msg)
+    return result

backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/prompt.py

@@ -762,9 +762,16 @@ def format_conversation_for_update(messages: list[Any]) -> str:
             if not content:
                 continue
 
-        # Truncate very long messages
+        # Truncate very long messages: keep the head (topic / opening) and the
+        # tail (conclusion / "remember X" instruction), dropping the middle.
+        # A head-only chop loses the tail's directives; a head+tail split
+        # preserves both. The separator is plain ASCII (no < > &) so the
+        # html.escape below leaves it intact and tells the LLM where text was
+        # cut. Escape happens after truncation, so the boundary never splits an
+        # entity (entities only exist after escaping).
         if len(str(content)) > 1000:
-            content = str(content)[:1000] + "..."
+            s = str(content)
+            content = s[:500] + "\n...[truncated]...\n" + s[-500:]
 
         # Escape < > & before embedding into the <conversation> block of
         # the memory_update prompt. This raw user turn is the most attacker-influenced

backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/queue.py

@@ -1,4 +1,18 @@
-"""Memory update queue with debounce mechanism."""
+"""Memory update queue with debounce mechanism.
+
+The queue collects conversation contexts and processes them after a
+configurable debounce period; multiple contexts for the same
+``(thread_id, user_id, agent_name)`` key are coalesced into one update.
+
+The queue is a process-local in-memory list plus a debounce
+:class:`~threading.Timer`. Items still pending at process exit are lost
+(best-effort :meth:`MemoryUpdateQueue.flush_sync` drain softens this for
+graceful shutdown). Memory updates are best-effort: a failed or lost update is
+re-fed on the next conversation turn (the middleware passes the full
+conversation each cycle, and the updater's watermark does not advance on
+failure), so an in-memory queue covers the realistic graceful-deploy case
+without a persistence layer.
+"""
 
 from __future__ import annotations
 
@@ -17,6 +31,25 @@ if TYPE_CHECKING:
 logger = logging.getLogger(__name__)
 
 
+class QueueFull(Exception):
+    """Raised when a non-signal update is rejected under backpressure.
+
+    Signal-bearing updates (any detected signal) are always admitted so that
+    important memories are never shed; only non-signal updates are rejected
+    once ``queue_max_depth`` is reached. Callers may catch this to degrade
+    (e.g. fall back to a synchronous write on the emergency path).
+    """
+
+
+def queue_key(
+    thread_id: str,
+    user_id: str | None,
+    agent_name: str | None,
+) -> tuple[str, str | None, str | None]:
+    """Return the debounce identity for a memory update target."""
+    return (thread_id, user_id, agent_name)
+
+
 @dataclass
 class ConversationContext:
     """Context for a conversation to be processed for memory update."""
@@ -27,8 +60,14 @@ class ConversationContext:
     agent_name: str | None = None
     user_id: str | None = None
     trace_id: str | None = None
-    correction_detected: bool = False
-    reinforcement_detected: bool = False
+    signals: frozenset[str] = field(default_factory=frozenset)
+    # Emergency (summarization) flushes bypass the updater's index watermark:
+    # the subset they carry is a one-shot "extract before removal" snapshot whose
+    # own length would otherwise regress the conversation watermark. Such contexts
+    # also coexist with (do not replace) a pending normal update for the same key
+    # so a flush cannot drop a pending normal update's un-extracted tail. See
+    # ``_enqueue_locked``'s match-key + backpressure handling.
+    bypass_watermark: bool = False
 
 
 class MemoryUpdateQueue:
@@ -43,7 +82,7 @@ class MemoryUpdateQueue:
         """Initialize the memory update queue with injected config + updater."""
         self._config = config
         self._updater = updater
-        self._queue: list[ConversationContext] = []
+        self._items: list[ConversationContext] = []
         self._lock = threading.Lock()
         self._timer: threading.Timer | None = None
         self._processing = False
@@ -54,15 +93,6 @@ class MemoryUpdateQueue:
         self._processing_thread: threading.Thread | None = None
         self._reprocess_pending = False
 
-    @staticmethod
-    def _queue_key(
-        thread_id: str,
-        user_id: str | None,
-        agent_name: str | None,
-    ) -> tuple[str, str | None, str | None]:
-        """Return the debounce identity for a memory update target."""
-        return (thread_id, user_id, agent_name)
-
     def add(
         self,
         thread_id: str,
@@ -70,8 +100,7 @@ class MemoryUpdateQueue:
         agent_name: str | None = None,
         user_id: str | None = None,
         trace_id: str | None = None,
-        correction_detected: bool = False,
-        reinforcement_detected: bool = False,
+        signals: frozenset[str] | None = None,
     ) -> None:
         """Add a conversation to the update queue.
 
@@ -84,8 +113,9 @@ class MemoryUpdateQueue:
                 raw threads).
             trace_id: Request trace id captured at enqueue time so the
                 later Timer thread can attach it to memory LLM tracing metadata.
-            correction_detected: Whether recent turns include an explicit correction signal.
-            reinforcement_detected: Whether recent turns include a positive reinforcement signal.
+            signals: Signal classes detected in the conversation (correction /
+                reinforcement / preference / ...), used as extraction hints. Any
+                signal is admitted under backpressure.
         """
         with self._lock:
             self._enqueue_locked(
@@ -94,12 +124,12 @@ class MemoryUpdateQueue:
                 agent_name=agent_name,
                 user_id=user_id,
                 trace_id=trace_id,
-                correction_detected=correction_detected,
-                reinforcement_detected=reinforcement_detected,
+                signals=frozenset(signals) if signals else frozenset(),
+                bypass_watermark=False,
             )
             self._reset_timer()
 
-        logger.info("Memory update queued for thread %s, queue size: %d", thread_id, len(self._queue))
+        logger.info("Memory update queued for thread %s, queue size: %d", thread_id, len(self._items))
 
     def add_nowait(
         self,
@@ -108,8 +138,7 @@ class MemoryUpdateQueue:
         agent_name: str | None = None,
         user_id: str | None = None,
         trace_id: str | None = None,
-        correction_detected: bool = False,
-        reinforcement_detected: bool = False,
+        signals: frozenset[str] | None = None,
     ) -> None:
         """Add a conversation and start processing immediately in the background."""
         with self._lock:
@@ -119,12 +148,12 @@ class MemoryUpdateQueue:
                 agent_name=agent_name,
                 user_id=user_id,
                 trace_id=trace_id,
-                correction_detected=correction_detected,
-                reinforcement_detected=reinforcement_detected,
+                signals=frozenset(signals) if signals else frozenset(),
+                bypass_watermark=True,
             )
             self._schedule_timer(0)
 
-        logger.info("Memory update queued for immediate processing on thread %s, queue size: %d", thread_id, len(self._queue))
+        logger.info("Memory update queued for immediate processing on thread %s, queue size: %d", thread_id, len(self._items))
 
     def _enqueue_locked(
         self,
@@ -134,28 +163,45 @@ class MemoryUpdateQueue:
         agent_name: str | None,
         user_id: str | None,
         trace_id: str | None,
-        correction_detected: bool,
-        reinforcement_detected: bool,
-    ) -> None:
-        queue_key = self._queue_key(thread_id, user_id, agent_name)
-        existing_context = next(
-            (context for context in self._queue if self._queue_key(context.thread_id, context.user_id, context.agent_name) == queue_key),
+        signals: frozenset[str],
+        bypass_watermark: bool = False,
+    ) -> ConversationContext:
+        key = queue_key(thread_id, user_id, agent_name)
+        # Emergency (bypass) and normal updates coexist: the match key includes
+        # ``bypass_watermark`` so a summarization flush (bypass=True) never
+        # replaces a pending normal update for the same (thread, user, agent) --
+        # replacing it would drop the normal update's un-extracted tail, which
+        # the next turn may not re-feed if the user stops. Both are processed
+        # independently instead.
+        existing = next(
+            (c for c in self._items if queue_key(c.thread_id, c.user_id, c.agent_name) == key and c.bypass_watermark == bypass_watermark),
             None,
         )
-        merged_correction_detected = correction_detected or (existing_context.correction_detected if existing_context is not None else False)
-        merged_reinforcement_detected = reinforcement_detected or (existing_context.reinforcement_detected if existing_context is not None else False)
+        # Backpressure: once depth reaches the cap, reject NEW non-signal normal
+        # items. Same-key updates merge (do not grow depth); signal-bearing items
+        # and emergency (bypass) flushes are always admitted. Signals capture
+        # important memories, and the emergency path captures messages about to
+        # be removed by summarization -- neither can be re-fed next turn, so
+        # shedding them under load would lose data rather than merely defer it.
+        max_depth = self._config.queue_max_depth
+        if max_depth > 0 and not bypass_watermark and not signals and existing is None and len(self._items) >= max_depth:
+            raise QueueFull(f"memory update queue is full (depth {len(self._items)} >= {max_depth}); non-signal update for thread {thread_id} rejected")
+
+        # Merge by signal union: a signal seen on any update for this key stays.
+        merged_signals = signals | (existing.signals if existing is not None else frozenset())
         context = ConversationContext(
             thread_id=thread_id,
             messages=messages,
             agent_name=agent_name,
             user_id=user_id,
             trace_id=trace_id,
-            correction_detected=merged_correction_detected,
-            reinforcement_detected=merged_reinforcement_detected,
+            signals=merged_signals,
+            bypass_watermark=bypass_watermark,
         )
-
-        self._queue = [context for context in self._queue if self._queue_key(context.thread_id, context.user_id, context.agent_name) != queue_key]
-        self._queue.append(context)
+        if existing is not None:
+            self._items = [c for c in self._items if not (queue_key(c.thread_id, c.user_id, c.agent_name) == key and c.bypass_watermark == bypass_watermark)]
+        self._items.append(context)
+        return context
 
     def _reset_timer(self) -> None:
         """Reset the debounce timer."""
@@ -196,13 +242,13 @@ class MemoryUpdateQueue:
                 self._reprocess_pending = True
                 return
 
-            if not self._queue:
+            if not self._items:
                 return
 
             self._processing = True
             self._processing_thread = threading.current_thread()
-            contexts_to_process = self._queue.copy()
-            self._queue.clear()
+            contexts_to_process = self._items
+            self._items = []
             self._timer = None
 
         logger.info("Processing %d queued memory updates", len(contexts_to_process))
@@ -217,10 +263,10 @@ class MemoryUpdateQueue:
                         messages=context.messages,
                         thread_id=context.thread_id,
                         agent_name=context.agent_name,
-                        correction_detected=context.correction_detected,
-                        reinforcement_detected=context.reinforcement_detected,
+                        signals=context.signals,
                         user_id=context.user_id,
                         trace_id=context.trace_id,
+                        bypass_watermark=context.bypass_watermark,
                     )
                     if success:
                         succeeded += 1
@@ -248,9 +294,16 @@ class MemoryUpdateQueue:
             with self._lock:
                 self._processing = False
                 self._processing_thread = None
+                # Reschedule inside the lock: ``_schedule_timer`` read-cancels-
+                # reassigns ``self._timer`` non-atomically, and a concurrent
+                # ``add``'s ``_reset_timer`` (also under the lock) touches the
+                # same field. Holding the lock makes the reschedule atomic w.r.t.
+                # ``add``. ``_schedule_timer`` only calls ``Timer.start()`` (no
+                # synchronous lock acquisition), so this cannot deadlock.
                 if self._reprocess_pending:
                     self._reprocess_pending = False
-                    if self._queue:
+                    if self._items:
+                        # New work arrived mid-processing: re-run immediately.
                         self._schedule_timer(0)
 
     def flush(self, *, skip_inter_item_delay: bool = False) -> None:
@@ -306,7 +359,7 @@ class MemoryUpdateQueue:
         # (1) Wait for an in-flight _process_queue first (bounded). Otherwise
         # flush() would see _processing=True, no-op, and we would report
         # success while that worker is still mid-LLM-call on a daemon thread
-        # that exit will kill — losing the contexts it already pulled out.
+        # that exit will kill - losing the contexts it already pulled out.
         with self._lock:
             in_flight = self._processing_thread
         if in_flight is not None:
@@ -356,7 +409,7 @@ class MemoryUpdateQueue:
             if self._timer is not None:
                 self._timer.cancel()
                 self._timer = None
-            self._queue.clear()
+            self._items = []
             self._processing = False
             self._processing_thread = None
             self._reprocess_pending = False
@@ -365,7 +418,7 @@ class MemoryUpdateQueue:
     def pending_count(self) -> int:
         """Get the number of pending updates."""
         with self._lock:
-            return len(self._queue)
+            return len(self._items)
 
     @property
     def is_processing(self) -> bool:

backend/packages/harness/deerflow/agents/memory/backends/deermem/deermem/core/updater.py

@@ -10,10 +10,12 @@ import logging
 import math
 import re
 import uuid
+from collections import OrderedDict
 from datetime import UTC, datetime, timedelta
 from typing import Any
 
 from ..config import DeerMemConfig
+from .message_processing import detect_signals, extract_message_text
 from .prompt import (
     format_conversation_for_update,
     load_prompt,
@@ -610,6 +612,52 @@ def _escape_memory_for_prompt(memory: Any) -> Any:
     return memory
 
 
+def _memory_with_manual_markers(memory: Any) -> Any:
+    """Return a deep copy of ``memory`` with ``[MANUAL]`` prefixed onto the
+    content of manually-authored facts (``source.type == "manual"``).
+
+    The marker is a prompt-only signal that tells the extraction LLM a fact is a
+    high-trust user edit; the persisted memory is untouched (this copy is only
+    fed to the prompt). Idempotent: a fact already carrying the prefix is not
+    double-marked.
+    """
+    display = copy.deepcopy(memory)
+    if not isinstance(display, dict):
+        return display
+    for fact in display.get("facts", []):
+        if not isinstance(fact, dict):
+            continue
+        src = fact.get("source")
+        src_type = src.get("type") if isinstance(src, dict) else None
+        if src_type == "manual":
+            content = fact.get("content")
+            if isinstance(content, str) and not content.startswith("[MANUAL]"):
+                fact["content"] = "[MANUAL] " + content
+    return display
+
+
+def _message_identity(msg: Any) -> tuple[str, ...] | None:
+    """Return a hashable identity for ``msg`` for watermark tracking.
+
+    The watermark is content/identity based rather than index based so it stays
+    valid when summarization removes the conversation front (an index watermark
+    would point at the wrong message after a front removal, silently skipping
+    un-extracted turns). Prefers the langgraph message ``id`` (unique, robust to
+    duplicate content); falls back to ``(type, content)`` when no id is set
+    (e.g. plain ``HumanMessage(content=...)`` in tests). Returns ``None`` for a
+    message with neither id nor extractable text -- the caller then feeds the
+    full list, which is safe over-extraction and never loss.
+    """
+    mid = getattr(msg, "id", None)
+    if isinstance(mid, str) and mid:
+        return ("id", mid)
+    text = extract_message_text(msg)
+    if not text:
+        return None
+    msg_type = getattr(msg, "type", "") or ""
+    return ("content", msg_type, text)
+
+
 class MemoryUpdater:
     """Updates memory using LLM based on conversation context."""
 
@@ -632,6 +680,12 @@ class MemoryUpdater:
         self._llm = llm
         self._prompts_dir = prompts_dir
         self._callbacks = callbacks
+        # Watermark: last-extracted message identity per (thread_id, user_id,
+        # agent_name), held in memory so a restart re-extracts one batch. The
+        # cache is a bounded LRU (config.watermark_max_keys) so a long-lived
+        # gateway handling many threads cannot grow it without limit; a dropped
+        # key re-extracts one batch on that thread's next turn.
+        self._watermarks: OrderedDict[tuple[str | None, str | None, str | None], tuple[str, ...] | None] = OrderedDict()
 
     # ── Data access + fact CRUD (formerly module-level functions; use self._storage) ──
 
@@ -917,37 +971,45 @@ class MemoryUpdater:
             raise OSError(f"Failed to save memory data after updating fact '{fact_id}'")
         return updated_memory
 
-    def _build_correction_hint(
-        self,
-        correction_detected: bool,
-        reinforcement_detected: bool,
-    ) -> str:
-        """Build optional prompt hints for correction and reinforcement signals."""
-        correction_hint = ""
-        if correction_detected:
-            correction_hint = (
+    def _build_signal_hints(self, signals: frozenset[str]) -> str:
+        """Build optional prompt hints for the detected signal classes.
+
+        Each present signal contributes one instruction nudging the extraction
+        LLM toward the right category and confidence. The variable is still
+        rendered into the template's ``{correction_hint}`` slot (the name is
+        historical -- it now carries the full signal-hint set, plus the manual
+        fact note appended by :meth:`_prepare_update_prompt`).
+        """
+        hints: list[str] = []
+        if "correction" in signals:
+            hints.append(
                 "IMPORTANT: Explicit correction signals were detected in this conversation. "
                 "Pay special attention to what the agent got wrong, what the user corrected, "
                 "and record the correct approach as a fact with category "
                 '"correction" and confidence >= 0.95 when appropriate.'
             )
-        if reinforcement_detected:
-            reinforcement_hint = (
+        if "reinforcement" in signals:
+            hints.append(
                 "IMPORTANT: Positive reinforcement signals were detected in this conversation. "
                 "The user explicitly confirmed the agent's approach was correct or helpful. "
                 "Record the confirmed approach, style, or preference as a fact with category "
                 '"preference" or "behavior" and confidence >= 0.9 when appropriate.'
             )
-            correction_hint = (correction_hint + "\n" + reinforcement_hint).strip() if correction_hint else reinforcement_hint
-
-        return correction_hint
+        if "preference" in signals:
+            hints.append('IMPORTANT: A preference signal was detected. Record the user\'s stated preference or dislike as a fact with category "preference" and high confidence.')
+        if "identity" in signals:
+            hints.append('IMPORTANT: An identity signal was detected. Record the user\'s stated role, profession, or background as a fact with category "identity" and high confidence.')
+        if "goal" in signals:
+            hints.append('IMPORTANT: A goal signal was detected. Record the user\'s stated objective or intent as a fact with category "goal" and high confidence.')
+        if "decision" in signals:
+            hints.append('IMPORTANT: A decision signal was detected. Record the user\'s decision or chosen option as a fact with category "decision" and high confidence.')
+        return "\n".join(hints)
 
     def _prepare_update_prompt(
         self,
         messages: list[Any],
         agent_name: str | None,
-        correction_detected: bool,
-        reinforcement_detected: bool,
+        signals: frozenset[str],
         user_id: str | None = None,
     ) -> tuple[dict[str, Any], list[Any]] | None:
         """Load memory and build the update prompt for a conversation."""
@@ -960,10 +1022,16 @@ class MemoryUpdater:
         if not conversation_text.strip():
             return None
 
-        correction_hint = self._build_correction_hint(
-            correction_detected=correction_detected,
-            reinforcement_detected=reinforcement_detected,
-        )
+        correction_hint = self._build_signal_hints(signals)
+
+        # Manual-fact signal: tag high-trust user-authored facts with a [MANUAL]
+        # prefix in the prompt's current_memory and instruct the model to preserve
+        # them unless the new conversation is an explicit, unambiguous correction.
+        display_memory = current_memory
+        if self._has_manual_facts(current_memory):
+            display_memory = _memory_with_manual_markers(current_memory)
+            manual_hint = "NOTE: Facts marked [MANUAL] are high-trust user-authored edits. Update them only when the new conversation is an explicit, unambiguous correction; otherwise preserve them as-is."
+            correction_hint = (correction_hint + "\n" + manual_hint).strip() if correction_hint else manual_hint
 
         # ── Build staleness review section ──
         staleness_section = ""
@@ -986,7 +1054,7 @@ class MemoryUpdater:
                 )
 
         variables = {
-            "current_memory": json.dumps(_escape_memory_for_prompt(current_memory), indent=2, ensure_ascii=False),
+            "current_memory": json.dumps(_escape_memory_for_prompt(display_memory), indent=2, ensure_ascii=False),
             "conversation": conversation_text,
             "correction_hint": correction_hint,
             "staleness_review_section": staleness_section,
@@ -995,6 +1063,45 @@ class MemoryUpdater:
         prompt = load_prompt_messages("memory_update", variables, agent_name=agent_name, prompts_dir=self._prompts_dir)
         return current_memory, prompt
 
+    def _has_manual_facts(self, memory: dict[str, Any]) -> bool:
+        """Return whether ``memory`` contains any user-authored (manual) fact."""
+        return any(isinstance(f, dict) and isinstance(f.get("source"), dict) and f.get("source", {}).get("type") == "manual" for f in memory.get("facts", []))
+
+    def _emit_extraction_metrics(
+        self,
+        metrics: dict[str, Any],
+        *,
+        thread_id: str | None,
+        user_id: str | None,
+        trace_id: str | None,
+        model_name: str | None,
+        response: Any,
+        success: bool,
+    ) -> None:
+        """Invoke the post-extraction observability callback (Langfuse span etc.).
+
+        No-op when ``extraction_callback`` is unset (default). Exceptions from
+        the callback are logged and swallowed so observability never breaks the
+        update path.
+        """
+        callback = self._config.extraction_callback
+        if callback is None:
+            return
+        usage = getattr(response, "usage_metadata", None)
+        payload: dict[str, Any] = {
+            "thread_id": thread_id,
+            "user_id": user_id,
+            "trace_id": trace_id,
+            "model_name": model_name,
+            "success": success,
+            "token_usage": usage if isinstance(usage, dict) else None,
+        }
+        payload.update(metrics)
+        try:
+            callback(payload)
+        except Exception:
+            logger.warning("extraction_callback raised; ignoring", exc_info=True)
+
     def _finalize_update(
         self,
         current_memory: dict[str, Any],
@@ -1002,9 +1109,18 @@ class MemoryUpdater:
         thread_id: str | None,
         agent_name: str | None,
         user_id: str | None = None,
+        *,
+        metrics: dict[str, Any] | None = None,
     ) -> bool:
         """Parse the model response, apply updates, and persist memory."""
         update_data = _parse_memory_update_response(response_content)
+        if metrics is not None:
+            extracted = update_data.get("newFacts", [])
+            extracted_list = extracted if isinstance(extracted, list) else []
+            metrics["facts_extracted"] = len(extracted_list)
+            # facts_passed_confidence / rejected_low_confidence are populated
+            # inside _apply_updates at the real confidence-filter site, so the
+            # metric tracks the actual filter rather than a re-derived copy here.
         if getattr(type(self._storage), "apply_changes", None) is not MemoryStorage.apply_changes:
             for attempt in range(3):
                 # Deep-copy before in-place mutation so a failed commit cannot
@@ -1012,7 +1128,7 @@ class MemoryUpdater:
                 # complete extraction result is reapplied to a fresh document;
                 # its trim/consolidation/delete decisions are snapshot-wide and
                 # must never be replayed as disjoint point writes.
-                updated_memory = self._apply_updates(copy.deepcopy(current_memory), update_data, thread_id)
+                updated_memory = self._apply_updates(copy.deepcopy(current_memory), update_data, thread_id, metrics=metrics)
                 updated_memory = _strip_upload_mentions_from_memory(updated_memory)
                 current_by_id = {str(fact.get("id")): fact for fact in current_memory.get("facts", [])}
                 updated_by_id = {str(fact.get("id")): fact for fact in updated_memory.get("facts", [])}
@@ -1044,7 +1160,7 @@ class MemoryUpdater:
             raise AssertionError("bounded extracted-update retry did not return or raise")
         # Deep-copy before in-place mutation so a subsequent save() failure
         # cannot corrupt the still-cached original object reference.
-        updated_memory = self._apply_updates(copy.deepcopy(current_memory), update_data, thread_id)
+        updated_memory = self._apply_updates(copy.deepcopy(current_memory), update_data, thread_id, metrics=metrics)
         updated_memory = _strip_upload_mentions_from_memory(updated_memory)
         return self._storage.save(
             updated_memory,
@@ -1058,10 +1174,11 @@ class MemoryUpdater:
         messages: list[Any],
         thread_id: str | None = None,
         agent_name: str | None = None,
-        correction_detected: bool = False,
-        reinforcement_detected: bool = False,
+        signals: frozenset[str] = frozenset(),
         user_id: str | None = None,
         trace_id: str | None = None,
+        *,
+        bypass_watermark: bool = False,
     ) -> bool:
         """Update memory asynchronously by delegating to the sync path.
 
@@ -1076,10 +1193,10 @@ class MemoryUpdater:
             messages=messages,
             thread_id=thread_id,
             agent_name=agent_name,
-            correction_detected=correction_detected,
-            reinforcement_detected=reinforcement_detected,
+            signals=signals,
             user_id=user_id,
             trace_id=trace_id,
+            bypass_watermark=bypass_watermark,
         )
 
     def _do_update_memory_sync(
@@ -1087,10 +1204,11 @@ class MemoryUpdater:
         messages: list[Any],
         thread_id: str | None = None,
         agent_name: str | None = None,
-        correction_detected: bool = False,
-        reinforcement_detected: bool = False,
+        signals: frozenset[str] = frozenset(),
         user_id: str | None = None,
         trace_id: str | None = None,
+        *,
+        bypass_watermark: bool = False,
     ) -> bool:
         """Pure-sync memory update; bind ``trace_id`` into the request-trace
         ContextVar for the worker thread, then delegate to the impl.
@@ -1110,45 +1228,128 @@ class MemoryUpdater:
                     messages=messages,
                     thread_id=thread_id,
                     agent_name=agent_name,
-                    correction_detected=correction_detected,
-                    reinforcement_detected=reinforcement_detected,
+                    signals=signals,
                     user_id=user_id,
                     trace_id=trace_id,
+                    bypass_watermark=bypass_watermark,
                 )
         return self._do_update_memory_sync_impl(
             messages=messages,
             thread_id=thread_id,
             agent_name=agent_name,
-            correction_detected=correction_detected,
-            reinforcement_detected=reinforcement_detected,
+            signals=signals,
             user_id=user_id,
             trace_id=trace_id,
+            bypass_watermark=bypass_watermark,
         )
 
+    def _watermark_get(self, key: tuple[str | None, str | None, str | None]) -> tuple[str, ...] | None:
+        """Return the watermark for ``key``, marking it most-recently-used.
+
+        Uses key presence (not value truthiness) so a stored ``None`` identity
+        still counts as a live entry for LRU ordering.
+        """
+        if key not in self._watermarks:
+            return None
+        self._watermarks.move_to_end(key)
+        return self._watermarks[key]
+
+    def _watermark_set(
+        self,
+        key: tuple[str | None, str | None, str | None],
+        value: tuple[str, ...] | None,
+    ) -> None:
+        """Store ``value`` for ``key``, evicting the least-recently-used entry
+        when the bounded LRU cache exceeds ``config.watermark_max_keys``.
+
+        A dropped key is safe: the next turn for that thread finds no watermark
+        and re-extracts one batch (the documented restart behavior). ``0`` =
+        unbounded (no eviction).
+        """
+        self._watermarks[key] = value
+        self._watermarks.move_to_end(key)
+        cap = self._config.watermark_max_keys
+        if cap > 0 and len(self._watermarks) > cap:
+            self._watermarks.popitem(last=False)
+
+    def _feed_after_watermark(
+        self,
+        watermark_key: tuple[str | None, str | None, str | None],
+        messages: list[Any],
+    ) -> list[Any]:
+        """Return the slice of ``messages`` not yet extracted.
+
+        The watermark stores the identity of the last-extracted message (see
+        :func:`_message_identity`). If that message is still present, everything
+        *after* it is fed; if it is absent (front removed by summarization, or
+        the first-ever extraction for this key) the full list is fed. Re-feeding
+        is safe over-extraction -- it never skips a turn, which is the only
+        failure direction that would lose facts.
+        """
+        last_id = self._watermark_get(watermark_key)
+        if last_id is None:
+            return messages
+        for i, msg in enumerate(messages):
+            if _message_identity(msg) == last_id:
+                return messages[i + 1 :]
+        return messages
+
     def _do_update_memory_sync_impl(
         self,
         messages: list[Any],
         thread_id: str | None = None,
         agent_name: str | None = None,
-        correction_detected: bool = False,
-        reinforcement_detected: bool = False,
+        signals: frozenset[str] = frozenset(),
         user_id: str | None = None,
         trace_id: str | None = None,
+        *,
+        bypass_watermark: bool = False,
     ) -> bool:
         """Pure-sync memory update using ``model.invoke()``.
 
         Uses the *sync* LLM call path so no event loop is created.  This
         guarantees that the langchain provider's globally cached async
         httpx ``AsyncClient`` / connection pool (the one shared with the
-        lead agent) is never touched — no cross-loop connection reuse is
+        lead agent) is never touched - no cross-loop connection reuse is
         possible.
+
+        Watermark: the middleware passes the full conversation each turn, so
+        without skipping already-extracted turns every update re-feeds old
+        messages. The watermark stores the identity of the last-extracted
+        message (content/id based, in-memory only) so it stays correct when
+        summarization removes the conversation front; a restart loses it and
+        re-extracts one batch. ``bypass_watermark`` is set by the emergency
+        (summarization) flush path: the subset it carries is a one-shot
+        "extract before removal" snapshot, so it is fed in full and does not
+        read or advance the conversation watermark (advancing it from the
+        subset's own length would regress the watermark and skip un-extracted
+        tail turns on the next normal feed).
         """
+        metrics: dict[str, Any] = {}
+        response: Any = None
+        model_name: str | None = None
+        success = False
+        attempted = False
         try:
+            watermark_key = (thread_id, user_id, agent_name)
+            if bypass_watermark:
+                # Emergency flush: extract the carried subset in full.
+                feed_messages = messages
+            else:
+                feed_messages = self._feed_after_watermark(watermark_key, messages)
+            if not feed_messages:
+                logger.debug("Memory update skipped: no new messages since watermark (thread=%s)", thread_id)
+                return True
+            # Re-detect signals on the post-watermark feed so extraction hints
+            # reference only turns the LLM will actually see. The admission-time
+            # ``signals`` (detected on the full conversation in DeerMem) already
+            # served their purpose (backpressure admission at enqueue); the hint
+            # is a soft nudge and must not point at turns the watermark excluded.
+            feed_signals = detect_signals(feed_messages, patterns_dir=self._config.patterns_dir)
             prepared = self._prepare_update_prompt(
-                messages=messages,
+                messages=feed_messages,
                 agent_name=agent_name,
-                correction_detected=correction_detected,
-                reinforcement_detected=reinforcement_detected,
+                signals=feed_signals,
                 user_id=user_id,
             )
             if prepared is None:
@@ -1173,30 +1374,56 @@ class MemoryUpdater:
                     model_name=model_name,
                 )
             logger.info("Invoking memory-update LLM (thread=%s trace_id=%s)", thread_id, trace_id)
+            attempted = True
             response = model.invoke(prompt, config=invoke_config)
-            return self._finalize_update(
+            success = self._finalize_update(
                 current_memory=current_memory,
                 response_content=response.content,
                 thread_id=thread_id,
                 agent_name=agent_name,
                 user_id=user_id,
+                metrics=metrics,
             )
+            if success and not bypass_watermark:
+                # Advance the watermark to the last message fed (the feed is a
+                # suffix, so this is messages[-1]). Skipped on the emergency
+                # path -- the subset's last message is older than the
+                # conversation's latest, so advancing from it would regress.
+                self._watermark_set(watermark_key, _message_identity(messages[-1]))
+            return success
         except json.JSONDecodeError as e:
             logger.warning("Failed to parse LLM response for memory update: %s", e)
             return False
         except Exception as e:
             logger.exception("Memory update failed: %s", e)
             return False
+        finally:
+            # Emit metrics even when _finalize_update (or invoke) raises, so the
+            # observability callback sees exception failures (parse errors,
+            # storage errors after retry) rather than only the happy path. The
+            # pre-attempt early returns (no new messages, empty conversation, no
+            # model) do not emit, matching the prior behavior.
+            if attempted:
+                self._emit_extraction_metrics(
+                    metrics,
+                    thread_id=thread_id,
+                    user_id=user_id,
+                    trace_id=trace_id,
+                    model_name=model_name,
+                    response=response,
+                    success=success,
+                )
 
     def update_memory(
         self,
         messages: list[Any],
         thread_id: str | None = None,
         agent_name: str | None = None,
-        correction_detected: bool = False,
-        reinforcement_detected: bool = False,
+        signals: frozenset[str] = frozenset(),
         user_id: str | None = None,
         trace_id: str | None = None,
+        *,
+        bypass_watermark: bool = False,
     ) -> bool:
         """Synchronously update memory using the sync LLM path.
 
@@ -1213,12 +1440,15 @@ class MemoryUpdater:
             messages: List of conversation messages.
             thread_id: Optional thread ID for tracking source.
             agent_name: If provided, updates per-agent memory. If None, updates global memory.
-            correction_detected: Whether recent turns include an explicit correction signal.
-            reinforcement_detected: Whether recent turns include a positive reinforcement signal.
+            signals: Signal classes detected in the conversation (correction /
+                reinforcement / preference / ...), used as extraction hints.
             user_id: If provided, scopes memory to a specific user.
 
         Returns:
-            True if update was successful, False otherwise.
+            True if the update persisted. False on any failure (no content,
+            unparseable response, LLM error); failures are swallowed (best-effort)
+            -- a failed update is re-fed on the next conversation turn because the
+            watermark does not advance on failure.
         """
         try:
             loop = asyncio.get_running_loop()
@@ -1232,10 +1462,10 @@ class MemoryUpdater:
                     messages=messages,
                     thread_id=thread_id,
                     agent_name=agent_name,
-                    correction_detected=correction_detected,
-                    reinforcement_detected=reinforcement_detected,
+                    signals=signals,
                     user_id=user_id,
                     trace_id=trace_id,
+                    bypass_watermark=bypass_watermark,
                 )
                 return future.result()
             except Exception:
@@ -1246,10 +1476,10 @@ class MemoryUpdater:
             messages=messages,
             thread_id=thread_id,
             agent_name=agent_name,
-            correction_detected=correction_detected,
-            reinforcement_detected=reinforcement_detected,
+            signals=signals,
             user_id=user_id,
             trace_id=trace_id,
+            bypass_watermark=bypass_watermark,
         )
 
     def _apply_updates(
@@ -1257,6 +1487,8 @@ class MemoryUpdater:
         current_memory: dict[str, Any],
         update_data: dict[str, Any],
         thread_id: str | None = None,
+        *,
+        metrics: dict[str, Any] | None = None,
     ) -> dict[str, Any]:
         """Apply LLM-generated updates to memory.
 
@@ -1264,6 +1496,11 @@ class MemoryUpdater:
             current_memory: Current memory data.
             update_data: Updates from LLM.
             thread_id: Optional thread ID for tracking.
+            metrics: Optional observability dict. When provided, populated with
+                ``facts_passed_confidence`` / ``rejected_low_confidence`` counted
+                at the real confidence-filter site below (the only acceptance
+                gate for new facts), so the metric cannot drift from the actual
+                filter the way a re-derived count in the caller could.
 
         Returns:
             Updated memory data.
@@ -1398,9 +1635,18 @@ class MemoryUpdater:
         # Creation-time lifetime cap shared with the consolidation path below, so
         # both fact-creation sites apply the identical bound in one place.
         creation_cap = int(config.staleness_age_days * config.staleness_max_lifetime_multiplier)
+        # Counted at the confidence-gate site (the only real accept filter for new
+        # facts) so the ``facts_passed_confidence`` metric mirrors the actual
+        # filter and cannot drift from it. Facts below the threshold are the
+        # reject count; duplicate / empty / over-cap facts that pass the
+        # threshold are still counted here -- the metric is a confidence-gate
+        # signal (the host's rejection-rate warning monitors confidence
+        # filtering, not dedup / over-cap), not a persisted-fact count.
+        passed_threshold = 0
         for fact in new_facts:
             confidence = fact.get("confidence", 0.5)
             if confidence >= config.fact_confidence_threshold:
+                passed_threshold += 1
                 raw_content = fact.get("content", "")
                 if not isinstance(raw_content, str):
                     continue
@@ -1439,6 +1685,10 @@ class MemoryUpdater:
                 if fact_key is not None:
                     existing_fact_keys.add(fact_key)
 
+        if metrics is not None:
+            metrics["facts_passed_confidence"] = passed_threshold
+            metrics["rejected_low_confidence"] = len(new_facts) - passed_threshold
+
         # Enforce max facts limit (coerced confidence -- see _trim_facts_to_max).
         current_memory["facts"] = _trim_facts_to_max(current_memory["facts"], config.max_facts)
 

backend/packages/harness/deerflow/agents/memory/manager.py

@@ -656,6 +656,51 @@ def _host_default_llm() -> Any:
         return None
 
 
+def _host_default_extraction_callback(payload: Any) -> None:
+    """deer-flow default for DeerMem's ``extraction_callback`` slot.
+
+    Logs post-extraction metrics (token usage, facts passing/rejected by the
+    confidence filter, gate rejection rate) for ops observability, and flags a
+    high rejection rate
+    (>60%) so a prompt/threshold regression is visible without inspecting every
+    trace. A Langfuse-aware callback can replace this to emit a dedicated
+    extraction span; the metrics keys are stable for that handoff. Exceptions
+    are never raised (the DeerMem side already wraps the call).
+    """
+    if not isinstance(payload, dict):
+        return
+    extracted = payload.get("facts_extracted")
+    passed_confidence = payload.get("facts_passed_confidence")
+    rejected = payload.get("rejected_low_confidence", 0)
+    thread_id = payload.get("thread_id")
+    model_name = payload.get("model_name")
+    if isinstance(extracted, int) and isinstance(passed_confidence, int) and extracted > 0:
+        rejection_rate = (extracted - passed_confidence) / extracted
+        logger.info(
+            "Memory extraction metrics: thread=%s model=%s extracted=%d passed_confidence=%d rejected=%d rejection_rate=%.2f",
+            thread_id,
+            model_name,
+            extracted,
+            passed_confidence,
+            rejected,
+            rejection_rate,
+        )
+        if rejection_rate > 0.6:
+            logger.warning(
+                "Memory extraction rejection rate %.0f%% exceeds 60%% - review extraction prompt / confidence threshold (thread=%s)",
+                rejection_rate * 100,
+                thread_id,
+            )
+    else:
+        logger.info(
+            "Memory extraction metrics: thread=%s model=%s success=%s token_usage=%s",
+            thread_id,
+            model_name,
+            payload.get("success"),
+            payload.get("token_usage"),
+        )
+
+
 def _collect_host_hooks() -> dict[str, Any]:
     """Provide host hook callables for backends to consume in ``from_config``.
 
@@ -674,6 +719,7 @@ def _collect_host_hooks() -> dict[str, Any]:
         "should_keep_hidden_message": _host_default_should_keep_hidden_message,
         "trace_context_manager": request_trace_context,
         "host_llm_factory": _host_default_llm,
+        "extraction_callback": _host_default_extraction_callback,
     }
 
 

backend/packages/harness/deerflow/agents/middlewares/model_length_finish_reason_middleware.py

@@ -0,0 +1,130 @@
+"""Surface provider length-capped model responses as run stop reasons.
+
+Background — see issue bytedance/deer-flow#4271.
+
+Some providers stop generation because the output budget is exhausted and
+surface that through ``finish_reason='length'`` while still returning assistant
+content. DeerFlow should preserve that content for audit, but it should not
+silently treat the run as an uncapped clean completion when the provider has
+explicitly signaled truncation.
+
+This middleware keeps that boundary narrow:
+- it only marks a run-level stop reason when the final AIMessage is capped
+  by a provider length signal and still has visible content;
+- it never rewrites the assistant content or reparses XML-like text into a
+  tool call;
+- it ignores any response that still carries tool-call intent, malformed
+  tool-call metadata, or no visible content, so only terminal assistant
+  responses with visible content can be marked capped.
+
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import Any, override
+
+from langchain.agents import AgentState
+from langchain.agents.middleware import AgentMiddleware
+from langchain_core.messages import AIMessage
+from langgraph.runtime import Runtime
+
+from deerflow.agents.middlewares.model_length_termination_detectors import (
+    ModelLengthTermination,
+    ModelLengthTerminationDetector,
+    default_detectors,
+)
+
+MODEL_LENGTH_CAPPED_STOP_REASON = "model_length_capped"
+logger = logging.getLogger(__name__)
+
+
+def _has_tool_call_intent_or_error(message: AIMessage) -> bool:
+    if message.tool_calls or getattr(message, "invalid_tool_calls", None):
+        return True
+    additional_kwargs = message.additional_kwargs or {}
+    return bool(additional_kwargs.get("tool_calls") or additional_kwargs.get("function_call"))
+
+
+def _has_visible_content(message: AIMessage) -> bool:
+    content = message.content
+    if isinstance(content, str):
+        return bool(content.strip())
+    if isinstance(content, list):
+        for block in content:
+            if isinstance(block, str) and block.strip():
+                return True
+            if isinstance(block, dict) and block.get("type") in {"text", "output_text"}:
+                text = block.get("text")
+                if isinstance(text, str) and text.strip():
+                    return True
+    return False
+
+
+class ModelLengthFinishReasonMiddleware(AgentMiddleware[AgentState]):
+    """Record provider length caps for terminal assistant responses with content.
+
+    If the last AIMessage still carries tool-call intent, this middleware
+    leaves it alone and lets the normal tool-handling path decide what to do.
+    """
+
+    def __init__(self, detectors: list[ModelLengthTerminationDetector] | None = None) -> None:
+        super().__init__()
+        self._detectors: list[ModelLengthTerminationDetector] = list(detectors) if detectors else default_detectors()
+
+    def _detect(self, message: AIMessage) -> ModelLengthTermination | None:
+        for detector in self._detectors:
+            try:
+                hit = detector.detect(message)
+            except Exception:  # noqa: BLE001 - provider detectors must not break a run
+                logger.exception("ModelLengthTerminationDetector %r raised; treating as no-match", getattr(detector, "name", type(detector).__name__))
+                continue
+            if hit is not None:
+                return hit
+        return None
+
+    def _apply(self, state: AgentState, runtime: Runtime) -> dict[str, Any] | None:
+        messages = list(state.get("messages") or [])
+        if not messages or not isinstance(messages[-1], AIMessage):
+            return None
+
+        last = messages[-1]
+        if _has_tool_call_intent_or_error(last):
+            return None
+        if not _has_visible_content(last):
+            return None
+
+        termination = self._detect(last)
+        if termination is None:
+            return None
+
+        ctx = getattr(runtime, "context", None)
+        thread_id = ctx.get("thread_id") if isinstance(ctx, dict) else None
+        run_id = ctx.get("run_id") if isinstance(ctx, dict) else None
+        stamped_stop_reason = False
+        if isinstance(ctx, dict):
+            # Preserve any earlier cap reason carried across hidden continuation turns.
+            if "stop_reason" not in ctx:
+                ctx["stop_reason"] = MODEL_LENGTH_CAPPED_STOP_REASON
+                stamped_stop_reason = True
+        logger.info(
+            "Provider model length cap detected",
+            extra={
+                "thread_id": thread_id,
+                "run_id": run_id,
+                "message_id": getattr(last, "id", None),
+                "detector": termination.detector,
+                "reason_field": termination.reason_field,
+                "reason_value": termination.reason_value,
+                "stamped_stop_reason": stamped_stop_reason,
+            },
+        )
+        return None
+
+    @override
+    def after_model(self, state: AgentState, runtime: Runtime) -> dict[str, Any] | None:
+        return self._apply(state, runtime)
+
+    @override
+    async def aafter_model(self, state: AgentState, runtime: Runtime) -> dict[str, Any] | None:
+        return self._apply(state, runtime)

backend/packages/harness/deerflow/agents/middlewares/model_length_termination_detectors.py

@@ -0,0 +1,126 @@
+"""Detectors for provider-side model length termination signals.
+
+Different providers report "the response hit the output-token limit" through
+different fields and values. Keep those provider details here so
+``ModelLengthFinishReasonMiddleware`` can stay focused on when to mark a run,
+not on which provider emitted which spelling.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any, Protocol, runtime_checkable
+
+from langchain_core.messages import AIMessage
+
+
+@dataclass(frozen=True)
+class ModelLengthTermination:
+    """A detected model-output length cap."""
+
+    detector: str
+    reason_field: str
+    reason_value: str
+    extras: dict[str, Any] = field(default_factory=dict)
+
+
+@runtime_checkable
+class ModelLengthTerminationDetector(Protocol):
+    """Strategy interface for provider length-cap detection."""
+
+    name: str
+
+    def detect(self, message: AIMessage) -> ModelLengthTermination | None:
+        """Return a hit when *message* indicates output length truncation."""
+        ...
+
+
+def _get_metadata_value(message: AIMessage, field_name: str) -> str | None:
+    """Read a string metadata value from common LangChain provider fields."""
+    for container_name in ("response_metadata", "additional_kwargs"):
+        container = getattr(message, container_name, None) or {}
+        if not isinstance(container, dict):
+            continue
+        value = container.get(field_name)
+        if isinstance(value, str) and value:
+            return value
+    return None
+
+
+class OpenAICompatibleLengthDetector:
+    """OpenAI-compatible ``finish_reason == "length"`` signal."""
+
+    name = "openai_compatible_length"
+
+    def __init__(self, finish_reasons: list[str] | tuple[str, ...] | None = None) -> None:
+        configured = finish_reasons if finish_reasons is not None else ("length",)
+        self._finish_reasons: frozenset[str] = frozenset(r.lower() for r in configured)
+
+    def detect(self, message: AIMessage) -> ModelLengthTermination | None:
+        value = _get_metadata_value(message, "finish_reason")
+        if value is None or value.lower() not in self._finish_reasons:
+            return None
+        return ModelLengthTermination(
+            detector=self.name,
+            reason_field="finish_reason",
+            reason_value=value,
+        )
+
+
+class AnthropicMaxTokensDetector:
+    """Anthropic ``stop_reason == "max_tokens"`` signal."""
+
+    name = "anthropic_max_tokens"
+
+    def __init__(self, stop_reasons: list[str] | tuple[str, ...] | None = None) -> None:
+        configured = stop_reasons if stop_reasons is not None else ("max_tokens",)
+        self._stop_reasons: frozenset[str] = frozenset(r.lower() for r in configured)
+
+    def detect(self, message: AIMessage) -> ModelLengthTermination | None:
+        value = _get_metadata_value(message, "stop_reason")
+        if value is None or value.lower() not in self._stop_reasons:
+            return None
+        return ModelLengthTermination(
+            detector=self.name,
+            reason_field="stop_reason",
+            reason_value=value,
+        )
+
+
+class GeminiMaxTokensDetector:
+    """Gemini / Vertex AI ``finish_reason == "MAX_TOKENS"`` signal."""
+
+    name = "gemini_max_tokens"
+
+    def __init__(self, finish_reasons: list[str] | tuple[str, ...] | None = None) -> None:
+        configured = finish_reasons if finish_reasons is not None else ("MAX_TOKENS",)
+        self._finish_reasons: frozenset[str] = frozenset(r.upper() for r in configured)
+
+    def detect(self, message: AIMessage) -> ModelLengthTermination | None:
+        value = _get_metadata_value(message, "finish_reason")
+        if value is None or value.upper() not in self._finish_reasons:
+            return None
+        return ModelLengthTermination(
+            detector=self.name,
+            reason_field="finish_reason",
+            reason_value=value,
+        )
+
+
+def default_detectors() -> list[ModelLengthTerminationDetector]:
+    """Built-in detector set used for provider length-cap signals."""
+    return [
+        OpenAICompatibleLengthDetector(),
+        AnthropicMaxTokensDetector(),
+        GeminiMaxTokensDetector(),
+    ]
+
+
+__all__ = [
+    "AnthropicMaxTokensDetector",
+    "GeminiMaxTokensDetector",
+    "ModelLengthTermination",
+    "ModelLengthTerminationDetector",
+    "OpenAICompatibleLengthDetector",
+    "default_detectors",
+]

backend/packages/harness/deerflow/agents/middlewares/summarization_middleware.py

@@ -21,6 +21,25 @@ from deerflow.models import create_chat_model
 
 logger = logging.getLogger(__name__)
 _SUMMARY_TRIGGER_MESSAGE_NAME = "summary"
+_UNSET = object()
+# Valid non-generated summaries for the empty / too-long-to-summarize edges; these
+# short-circuit model invocation (and must not be treated as generation failures).
+_CANNED_SUMMARIES = frozenset(
+    {
+        "No previous conversation history.",
+        "Previous conversation was too long to summarize.",
+    }
+)
+
+
+class SummaryGenerationError(RuntimeError):
+    """Summary generation failed after exhausting the run-model fallback.
+
+    Raised only when a caller opts in via ``raise_on_failure`` (the manual
+    ``/compact`` path) so a real failure is reported distinctly from "nothing to
+    compact". The automatic path leaves ``raise_on_failure`` False and swallows the
+    failure, leaving compaction state unchanged for the turn.
+    """
 
 
 @dataclass(frozen=True)
@@ -82,22 +101,115 @@ class DeerFlowSummarizationMiddleware(SummarizationMiddleware):
         self,
         *args,
         before_summarization: list[BeforeSummarizationHook] | None = None,
+        app_config: Any | None = None,
+        configured_model_name: str | None = None,
+        run_model_name: str | None = None,
+        anchor_model_name: str | None = _UNSET,  # type: ignore[assignment]
         **kwargs,
     ) -> None:
         super().__init__(*args, **kwargs)
         self._before_summarization_hooks = before_summarization or []
+        # Model-ownership state. The model that actually executes the run is selected
+        # per run and is the authoritative source of truth, so the caller (lead /
+        # subagent / manual builders) supplies it directly as ``run_model_name``
+        # instead of the middleware re-deriving it from ``runtime.context`` /
+        # ``get_config()`` — those fields do not carry a custom agent's or a subagent's
+        # resolved model.
+        #
+        # ``configured_model_name`` is the explicitly configured summary model
+        # (``None`` => summarize with the run's own model). ``run_model_name`` is the
+        # model the run executes with; when they differ and the summary provider is
+        # broken (expired key, quota, outage) the run's own working model can still
+        # compact.
+        self._app_config = app_config
+        self._configured_summary_model_name = configured_model_name
+        self._run_model_name = run_model_name
         # The summary LLM call runs inside a LangGraph middleware hook, so its token
         # stream would otherwise be captured by the messages-tuple stream callback and
         # broadcast to the frontend as a phantom AI message. Tag a dedicated model copy
         # with TAG_NOSTREAM so the streaming handler skips it.
         # Keep self.model untagged so the parent's profile / ls_params inspection still works.
-        #
-        # Preserve any tags already bound on the model (e.g. "middleware:summarize" set in
-        # lead_agent/agent.py for RunJournal attribution): RunnableBinding.with_config does a
-        # shallow merge that would otherwise overwrite the existing tags list entirely.
-        existing_tags = list((getattr(self.model, "config", None) or {}).get("tags") or [])
+        self._summary_model = self._tag_nostream(self.model)
+        # ``self.model`` is the pre-built *anchor* model: it drives the parent's token
+        # counter / profile inspection and is reused verbatim by generation when a
+        # candidate matches its name. The factory builds it guarded and passes its name
+        # explicitly; direct construction (tests) mirrors the old factory choice
+        # (configured model, else default) so the passed ``model`` is the primary.
+        if anchor_model_name is _UNSET:
+            self._anchor_model_name = configured_model_name or self._default_model_name()
+        else:
+            self._anchor_model_name = anchor_model_name
+        # Nostream generation models built lazily by name and cached (None = a build
+        # that failed, so a broken candidate config is not retried every turn and does
+        # not escape the fail-open boundary).
+        self._model_cache: dict[str | None, Any] = {}
+
+    def _tag_nostream(self, model: Any) -> Any:
+        """Return a copy of ``model`` carrying TAG_NOSTREAM without clobbering tags.
+
+        lead_agent/agent.py binds "middleware:summarize" for RunJournal attribution;
+        RunnableBinding.with_config shallow-merges config, so existing tags must be
+        preserved explicitly instead of being overwritten with just [TAG_NOSTREAM].
+        """
+        existing_tags = list((getattr(model, "config", None) or {}).get("tags") or [])
         merged_tags = [*existing_tags, TAG_NOSTREAM] if TAG_NOSTREAM not in existing_tags else existing_tags
-        self._summary_model = self.model.with_config(tags=merged_tags)
+        return model.with_config(tags=merged_tags)
+
+    def _default_model_name(self) -> str | None:
+        if self._app_config is None:
+            return None
+        models = getattr(self._app_config, "models", None)
+        return models[0].name if models else None
+
+    def _generation_candidate_names(self) -> list[str | None]:
+        """Ordered summary-generation candidates by name (deduplicated).
+
+        Explicit summary model: the configured model first, then the run's own model
+        as a distinct fallback. ``model_name: null``: the run's own model only — its
+        construction is the primary, so there is no eager dependency on
+        ``config.models[0]`` (a bare default is used only when no run model was
+        resolved). A ``None`` entry means "let ``create_chat_model`` pick the default",
+        which only occurs when nothing resolves a name.
+        """
+        default = self._default_model_name()
+        if self._configured_summary_model_name is not None:
+            names = [self._configured_summary_model_name, self._run_model_name or default]
+        else:
+            names = [self._run_model_name or default]
+        deduped: list[str | None] = []
+        seen: set[str | None] = set()
+        for name in names:
+            if name in seen:
+                continue
+            seen.add(name)
+            deduped.append(name)
+        return deduped
+
+    def _model_for(self, name: str | None) -> Any | None:
+        """The nostream summary model for ``name``, built lazily and guarded.
+
+        Returns the pre-built anchor when ``name`` matches it (no rebuild), otherwise
+        constructs and caches. A construction failure is caught and cached as ``None``
+        so a broken candidate config never escapes the fail-open boundary, is never
+        retried this turn, and still lets the next candidate run.
+        """
+        if name == self._anchor_model_name:
+            return self._summary_model
+        if name in self._model_cache:
+            return self._model_cache[name]
+        try:
+            model = create_chat_model(
+                name=name,
+                thinking_enabled=False,
+                app_config=self._app_config,
+                attach_tracing=False,
+            )
+            built = self._tag_nostream(model.with_config(tags=["middleware:summarize"]))
+        except Exception:
+            logger.exception("Failed to build summary model %r; trying the next candidate", name)
+            built = None
+        self._model_cache[name] = built
+        return built
 
     @override
     def _create_summary(self, messages_to_summarize: list[AnyMessage]) -> str | None:
@@ -107,6 +219,31 @@ class DeerFlowSummarizationMiddleware(SummarizationMiddleware):
     async def _acreate_summary(self, messages_to_summarize: list[AnyMessage]) -> str | None:
         return await self._asummarize_with(messages_to_summarize)
 
+    def _prepare_summary_prompt(self, messages_to_summarize: list[AnyMessage], previous_summary: str | None) -> str | None:
+        """Return the formatted prompt, or a canned string for the empty/too-long edges.
+
+        A non-``None`` return that is not a real prompt (the two canned strings) is a
+        valid summary and short-circuits generation; ``None`` means "build a prompt".
+        """
+        if not messages_to_summarize:
+            return "No previous conversation history."
+        prompt = self._build_summary_prompt(messages_to_summarize, previous_summary=previous_summary)
+        if prompt is None:
+            return "Previous conversation was too long to summarize."
+        return prompt
+
+    @staticmethod
+    def _nonempty_summary(text: Any) -> str | None:
+        """Normalize a model response's text; a blank/whitespace-only body is a failure.
+
+        Committing ``""`` as a summary would fire the before_summarization hooks and
+        remove all prior history for an empty replacement, so an empty body is treated
+        as generation failure (try the fallback, or leave state unchanged) rather than
+        a valid summary.
+        """
+        stripped = text.strip() if isinstance(text, str) else ""
+        return stripped or None
+
     def _summarize_with(self, messages_to_summarize: list[AnyMessage], previous_summary: str | None = None) -> str | None:
         """Mirror the parent ``_create_summary`` but invoke the nostream-tagged model.
 
@@ -114,38 +251,84 @@ class DeerFlowSummarizationMiddleware(SummarizationMiddleware):
         cached and reused across concurrent runs, so a temporary swap would leak the
         ``RunnableBinding`` to other coroutines during ``await`` and break parent logic
         that inspects the raw model (``profile`` / ``_get_ls_params``).
+
+        Generation uses the run's own model (``model_name: null``) or the explicitly
+        configured summary model, falling back to the run model on failure so a broken
+        summary provider cannot disable compaction while a working model is available.
         """
-        if not messages_to_summarize:
-            return "No previous conversation history."
-        prompt = self._build_summary_prompt(messages_to_summarize, previous_summary=previous_summary)
-        if prompt is None:
-            return "Previous conversation was too long to summarize."
-        try:
-            response = self._summary_model.invoke(
-                prompt,
-                config={"metadata": {"lc_source": "summarization"}},
-            )
-            return response.text.strip()
-        except Exception:
-            logger.exception("Summary generation failed; skipping compaction this turn")
-            return None
+        prompt = self._prepare_summary_prompt(messages_to_summarize, previous_summary)
+        if prompt is None or prompt in _CANNED_SUMMARIES:
+            return prompt
+        # Walk the ordered candidates; each attempt owns its full lifecycle (lazy
+        # guarded construction -> invoke -> text extraction -> non-empty validation),
+        # and any failure at any stage falls through to the next candidate. When all
+        # candidates fail the caller leaves compaction state unchanged.
+        names = self._generation_candidate_names()
+        for index, name in enumerate(names):
+            text = self._invoke_summary(self._model_for(name), prompt, last=index == len(names) - 1)
+            if text is not None:
+                return text
+        return None
 
     async def _asummarize_with(self, messages_to_summarize: list[AnyMessage], previous_summary: str | None = None) -> str | None:
         """Async counterpart of :meth:`_summarize_with` using the nostream model."""
-        if not messages_to_summarize:
-            return "No previous conversation history."
-        prompt = self._build_summary_prompt(messages_to_summarize, previous_summary=previous_summary)
-        if prompt is None:
-            return "Previous conversation was too long to summarize."
-        try:
-            response = await self._summary_model.ainvoke(
-                prompt,
-                config={"metadata": {"lc_source": "summarization"}},
-            )
-            return response.text.strip()
-        except Exception:
-            logger.exception("Summary generation failed; skipping compaction this turn")
+        prompt = self._prepare_summary_prompt(messages_to_summarize, previous_summary)
+        if prompt is None or prompt in _CANNED_SUMMARIES:
+            return prompt
+        names = self._generation_candidate_names()
+        for index, name in enumerate(names):
+            text = await self._ainvoke_summary(self._model_for(name), prompt, last=index == len(names) - 1)
+            if text is not None:
+                return text
+        return None
+
+    def _invoke_summary(self, model: Any | None, prompt: str, *, last: bool = False) -> str | None:
+        """Invoke ``model`` for a summary; ``None`` on error or a blank response.
+
+        Text extraction / non-empty validation runs *inside* the try: reading the
+        response's ``.text`` is part of consuming the provider result, so a failing
+        accessor must convert to a candidate failure (fall through) rather than escape
+        the fail-open boundary.
+        """
+        if model is None:
             return None
+        try:
+            response = model.invoke(prompt, config={"metadata": {"lc_source": "summarization"}})
+            return self._checked_summary(response, last)
+        except Exception:
+            self._log_summary_error(last)
+            return None
+
+    async def _ainvoke_summary(self, model: Any | None, prompt: str, *, last: bool = False) -> str | None:
+        """Async counterpart of :meth:`_invoke_summary`."""
+        if model is None:
+            return None
+        try:
+            response = await model.ainvoke(prompt, config={"metadata": {"lc_source": "summarization"}})
+            return self._checked_summary(response, last)
+        except Exception:
+            self._log_summary_error(last)
+            return None
+
+    def _checked_summary(self, response: Any, last: bool) -> str | None:
+        summary = self._nonempty_summary(getattr(response, "text", None))
+        if summary is None:
+            self._log_summary_empty(last)
+        return summary
+
+    @staticmethod
+    def _log_summary_error(last: bool) -> None:
+        if last:
+            logger.exception("Summary generation failed; skipping compaction this turn")
+        else:
+            logger.warning("Summary generation failed; falling back to the run model", exc_info=True)
+
+    @staticmethod
+    def _log_summary_empty(last: bool) -> None:
+        if last:
+            logger.warning("Summary model returned empty text; skipping compaction this turn")
+        else:
+            logger.warning("Summary model returned empty text; falling back to the run model")
 
     @staticmethod
     def _summary_count_message(summary_text: str) -> HumanMessage:
@@ -303,15 +486,30 @@ class DeerFlowSummarizationMiddleware(SummarizationMiddleware):
         runtime: Runtime,
         *,
         force: bool = False,
+        raise_on_failure: bool = False,
     ) -> ContextCompactionResult | None:
+        """Summarize old context and retain the active tail.
+
+        ``force`` bypasses the automatic trigger threshold (a manual caller always
+        wants to compact). ``raise_on_failure`` is a *separate* concern: when set (the
+        manual ``/compact`` path), a generation failure raises ``SummaryGenerationError``
+        so it can be reported distinctly from "nothing to compact"; the automatic path
+        leaves it False and swallows the failure, retrying on a later triggered turn.
+        """
         prepared = self._prepare_compaction(state, force=force)
         if prepared is None:
             return None
         messages_to_summarize, preserved_messages, previous_summary, total_tokens = prepared
-        self._fire_hooks(messages_to_summarize, preserved_messages, runtime)
         summary = self._summarize_with(messages_to_summarize, previous_summary=previous_summary)
         if summary is None:
+            if raise_on_failure:
+                raise SummaryGenerationError("summary generation failed")
             return None
+        # Fire hooks only once a replacement summary exists — flushing pre-compaction
+        # messages into durable memory for a summary that never materializes would
+        # duplicate that work on the next attempt. Messages are still removed after
+        # this returns (in _maybe_summarize), so hooks run before they are gone.
+        self._fire_hooks(messages_to_summarize, preserved_messages, runtime)
         return ContextCompactionResult(
             summary_text=summary,
             messages_to_summarize=tuple(messages_to_summarize),
@@ -325,15 +523,20 @@ class DeerFlowSummarizationMiddleware(SummarizationMiddleware):
         runtime: Runtime,
         *,
         force: bool = False,
+        raise_on_failure: bool = False,
     ) -> ContextCompactionResult | None:
+        """Async counterpart of :meth:`compact_state` (see it for ``raise_on_failure``)."""
         prepared = self._prepare_compaction(state, force=force)
         if prepared is None:
             return None
         messages_to_summarize, preserved_messages, previous_summary, total_tokens = prepared
-        self._fire_hooks(messages_to_summarize, preserved_messages, runtime)
         summary = await self._asummarize_with(messages_to_summarize, previous_summary=previous_summary)
         if summary is None:
+            if raise_on_failure:
+                raise SummaryGenerationError("summary generation failed")
             return None
+        # Fire hooks only once a replacement summary exists (see compact_state).
+        self._fire_hooks(messages_to_summarize, preserved_messages, runtime)
         return ContextCompactionResult(
             summary_text=summary,
             messages_to_summarize=tuple(messages_to_summarize),
@@ -444,11 +647,37 @@ class DeerFlowSummarizationMiddleware(SummarizationMiddleware):
                 logger.exception("before_summarization hook %s failed", hook_name)
 
 
+def _build_summary_anchor(candidate_names: list[str | None], app_config: Any) -> tuple[Any | None, str | None]:
+    """Build the first constructible model among ``candidate_names`` (guarded).
+
+    The returned model is tagged for RunJournal attribution but *not* TAG_NOSTREAM (the
+    middleware wraps a nostream copy). It becomes the parent's token-counter / profile
+    anchor and is reused for generation when a candidate matches its name. A per-name
+    construction failure is swallowed and the next candidate tried, so a broken primary
+    constructor neither breaks agent construction nor skips the healthy run model; a
+    trailing ``None`` name asks ``create_chat_model`` for its own default. Returns
+    ``(None, None)`` when nothing can be constructed.
+    """
+    tried: set[str | None] = set()
+    for name in candidate_names:
+        if name in tried:
+            continue
+        tried.add(name)
+        try:
+            model = create_chat_model(name=name, thinking_enabled=False, app_config=app_config, attach_tracing=False)
+        except Exception:
+            logger.exception("Failed to build summary anchor model %r; trying the next candidate", name)
+            continue
+        return model.with_config(tags=["middleware:summarize"]), name
+    return None, None
+
+
 def create_summarization_middleware(
     *,
     app_config: Any | None = None,
     keep: tuple[str, int | float] | None = None,
     skip_memory_flush: bool = False,
+    run_model_name: str | None = None,
 ) -> DeerFlowSummarizationMiddleware | None:
     """Create the configured summarization middleware.
 
@@ -456,6 +685,13 @@ def create_summarization_middleware(
     use this factory so model resolution, hooks, prompt config, and retention
     defaults cannot drift.
 
+    ``run_model_name`` is the model the run actually executes with, resolved by the
+    caller (the lead / subagent / manual builders each already resolve it) and passed
+    in as the authoritative source of truth for ``model_name: null`` summarization and
+    the explicit-summary-model fallback. The middleware does not re-derive it from
+    ``runtime.context`` / ``get_config()``, which do not carry a custom agent's or a
+    subagent's resolved model.
+
     ``skip_memory_flush`` omits the ``memory_flush_hook`` that otherwise
     flushes pre-compaction messages into the durable memory queue. The lead
     chain keeps it (research should persist); the subagent chain sets it so a
@@ -477,23 +713,25 @@ def create_summarization_middleware(
         else:
             trigger = config.trigger.to_tuple()
 
-    if config.model_name:
-        model = create_chat_model(
-            name=config.model_name,
-            thinking_enabled=False,
-            app_config=resolved_app_config,
-            attach_tracing=False,
-        )
-    else:
-        model = create_chat_model(
-            thinking_enabled=False,
-            app_config=resolved_app_config,
-            attach_tracing=False,
-        )
-    model = model.with_config(tags=["middleware:summarize"])
+    default_name = resolved_app_config.models[0].name if getattr(resolved_app_config, "models", None) else None
+    # Build the anchor (token-counter / profile model, reused for generation) guarded,
+    # rather than eagerly building the configured/default model and letting a broken
+    # constructor escape. Candidates in order: the primary generation model (configured
+    # summary model, else the run's own model), then the run model, then the default,
+    # then ``None`` (create_chat_model's default) as a last resort. So the null case
+    # builds from ``run_model_name`` — not ``config.models[0]`` — and a broken primary
+    # falls through to the healthy run model instead of failing agent construction.
+    primary_name = config.model_name or run_model_name or default_name
+    anchor_model, anchor_name = _build_summary_anchor(
+        [primary_name, run_model_name or default_name, default_name, None],
+        resolved_app_config,
+    )
+    if anchor_model is None:
+        logger.warning("Summarization is enabled but no summary model could be constructed; compaction is unavailable for this build")
+        return None
 
     kwargs: dict[str, Any] = {
-        "model": model,
+        "model": anchor_model,
         "trigger": trigger,
         "keep": keep or config.keep.to_tuple(),
     }
@@ -511,4 +749,8 @@ def create_summarization_middleware(
     return DeerFlowSummarizationMiddleware(
         **kwargs,
         before_summarization=hooks,
+        app_config=resolved_app_config,
+        configured_model_name=config.model_name,
+        run_model_name=run_model_name,
+        anchor_model_name=anchor_name,
     )

backend/packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py

@@ -475,6 +475,11 @@ def build_subagent_runtime_middlewares(
     summarization_middleware = create_summarization_middleware(
         app_config=app_config,
         skip_memory_flush=True,
+        # The subagent's resolved model is the source of truth for null-model
+        # summarization: the subagent context/configurable does not carry the child
+        # model (it inherits the parent's), so passing it directly is what makes a
+        # distinct-model subagent summarize with its own model, not the parent's.
+        run_model_name=model_name,
     )
     if summarization_middleware is not None:
         middlewares.append(summarization_middleware)

backend/packages/harness/deerflow/agents/thread_state.py

@@ -1,12 +1,19 @@
 import copy
+import uuid
 from collections.abc import Mapping, Sequence
 from functools import cache
-from typing import Annotated, Any, NotRequired, TypedDict, get_type_hints
+from typing import Annotated, Any, NotRequired, TypedDict, cast, get_type_hints
 
 from langchain.agents import AgentState
-from langchain_core.messages import AnyMessage
+from langchain_core.messages import (
+    AnyMessage,
+    BaseMessageChunk,
+    RemoveMessage,
+    convert_to_messages,
+    message_chunk_to_message,
+)
 from langgraph.channels import DeltaChannel
-from langgraph.graph.message import add_messages
+from langgraph.graph.message import REMOVE_ALL_MESSAGES
 
 import deerflow.checkpoint_patches as _checkpoint_patches  # noqa: F401 - import-time saver fixes
 from deerflow.agents.goal_state import GoalState
@@ -258,11 +265,91 @@ class ThreadState(AgentState):
     summary_text: NotRequired[str | None]
 
 
+def _normalize_messages(value: Any) -> list[AnyMessage]:
+    values = value if isinstance(value, list) else [value]
+    messages = [message_chunk_to_message(cast(BaseMessageChunk, message)) for message in convert_to_messages(values)]
+    for message in messages:
+        if message.id is None:
+            message.id = str(uuid.uuid4())
+    return messages
+
+
+def _index_messages(
+    messages: list[AnyMessage | None],
+) -> tuple[dict[str, int], dict[str, list[int]]]:
+    latest_position: dict[str, int] = {}
+    positions_by_id: dict[str, list[int]] = {}
+    for position, message in enumerate(messages):
+        if message is None:
+            continue
+        message_id = cast(str, message.id)
+        latest_position[message_id] = position
+        positions_by_id.setdefault(message_id, []).append(position)
+    return latest_position, positions_by_id
+
+
+def _raise_null_write(has_messages: bool) -> None:
+    # ``add_messages(left, None)`` reports only ``left`` when the accumulated
+    # message list is non-empty; with an empty list, it reports only ``right``.
+    received = "left" if has_messages else "right"
+    raise ValueError(f"Must specify non-null arguments for both 'left' and 'right'. Only received: '{received}'.")
+
+
 def merge_message_writes(state: list[AnyMessage], writes: Sequence[Any]) -> list[AnyMessage]:
-    result = list(state)
+    """Fold DeltaChannel writes with ``add_messages`` semantics in linear time.
+
+    LangGraph's private ``_messages_delta_reducer`` is also linear, but does
+    not preserve the public reducer's full coercion, ID, removal, and
+    ``REMOVE_ALL_MESSAGES`` behavior.
+    """
+    if not writes:
+        return list(state)
+    if writes[0] is None:
+        _raise_null_write(bool(state))
+
+    messages: list[AnyMessage | None] = _normalize_messages(state)
+    latest_position, positions_by_id = _index_messages(messages)
+
     for write in writes:
-        result = list(add_messages(result, write))
-    return result
+        if write is None:
+            _raise_null_write(bool(latest_position))
+        normalized_write = _normalize_messages(write)
+        remove_all_idx = None
+        for position, message in enumerate(normalized_write):
+            if isinstance(message, RemoveMessage) and message.id == REMOVE_ALL_MESSAGES:
+                remove_all_idx = position
+
+        if remove_all_idx is not None:
+            messages = list(normalized_write[remove_all_idx + 1 :])
+            latest_position, positions_by_id = _index_messages(messages)
+            continue
+
+        ids_to_remove: set[str] = set()
+        for message in normalized_write:
+            message_id = cast(str, message.id)
+            existing_position = latest_position.get(message_id)
+            if existing_position is not None:
+                if isinstance(message, RemoveMessage):
+                    ids_to_remove.add(message_id)
+                else:
+                    ids_to_remove.discard(message_id)
+                    messages[existing_position] = message
+                continue
+
+            if isinstance(message, RemoveMessage):
+                raise ValueError(f"Attempting to delete a message with an ID that doesn't exist ('{message_id}')")
+
+            position = len(messages)
+            messages.append(message)
+            latest_position[message_id] = position
+            positions_by_id[message_id] = [position]
+
+        for message_id in ids_to_remove:
+            for position in positions_by_id.pop(message_id):
+                messages[position] = None
+            del latest_position[message_id]
+
+    return [message for message in messages if message is not None]
 
 
 DELTA_MESSAGES_FIELD = Annotated[

backend/packages/harness/deerflow/checkpoint_patches.py

@@ -1,4 +1,4 @@
-"""Compatibility patches for third-party checkpoint savers.
+"""Compatibility patches for third-party checkpoint machinery.
 
 Lives at the top-level package (not ``deerflow.runtime``) so it can be
 imported from ``deerflow.agents.thread_state`` without pulling in the heavy
@@ -12,10 +12,14 @@ from __future__ import annotations
 
 import importlib.metadata
 import logging
+from collections.abc import Sequence
 from typing import Any
 
+from langgraph.channels.binop import BinaryOperatorAggregate
 from langgraph.checkpoint.base import BaseCheckpointSaver
 from langgraph.checkpoint.memory import InMemorySaver
+from langgraph.errors import ErrorCode, InvalidUpdateError, create_error_message
+from langgraph.types import Overwrite
 from packaging.version import Version
 
 logger = logging.getLogger(__name__)
@@ -91,9 +95,98 @@ def ensure_inmemory_delta_history_patch() -> None:
             return
         InMemorySaver.get_delta_channel_history = _get_delta_channel_history_via_base  # type: ignore[method-assign]
         InMemorySaver.aget_delta_channel_history = _aget_delta_channel_history_via_base  # type: ignore[method-assign]
-        InMemorySaver._deerflow_delta_history_patched = True  # type: ignore[attr-defined]
+        setattr(InMemorySaver, _PATCH_FLAG, True)
     except (AttributeError, TypeError):
         logger.warning("Failed to apply the InMemorySaver delta-history patch; leaving the upstream implementation untouched.", exc_info=True)
 
 
+_BINOP_PATCH_FLAG = "_deerflow_overwrite_first_write_patched"
+_unpatched_binop_update = BinaryOperatorAggregate.update
+
+
+def _as_overwrite(value: Any) -> tuple[bool, Any]:
+    """Local stand-in for langgraph's private ``_get_overwrite``.
+
+    Matches only the public ``Overwrite`` *class* form - the sole form
+    DeerFlow's write paths produce for these Union channels (the branch and
+    ``/state`` routes wrap replace-style writes in ``Overwrite(...)``).
+    Avoiding the underscored ``_get_overwrite`` import keeps an upstream
+    refactor that drops it - plausibly the very release that fixes the bug -
+    from failing this module's import and crashing startup before the probe
+    can stand the patch down. The dict sentinel form upstream also accepts is
+    an internal serialization detail DeerFlow never emits into these channels.
+    """
+    if isinstance(value, Overwrite):
+        return True, value.value
+    return False, None
+
+
+def _binop_first_write_stores_overwrite_wrapper() -> bool:
+    """Probe whether upstream still stores an Overwrite first write literally.
+
+    Uses a Union-typed channel (no constructible default, so it starts
+    MISSING) - the same shape as ``ThreadState``'s ``sandbox`` / ``goal`` /
+    ``todos`` / ``promoted`` channels.
+    """
+    channel = BinaryOperatorAggregate(dict | None, lambda existing, new: new)
+    channel.key = "deerflow-overwrite-probe"
+    channel.update([Overwrite({"probe": True})])
+    return isinstance(channel.get(), Overwrite)
+
+
+def _binop_update_unwrapping_empty_channel(self: Any, values: Sequence[Any]) -> bool:
+    """``BinaryOperatorAggregate.update`` that unwraps an Overwrite first write.
+
+    Only intercepts the empty-channel + leading-Overwrite case; everything
+    else delegates to the upstream implementation. The intercepted case
+    mirrors upstream's own post-Overwrite batch semantics: later plain values
+    are skipped and a second Overwrite raises ``InvalidUpdateError``.
+    """
+    if not self.is_available() and values:
+        is_overwrite, overwrite_value = _as_overwrite(values[0])
+        if is_overwrite:
+            self.value = overwrite_value
+            for value in values[1:]:
+                if _as_overwrite(value)[0]:
+                    msg = create_error_message(
+                        message="Can receive only one Overwrite value per super-step.",
+                        error_code=ErrorCode.INVALID_CONCURRENT_GRAPH_UPDATE,
+                    )
+                    raise InvalidUpdateError(msg)
+            return True
+    return _unpatched_binop_update(self, values)
+
+
+def ensure_binop_overwrite_first_write_patch() -> None:
+    """Fix ``Overwrite`` first writes being stored literally on empty channels.
+
+    Upstream ``BinaryOperatorAggregate.update`` seeds an empty channel
+    (``self.value is MISSING``) with ``values[0]`` verbatim - without the
+    Overwrite unwrapping the rest of the method applies. Channels whose type
+    is a Union (``SandboxState | None``, ``GoalState | None``, ...) have no
+    constructible default, so they start MISSING; a replace-style write into
+    a fresh thread (thread branching) or a never-written channel (state
+    update) then persists the ``Overwrite`` wrapper itself into the
+    checkpoint, and the next consumer crashes with ``TypeError: 'Overwrite'
+    object is not subscriptable`` (#4380). ``DeltaChannel.update`` already
+    unwraps in the same situation, so this also removes a behavioral
+    inconsistency between the two reducer channel types.
+
+    Idempotent. Guarded by a behavioral probe instead of a version pin: if a
+    future LangGraph unwraps the first write itself, the probe reports the
+    bug as absent and the patch stands down.
+    """
+    if getattr(BinaryOperatorAggregate, _BINOP_PATCH_FLAG, False):
+        return
+    try:
+        if not _binop_first_write_stores_overwrite_wrapper():
+            # Upstream unwraps the first write itself: nothing to patch.
+            return
+        BinaryOperatorAggregate.update = _binop_update_unwrapping_empty_channel  # type: ignore[method-assign]
+        setattr(BinaryOperatorAggregate, _BINOP_PATCH_FLAG, True)
+    except Exception:
+        logger.warning("Failed to apply the BinaryOperatorAggregate Overwrite first-write patch; leaving the upstream implementation untouched.", exc_info=True)
+
+
 ensure_inmemory_delta_history_patch()
+ensure_binop_overwrite_first_write_patch()

backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox.py

@@ -5,6 +5,7 @@ import shlex
 import threading
 import uuid
 
+import httpx
 from agent_sandbox import Sandbox as AioSandboxClient
 from agent_sandbox.core.api_error import ApiError
 
@@ -12,6 +13,8 @@ from deerflow.config.paths import VIRTUAL_PATH_PREFIX
 from deerflow.sandbox.sandbox import Sandbox, _validate_extra_env
 from deerflow.sandbox.search import GrepMatch, path_matches, should_ignore_path, truncate_line
 
+from .backend import sandbox_http_trust_env
+
 logger = logging.getLogger(__name__)
 
 _MAX_DOWNLOAD_SIZE = 100 * 1024 * 1024  # 100 MB
@@ -50,7 +53,15 @@ class AioSandbox(Sandbox):
         """
         super().__init__(id)
         self._base_url = base_url
-        self._client = AioSandboxClient(base_url=base_url, timeout=600)
+        if sandbox_http_trust_env(base_url):
+            self._client = AioSandboxClient(base_url=base_url, timeout=600)
+        else:
+            direct_client = httpx.Client(timeout=600, follow_redirects=True, trust_env=False)
+            self._client = AioSandboxClient(
+                base_url=base_url,
+                timeout=600,
+                httpx_client=direct_client,
+            )
         self._home_dir = home_dir
         self._lock = threading.Lock()
         self._closed = False

backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox_provider.py

@@ -39,6 +39,8 @@ from deerflow.community.warm_pool_lifecycle import (
 )
 from deerflow.config import get_app_config
 from deerflow.config.paths import VIRTUAL_PATH_PREFIX, get_paths, join_host_path
+from deerflow.integrations.lark_cli import INTEGRATION_ID as LARK_CLI_INTEGRATION_ID
+from deerflow.integrations.lark_cli import LARK_CLI_SANDBOX_CONFIG_DIR, LARK_CLI_SANDBOX_DATA_DIR, LARK_CLI_SANDBOX_RUNTIME_DIR, ensure_lark_cli_credential_tree, lark_skills_installed
 from deerflow.runtime.user_context import get_effective_user_id
 from deerflow.sandbox.sandbox import Sandbox
 from deerflow.sandbox.sandbox_provider import SandboxProvider
@@ -85,6 +87,19 @@ class SandboxBeingDestroyedError(RuntimeError):
         self.sandbox_id = sandbox_id
 
 
+class SandboxIdentityCollisionError(RuntimeError):
+    """A deterministic ID is already tracked for a different user/thread."""
+
+    def __init__(
+        self,
+        sandbox_id: str,
+        stored_key: tuple[str, str] | None,
+        requested_key: tuple[str, str],
+    ) -> None:
+        super().__init__(f"sandbox ID collision for {sandbox_id}: tracked identity is {stored_key!r}, requested identity is {requested_key!r}")
+        self.sandbox_id = sandbox_id
+
+
 def _lock_file_exclusive(lock_file) -> None:
     if fcntl is not None:
         fcntl.flock(lock_file, fcntl.LOCK_EX)
@@ -181,6 +196,8 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
         # Containers here can be reclaimed quickly (no cold-start) or destroyed
         # when replicas capacity is exhausted.
         self._warm_pool: dict[str, tuple[SandboxInfo, float]] = {}
+        self._active_sandbox_identity: dict[str, tuple[str, str] | None] = {}
+        self._warm_pool_identity: dict[str, tuple[str, str] | None] = {}
         # sandbox_id -> when reconciliation first saw it running with no lease.
         # Gates adoption behind a recovery grace (see _adoptable_after_grace).
         self._unowned_since: dict[str, float] = {}
@@ -698,6 +715,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
                     logger.debug("Deferring container %s during reconciliation: this instance is tearing it down", info.sandbox_id)
                     continue
                 self._warm_pool[info.sandbox_id] = (info, current_time)
+                self._warm_pool_identity[info.sandbox_id] = None
             self._unowned_since.pop(info.sandbox_id, None)
             adopted += 1
             logger.info(f"Adopted container {info.sandbox_id} into warm pool (age: {age:.0f}s)")
@@ -726,8 +744,42 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
 
         Includes user_id so a previously-created default-bucket sandbox cannot be
         reused for an auth/channel run that should mount a user-scoped bucket.
+
+        During a mixed-version rollout, older 8-character containers are not
+        reused under the new 16-character identity. They remain eligible for
+        normal orphan cleanup while the first new-version acquire cold-starts.
         """
-        return hashlib.sha256(f"{user_id}:{thread_id}".encode()).hexdigest()[:8]
+        return hashlib.sha256(f"{user_id}:{thread_id}".encode()).hexdigest()[:16]
+
+    def _assert_active_identity_available_locked(
+        self,
+        sandbox_id: str,
+        requested_key: tuple[str, str],
+    ) -> None:
+        """Fail closed if an active truncated ID belongs to another identity."""
+        if sandbox_id not in self._sandboxes and sandbox_id not in self._sandbox_infos:
+            return
+
+        stored_key = self._active_sandbox_identity.get(sandbox_id)
+        if stored_key is None:
+            matching_keys = [key for key, mapped_id in self._thread_sandboxes.items() if mapped_id == sandbox_id]
+            if len(matching_keys) == 1:
+                stored_key = matching_keys[0]
+        if stored_key != requested_key:
+            raise SandboxIdentityCollisionError(sandbox_id, stored_key, requested_key)
+
+    def _assert_warm_identity_available_locked(
+        self,
+        sandbox_id: str,
+        requested_key: tuple[str, str],
+    ) -> None:
+        """Fail closed if a warm ID changed tenants during an acquire."""
+        if sandbox_id not in self._warm_pool:
+            return
+        # Startup-adopted entries have unknown identity until their first reclaim.
+        stored_key = self._warm_pool_identity.get(sandbox_id)
+        if stored_key is not None and stored_key != requested_key:
+            raise SandboxIdentityCollisionError(sandbox_id, stored_key, requested_key)
 
     # ── Mount helpers ────────────────────────────────────────────────────
 
@@ -744,7 +796,40 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
             mounts.extend(skills_mounts)
             logger.info(f"Adding skills mounts: {skills_mounts}")
 
-        return mounts
+        user_skill_mounts = self._get_user_skill_mounts(user_id=user_id)
+        if user_skill_mounts:
+            mounts.extend(user_skill_mounts)
+            logger.info(f"Adding user skill mounts: {user_skill_mounts}")
+
+        lark_cli_mounts = self._get_lark_cli_runtime_mounts(user_id=user_id)
+        if lark_cli_mounts:
+            mounts.extend(lark_cli_mounts)
+            logger.info(f"Adding Lark CLI runtime mounts: {lark_cli_mounts}")
+
+        return self._dedupe_mounts_by_container_path(mounts)
+
+    @staticmethod
+    def _dedupe_mounts_by_container_path(mounts: list[tuple[str, str, bool]]) -> list[tuple[str, str, bool]]:
+        """Keep the first mount for each container path.
+
+        Duplicate container paths are rejected by the provisioner and can also
+        fail local Docker creation. The earlier mount wins because mount helpers
+        are appended in priority order: thread data, skill roots, integration
+        skill roots, then integration runtimes/credentials.
+        """
+        seen: set[str] = set()
+        deduped: list[tuple[str, str, bool]] = []
+        for host_path, container_path, read_only in mounts:
+            if container_path in seen:
+                logger.warning(
+                    "Skipping duplicate sandbox mount for container path %s from host %s",
+                    container_path,
+                    host_path,
+                )
+                continue
+            seen.add(container_path)
+            deduped.append((host_path, container_path, read_only))
+        return deduped
 
     @staticmethod
     def _get_thread_mounts(thread_id: str, *, user_id: str | None = None) -> list[tuple[str, str, bool]]:
@@ -839,6 +924,84 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
 
         return mounts
 
+    @staticmethod
+    def _get_user_skill_mounts(*, user_id: str | None = None) -> list[tuple[str, str, bool]]:
+        """Mount managed integration skills into AIO sandboxes.
+
+        Per-user custom skills are already mounted by ``_get_skills_mounts``.
+        This helper adds the shared integration skill root so sandbox paths match
+        the skill registry without duplicating ``/mnt/skills/custom``.
+        """
+        try:
+            config = get_app_config()
+            paths = get_paths()
+            skills_container_path = config.skills.container_path
+            paths.integration_skills_dir().mkdir(parents=True, exist_ok=True)
+            return [
+                (paths.host_integration_skills_dir(), f"{skills_container_path}/integrations", True),
+            ]
+        except Exception as e:
+            logger.warning(f"Could not setup user skill mounts: {e}")
+            return []
+
+    @staticmethod
+    def _lark_integration_active(user_id: str | None = None) -> bool:
+        """Whether the managed Lark skill pack is installed for this user.
+
+        Drives whether a sandbox requests the lark-cli runtime (init container /
+        Gateway-download mount). Independent of whether a local ``sandbox-cli``
+        dir exists, so remote/K8s can opt in without a Gateway-side download.
+        """
+        try:
+            effective_user_id = AioSandboxProvider._effective_acquire_user_id(user_id)
+            return lark_skills_installed(effective_user_id)
+        except Exception as e:  # pragma: no cover - defensive
+            logger.warning(f"Could not determine Lark integration state: {e}")
+            return False
+
+    @staticmethod
+    def _get_lark_cli_runtime_mounts(*, user_id: str | None = None) -> list[tuple[str, str, bool]]:
+        """Mount the per-user lark-cli config/data dirs used by Settings auth.
+
+        Settings endpoints run ``lark-cli`` on the Gateway with
+        ``LARKSUITE_CLI_CONFIG_DIR`` / ``DATA_DIR`` pointing at
+        ``users/{user}/integrations/lark-cli``. Agent conversations run
+        ``lark-cli`` inside the sandbox, so those same directories must be
+        mounted into the container or the CLI sees a separate unauthenticated
+        profile.
+
+        The ``config`` dir holds the long-lived Lark ``appSecret`` (written by
+        ``lark-cli config init`` on the Gateway, never in-sandbox), so it is
+        mounted **read-only**: sandbox processes only need to read it, and a
+        read-only bind stops a compromised agent from tampering with or
+        replacing the app credentials. The ``data`` dir holds refreshable OAuth
+        tokens that ``lark-cli auth`` updates in-sandbox, so it stays writable.
+        This is defense-in-depth only — both dirs remain readable to arbitrary
+        sandbox processes until the auth-proxy follow-up (issue #4338) lands.
+        See the sandbox trust-boundary note in ``backend/AGENTS.md``.
+        """
+        try:
+            paths = get_paths()
+            effective_user_id = AioSandboxProvider._effective_acquire_user_id(user_id)
+            ensure_lark_cli_credential_tree(effective_user_id, paths=paths)
+            mounts = [
+                (paths.host_user_integration_config_dir(effective_user_id, LARK_CLI_INTEGRATION_ID), LARK_CLI_SANDBOX_CONFIG_DIR, True),
+                (paths.host_user_integration_data_dir(effective_user_id, LARK_CLI_INTEGRATION_ID), LARK_CLI_SANDBOX_DATA_DIR, False),
+            ]
+            runtime_dir = paths.base_dir / "integrations" / LARK_CLI_INTEGRATION_ID / "sandbox-cli"
+            if runtime_dir.is_dir():
+                mounts.append(
+                    (
+                        join_host_path(str(paths.host_base_dir), "integrations", LARK_CLI_INTEGRATION_ID, "sandbox-cli"),
+                        LARK_CLI_SANDBOX_RUNTIME_DIR,
+                        True,
+                    )
+                )
+            return mounts
+        except Exception as e:
+            logger.warning(f"Could not setup Lark CLI runtime mounts: {e}")
+            return []
+
     # ── Idle timeout management ──────────────────────────────────────────
 
     def _cleanup_idle_resources(self, idle_timeout: float) -> None:
@@ -956,8 +1119,10 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
 
             sandbox = self._sandboxes.pop(sandbox_id, None)
             self._sandbox_infos.pop(sandbox_id, None)
+            self._active_sandbox_identity.pop(sandbox_id, None)
             self._last_activity.pop(sandbox_id, None)
             self._warm_pool.pop(sandbox_id, None)
+            self._warm_pool_identity.pop(sandbox_id, None)
             self._acquire_epoch.pop(sandbox_id, None)
             for key, mapped_id in list(self._thread_sandboxes.items()):
                 if mapped_id == sandbox_id:
@@ -1197,6 +1362,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
         with self._lock:
             if sandbox_id not in self._warm_pool:
                 return None
+            self._assert_warm_identity_available_locked(sandbox_id, key)
             if self._being_torn_down_locally(sandbox_id):
                 # The entry deliberately stays in `_warm_pool` for the whole stop
                 # (so a refused claim does not lose the container), so pool
@@ -1237,13 +1403,16 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
                 # that is already stopped.
                 logger.info("Warm-pool sandbox %s was claimed for teardown while publishing ownership; not reclaiming it", sandbox_id)
                 return None
+            self._assert_warm_identity_available_locked(sandbox_id, key)
             warm_item = self._warm_pool.pop(sandbox_id, None)
             if warm_item is None:
                 return None
+            self._warm_pool_identity.pop(sandbox_id, None)
             info, _ = warm_item
             sandbox = AioSandbox(id=sandbox_id, base_url=info.sandbox_url)
             self._sandboxes[sandbox_id] = sandbox
             self._sandbox_infos[sandbox_id] = info
+            self._active_sandbox_identity[sandbox_id] = key
             self._last_activity[sandbox_id] = time.time()
             self._thread_sandboxes[key] = sandbox_id
 
@@ -1272,6 +1441,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
                 prevent. The window is a peer's in-flight container stop, so the
                 thread's next turn discovers nothing and cold-starts cleanly.
         """
+        key = self._thread_key(thread_id, user_id)
         with self._lock:
             if self._being_torn_down_locally(info.sandbox_id):
                 # Discovery is the fall-through once the caches miss, so it is
@@ -1279,9 +1449,10 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
                 # only refuse this once the reaper's `del:` claim has landed;
                 # until then it succeeds against our own lease.
                 raise SandboxBeingDestroyedError(info.sandbox_id)
+            self._assert_active_identity_available_locked(info.sandbox_id, key)
+            self._assert_warm_identity_available_locked(info.sandbox_id, key)
 
         sandbox = AioSandbox(id=info.sandbox_id, base_url=info.sandbox_url)
-        key = self._thread_key(thread_id, user_id)
         # Ownership first, so a failure cannot leave a tracked-but-unowned sandbox.
         # There is no container to roll back (we did not create it), but the
         # host-side HTTP client constructed above is ours and must not leak —
@@ -1295,6 +1466,8 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
                     # trip. Do not install a client for a container that reaper
                     # has already committed to stopping.
                     raise SandboxBeingDestroyedError(info.sandbox_id)
+                self._assert_active_identity_available_locked(info.sandbox_id, key)
+                self._assert_warm_identity_available_locked(info.sandbox_id, key)
                 # Active and warm are exclusive states, and only this insert can
                 # violate that: a warm entry for the same id is stale the moment
                 # the id becomes active. Leaving it there gives the container two
@@ -1303,11 +1476,17 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
                 # agent is actively using while `_sandboxes` still hands out its
                 # client.
                 self._warm_pool.pop(info.sandbox_id, None)
+                self._warm_pool_identity.pop(info.sandbox_id, None)
                 self._sandboxes[info.sandbox_id] = sandbox
                 self._sandbox_infos[info.sandbox_id] = info
+                self._active_sandbox_identity[info.sandbox_id] = key
                 self._last_activity[info.sandbox_id] = time.time()
                 self._thread_sandboxes[key] = info.sandbox_id
-        except (OwnershipBackendError, SandboxBeingDestroyedError):
+        except (
+            OwnershipBackendError,
+            SandboxBeingDestroyedError,
+            SandboxIdentityCollisionError,
+        ):
             try:
                 sandbox.close()
             except Exception as e:
@@ -1320,6 +1499,14 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
     def _register_created_sandbox(self, thread_id: str | None, sandbox_id: str, info: SandboxInfo, *, user_id: str | None = None) -> str:
         """Track a newly-created sandbox in the active maps."""
         sandbox = AioSandbox(id=sandbox_id, base_url=info.sandbox_url)
+        key = (
+            self._thread_key(
+                thread_id,
+                self._effective_acquire_user_id(user_id),
+            )
+            if thread_id
+            else None
+        )
         # Ownership first. Unlike the discover path there IS something to roll
         # back: we just started this container, and an unowned running container
         # is exactly what a peer's reconciliation adopts. Leaking it would hand a
@@ -1328,9 +1515,34 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
         # mid-stop leaves a teardown marker until its TTL lapses. Roll back on
         # both, or the container we just started is leaked.
         try:
+            if key is not None:
+                with self._lock:
+                    self._assert_active_identity_available_locked(sandbox_id, key)
+                    self._assert_warm_identity_available_locked(sandbox_id, key)
             self._publish_ownership(sandbox_id)
-        except (OwnershipBackendError, SandboxBeingDestroyedError):
-            logger.error("Could not publish ownership for new sandbox %s; destroying it rather than leaking an unowned container", sandbox_id)
+
+            with self._lock:
+                if key is not None:
+                    self._assert_active_identity_available_locked(sandbox_id, key)
+                    self._assert_warm_identity_available_locked(sandbox_id, key)
+                # Same exclusivity rule as the discover path.
+                self._warm_pool.pop(sandbox_id, None)
+                self._warm_pool_identity.pop(sandbox_id, None)
+                self._sandboxes[sandbox_id] = sandbox
+                self._sandbox_infos[sandbox_id] = info
+                self._active_sandbox_identity[sandbox_id] = key
+                self._last_activity[sandbox_id] = time.time()
+                if key is not None:
+                    self._thread_sandboxes[key] = sandbox_id
+        except (
+            OwnershipBackendError,
+            SandboxBeingDestroyedError,
+            SandboxIdentityCollisionError,
+        ):
+            logger.error(
+                "Could not register new sandbox %s; destroying it rather than leaking an untracked container",
+                sandbox_id,
+            )
             try:
                 sandbox.close()
             except Exception as e:
@@ -1338,18 +1550,13 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
             try:
                 self._backend.destroy(info)
             except Exception as e:
-                logger.error("Failed to destroy unowned sandbox %s after ownership failure: %s", sandbox_id, e)
+                logger.error(
+                    "Failed to destroy sandbox %s after registration failure: %s",
+                    sandbox_id,
+                    e,
+                )
             raise
 
-        with self._lock:
-            # Same exclusivity rule as the discover path.
-            self._warm_pool.pop(sandbox_id, None)
-            self._sandboxes[sandbox_id] = sandbox
-            self._sandbox_infos[sandbox_id] = info
-            self._last_activity[sandbox_id] = time.time()
-            if thread_id:
-                self._thread_sandboxes[self._thread_key(thread_id, self._effective_acquire_user_id(user_id))] = sandbox_id
-
         logger.info(f"Created sandbox {sandbox_id} for thread {thread_id} at {info.sandbox_url}")
         return sandbox_id
 
@@ -1385,6 +1592,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
 
             sandbox = self._sandboxes.pop(sandbox_id, None)
             info = self._sandbox_infos.pop(sandbox_id, None)
+            self._active_sandbox_identity.pop(sandbox_id, None)
             thread_keys_to_remove = [key for key, sid in self._thread_sandboxes.items() if sid == sandbox_id]
             for key in thread_keys_to_remove:
                 del self._thread_sandboxes[key]
@@ -1394,6 +1602,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
                 info, _ = self._warm_pool.pop(sandbox_id)
             else:
                 self._warm_pool.pop(sandbox_id, None)
+            self._warm_pool_identity.pop(sandbox_id, None)
 
         return sandbox, info, True
 
@@ -1510,7 +1719,10 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
             # The pop stays deferred relative to the *stop* (a refused or failed
             # stop keeps the entry), just no longer relative to the reservation.
             with self._lock:
-                self._warm_pool.pop(sandbox_id, None)
+                current = self._warm_pool.get(sandbox_id)
+                if current is not None and current[0] is entry:
+                    self._warm_pool.pop(sandbox_id, None)
+                    self._warm_pool_identity.pop(sandbox_id, None)
         finally:
             self._finish_local_teardown(sandbox_id)
 
@@ -1579,6 +1791,10 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
 
         # Deterministic ID for thread-specific, random for anonymous
         sandbox_id = self._sandbox_id_for_thread(thread_id, user_id)
+        if thread_id:
+            key = self._thread_key(thread_id, user_id)
+            with self._lock:
+                self._assert_active_identity_available_locked(sandbox_id, key)
 
         # ── Layer 1.5: Warm pool (container still running, no cold-start) ──
         reclaimed_id = self._reclaim_warm_pool_sandbox(thread_id, sandbox_id, user_id=user_id)
@@ -1602,6 +1818,10 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
 
         # Deterministic ID for thread-specific, random for anonymous
         sandbox_id = self._sandbox_id_for_thread(thread_id, user_id)
+        if thread_id:
+            key = self._thread_key(thread_id, user_id)
+            with self._lock:
+                self._assert_active_identity_available_locked(sandbox_id, key)
 
         # ── Layer 1.5: Warm pool (container still running, no cold-start) ──
         reclaimed_id = await asyncio.to_thread(self._reclaim_warm_pool_sandbox, thread_id, sandbox_id, user_id=user_id)
@@ -1694,6 +1914,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
         """
         effective_user_id = self._effective_acquire_user_id(user_id)
         extra_mounts = self._get_extra_mounts(thread_id, user_id=effective_user_id)
+        provision_lark_cli_runtime = self._lark_integration_active(effective_user_id)
 
         # Enforce replicas: only warm-pool containers count toward eviction budget.
         # Active sandboxes are in use by live threads and must not be forcibly stopped.
@@ -1702,7 +1923,13 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
             evicted = self._evict_oldest_warm()
             self._log_replicas_soft_cap(replicas, sandbox_id, evicted)
 
-        info = self._backend.create(thread_id, sandbox_id, extra_mounts=extra_mounts or None, user_id=effective_user_id)
+        info = self._backend.create(
+            thread_id,
+            sandbox_id,
+            extra_mounts=extra_mounts or None,
+            user_id=effective_user_id,
+            provision_lark_cli_runtime=provision_lark_cli_runtime,
+        )
 
         # Wait for sandbox to be ready
         if not wait_for_sandbox_ready(info.sandbox_url, timeout=60):
@@ -1715,6 +1942,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
         """Async counterpart to ``_create_sandbox``."""
         effective_user_id = self._effective_acquire_user_id(user_id)
         extra_mounts = await asyncio.to_thread(self._get_extra_mounts, thread_id, user_id=effective_user_id)
+        provision_lark_cli_runtime = await asyncio.to_thread(self._lark_integration_active, effective_user_id)
 
         # Enforce replicas: only warm-pool containers count toward eviction budget.
         # Active sandboxes are in use by live threads and must not be forcibly stopped.
@@ -1723,7 +1951,14 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
             evicted = await asyncio.to_thread(self._evict_oldest_warm)
             self._log_replicas_soft_cap(replicas, sandbox_id, evicted)
 
-        info = await asyncio.to_thread(self._backend.create, thread_id, sandbox_id, extra_mounts=extra_mounts or None, user_id=effective_user_id)
+        info = await asyncio.to_thread(
+            self._backend.create,
+            thread_id,
+            sandbox_id,
+            extra_mounts=extra_mounts or None,
+            user_id=effective_user_id,
+            provision_lark_cli_runtime=provision_lark_cli_runtime,
+        )
 
         # Wait for sandbox to be ready without blocking the event loop.
         if not await wait_for_sandbox_ready_async(info.sandbox_url, timeout=60):
@@ -1781,10 +2016,12 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
             thread_keys_to_remove = [key for key, sid in self._thread_sandboxes.items() if sid == sandbox_id]
             for key in thread_keys_to_remove:
                 del self._thread_sandboxes[key]
+            active_identity = self._active_sandbox_identity.pop(sandbox_id, None)
             self._last_activity.pop(sandbox_id, None)
             # Park in warm pool — container keeps running
             if info and sandbox_id not in self._warm_pool:
                 self._warm_pool[sandbox_id] = (info, time.time())
+                self._warm_pool_identity[sandbox_id] = thread_keys_to_remove[0] if thread_keys_to_remove else active_identity
 
         if sandbox is not None:
             # Defense-in-depth: close() already swallows its own errors; this
@@ -1887,6 +2124,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider):
             sandbox_ids = list(self._sandboxes.keys())
             warm_items = list(self._warm_pool.items())
             self._warm_pool.clear()
+            self._warm_pool_identity.clear()
 
         self._stop_idle_checker()
         # Stop renewing before destroying: the destroy paths claim ownership

backend/packages/harness/deerflow/community/aio_sandbox/backend.py

@@ -3,9 +3,11 @@
 from __future__ import annotations
 
 import asyncio
+import ipaddress
 import logging
 import time
 from abc import ABC, abstractmethod
+from urllib.parse import urlparse
 
 import httpx
 import requests
@@ -15,6 +17,30 @@ from .sandbox_info import SandboxInfo
 logger = logging.getLogger(__name__)
 
 
+def sandbox_http_trust_env(sandbox_url: str) -> bool:
+    """Whether HTTP clients for *sandbox_url* should inherit proxy settings.
+
+    Local Docker, DooD, and Kubernetes sandbox endpoints are control-plane
+    connections, not internet traffic. Sending them through ``HTTP_PROXY`` can
+    produce a misleading proxy-generated 502 even though the sandbox container
+    is healthy (#3441). External fully-qualified hosts retain normal environment
+    proxy behavior.
+    """
+    try:
+        hostname = (urlparse(sandbox_url).hostname or "").rstrip(".").lower()
+    except ValueError:
+        return True
+    if not hostname:
+        return True
+    if hostname == "localhost" or hostname.endswith(".localhost") or hostname.endswith(".docker.internal") or hostname.endswith(".containers.internal"):
+        return False
+    try:
+        address = ipaddress.ip_address(hostname)
+    except ValueError:
+        return "." in hostname
+    return not (address.is_loopback or address.is_private or address.is_link_local)
+
+
 def wait_for_sandbox_ready(sandbox_url: str, timeout: int = 30) -> bool:
     """Poll sandbox health endpoint until ready or timeout.
 
@@ -26,14 +52,16 @@ def wait_for_sandbox_ready(sandbox_url: str, timeout: int = 30) -> bool:
         True if sandbox is ready, False otherwise.
     """
     start_time = time.time()
-    while time.time() - start_time < timeout:
-        try:
-            response = requests.get(f"{sandbox_url}/v1/sandbox", timeout=5)
-            if response.status_code == 200:
-                return True
-        except requests.exceptions.RequestException:
-            pass
-        time.sleep(1)
+    with requests.Session() as session:
+        session.trust_env = sandbox_http_trust_env(sandbox_url)
+        while time.time() - start_time < timeout:
+            try:
+                response = session.get(f"{sandbox_url}/v1/sandbox", timeout=5)
+                if response.status_code == 200:
+                    return True
+            except requests.exceptions.RequestException:
+                pass
+            time.sleep(1)
     return False
 
 
@@ -47,7 +75,7 @@ async def wait_for_sandbox_ready_async(sandbox_url: str, timeout: int = 30, poll
     loop = asyncio.get_running_loop()
     deadline = loop.time() + timeout
 
-    async with httpx.AsyncClient(timeout=5) as client:
+    async with httpx.AsyncClient(timeout=5, trust_env=sandbox_http_trust_env(sandbox_url)) as client:
         while True:
             remaining = deadline - loop.time()
             if remaining <= 0:
@@ -81,6 +109,7 @@ class SandboxBackend(ABC):
         extra_mounts: list[tuple[str, str, bool]] | None = None,
         *,
         user_id: str | None = None,
+        provision_lark_cli_runtime: bool = False,
     ) -> SandboxInfo:
         """Create/provision a new sandbox.
 
@@ -90,6 +119,9 @@ class SandboxBackend(ABC):
             extra_mounts: Additional volume mounts as (host_path, container_path, read_only) tuples.
                 Ignored by backends that don't manage containers (e.g., remote).
             user_id: User bucket that the sandbox should mount or provision for.
+            provision_lark_cli_runtime: Ask the backend to provision the sandbox
+                lark-cli runtime via its native mechanism (e.g. the provisioner's
+                init container + emptyDir). Backends that can't do this ignore it.
 
         Returns:
             SandboxInfo with connection details.

backend/packages/harness/deerflow/community/aio_sandbox/local_backend.py

@@ -271,6 +271,7 @@ class LocalContainerBackend(SandboxBackend):
         extra_mounts: list[tuple[str, str, bool]] | None = None,
         *,
         user_id: str | None = None,
+        provision_lark_cli_runtime: bool = False,
     ) -> SandboxInfo:
         """Start a new container and return its connection info.
 
@@ -280,6 +281,8 @@ class LocalContainerBackend(SandboxBackend):
             extra_mounts: Additional volume mounts as (host_path, container_path, read_only) tuples.
             user_id: User bucket already reflected in extra_mounts. Accepted for
                 interface compatibility with remote backends.
+            provision_lark_cli_runtime: Ignored — the local backend provisions the
+                lark-cli runtime via the Gateway-download bind mount in extra_mounts.
 
         Returns:
             SandboxInfo with container details.
@@ -287,7 +290,7 @@ class LocalContainerBackend(SandboxBackend):
         Raises:
             RuntimeError: If the container fails to start.
         """
-        del user_id
+        del user_id, provision_lark_cli_runtime
         container_name = f"{self._container_prefix}-{sandbox_id}"
 
         # Retry loop: if Docker rejects the port (e.g. a stale container still

backend/packages/harness/deerflow/community/aio_sandbox/remote_backend.py

@@ -29,6 +29,48 @@ from .sandbox_info import SandboxInfo
 
 logger = logging.getLogger(__name__)
 
+_PROVISIONER_EXTRA_MOUNT_PATHS = {
+    "/mnt/acp-workspace",
+    "/mnt/skills/custom",
+    "/mnt/skills/integrations",
+    "/mnt/integrations/lark-cli/config",
+    "/mnt/integrations/lark-cli/data",
+    "/mnt/integrations/lark-cli/runtime",
+}
+
+_LARK_CLI_RUNTIME_CONTAINER_PATH = "/mnt/integrations/lark-cli/runtime"
+
+
+def _provisioner_extra_mounts_payload(
+    extra_mounts: list[tuple[str, str, bool]] | None,
+    *,
+    provision_lark_cli_runtime: bool = False,
+) -> list[dict[str, object]]:
+    """Return only extra mounts the provisioner knows how to recreate safely.
+
+    When ``provision_lark_cli_runtime`` is set, the provisioner supplies the
+    lark-cli runtime via an init container + emptyDir, so the runtime extra mount
+    is dropped here to avoid a colliding hostPath/PVC mount at the same path. The
+    per-user config/data credential mounts are always forwarded.
+    """
+    if not extra_mounts:
+        return []
+
+    payload: list[dict[str, object]] = []
+    for host_path, container_path, read_only in extra_mounts:
+        if container_path not in _PROVISIONER_EXTRA_MOUNT_PATHS:
+            continue
+        if provision_lark_cli_runtime and container_path == _LARK_CLI_RUNTIME_CONTAINER_PATH:
+            continue
+        payload.append(
+            {
+                "host_path": host_path,
+                "container_path": container_path,
+                "read_only": read_only,
+            }
+        )
+    return payload
+
 
 class RemoteSandboxBackend(SandboxBackend):
     """Backend that delegates sandbox lifecycle to the provisioner service.
@@ -72,13 +114,20 @@ class RemoteSandboxBackend(SandboxBackend):
         extra_mounts: list[tuple[str, str, bool]] | None = None,
         *,
         user_id: str | None = None,
+        provision_lark_cli_runtime: bool = False,
     ) -> SandboxInfo:
         """Create a sandbox Pod + Service via the provisioner.
 
         Calls ``POST /api/sandboxes`` which creates a dedicated Pod +
         NodePort Service in k3s.
         """
-        return self._provisioner_create(thread_id, sandbox_id, extra_mounts, user_id=user_id)
+        return self._provisioner_create(
+            thread_id,
+            sandbox_id,
+            extra_mounts,
+            user_id=user_id,
+            provision_lark_cli_runtime=provision_lark_cli_runtime,
+        )
 
     def destroy(self, info: SandboxInfo) -> None:
         """Destroy a sandbox Pod + Service via the provisioner."""
@@ -149,20 +198,28 @@ class RemoteSandboxBackend(SandboxBackend):
         extra_mounts: list[tuple[str, str, bool]] | None = None,
         *,
         user_id: str | None = None,
+        provision_lark_cli_runtime: bool = False,
     ) -> SandboxInfo:
         """POST /api/sandboxes → create Pod + Service."""
-        del extra_mounts
         effective_user_id = user_id or get_effective_user_id()
         include_legacy_skills = user_should_see_legacy_skills(effective_user_id)
+        payload = {
+            "sandbox_id": sandbox_id,
+            "thread_id": thread_id,
+            "user_id": effective_user_id,
+            "include_legacy_skills": include_legacy_skills,
+            "provision_lark_cli_runtime": provision_lark_cli_runtime,
+        }
+        provisioner_extra_mounts = _provisioner_extra_mounts_payload(
+            extra_mounts,
+            provision_lark_cli_runtime=provision_lark_cli_runtime,
+        )
+        if provisioner_extra_mounts:
+            payload["extra_mounts"] = provisioner_extra_mounts
         try:
             resp = requests.post(
                 f"{self._provisioner_url}/api/sandboxes",
-                json={
-                    "sandbox_id": sandbox_id,
-                    "thread_id": thread_id,
-                    "user_id": effective_user_id,
-                    "include_legacy_skills": include_legacy_skills,
-                },
+                json=payload,
                 headers=self._auth_headers(),
                 timeout=30,
             )

backend/packages/harness/deerflow/community/boxlite/provider.py

@@ -41,6 +41,7 @@ T = TypeVar("T")
 
 DEFAULT_IMAGE = "python:3.12-slim"
 _BOX_NAME_PREFIX = "deer-flow-boxlite-"
+_NO_ACTIVE_IDENTITY = object()
 # DeerFlow's virtual prefixes, materialised on the box rootfs at start so the
 # Sandbox file APIs (which address /mnt/user-data/...) resolve natively.
 _VIRTUAL_DIRS = (
@@ -51,6 +52,18 @@ _VIRTUAL_DIRS = (
 )
 
 
+class SandboxIdentityCollisionError(RuntimeError):
+    """A deterministic ID is already tracked for a different user/thread."""
+
+    def __init__(
+        self,
+        sandbox_id: str,
+        stored_key: tuple[str, str] | None,
+        requested_key: tuple[str, str],
+    ) -> None:
+        super().__init__(f"Sandbox ID collision for {sandbox_id}: active box belongs to {stored_key!r}, not {requested_key!r}")
+
+
 def _import_simplebox() -> type[SimpleBox]:
     """Import BoxLite's async ``SimpleBox`` lazily.
 
@@ -166,8 +179,13 @@ class BoxliteProvider(WarmPoolLifecycleMixin[BoxliteBox], SandboxProvider):
 
         Includes user_id so a box created for one user's bucket cannot be
         reclaimed by another user's thread with the same thread_id.
+
+        The 8-to-16-character change intentionally causes one cold start during
+        a mixed-version rollout. Older 8-character boxes remain in the warm pool
+        until the normal orphan cleanup removes them; they are never reused
+        under a new 16-character identity.
         """
-        return hashlib.sha256(f"{user_id}:{thread_id}".encode()).hexdigest()[:8]
+        return hashlib.sha256(f"{user_id}:{thread_id}".encode()).hexdigest()[:16]
 
     # ── Provider ────────────────────────────────────────────────────────
 
@@ -176,6 +194,8 @@ class BoxliteProvider(WarmPoolLifecycleMixin[BoxliteBox], SandboxProvider):
         self._boxes: dict[str, BoxliteBox] = {}
         self._thread_boxes: dict[tuple[str, str], str] = {}
         self._warm_pool: dict[str, tuple[BoxliteBox, float]] = {}
+        self._active_box_identity: dict[str, tuple[str, str] | None] = {}
+        self._warm_pool_identity: dict[str, tuple[str, str] | None] = {}
         self._skip_health_check_warm_ids: set[str] = set()
         self._acquire_locks: dict[str, threading.Lock] = {}
         self._idle_checker_stop = threading.Event()
@@ -245,7 +265,9 @@ class BoxliteProvider(WarmPoolLifecycleMixin[BoxliteBox], SandboxProvider):
     def _destroy_warm_entry(self, sandbox_id: str, entry: BoxliteBox, *, reason: str) -> None:
         """Close a removed warm-pool entry and log with context."""
         with self._lock:
-            self._skip_health_check_warm_ids.discard(sandbox_id)
+            if sandbox_id not in self._warm_pool:
+                self._skip_health_check_warm_ids.discard(sandbox_id)
+                self._warm_pool_identity.pop(sandbox_id, None)
         try:
             entry.close()
             if reason == "idle_timeout":
@@ -268,6 +290,8 @@ class BoxliteProvider(WarmPoolLifecycleMixin[BoxliteBox], SandboxProvider):
         with self._lock:
             active_box = self._boxes.pop(sandbox_id, None)
             warm_entry = self._warm_pool.pop(sandbox_id, None)
+            self._active_box_identity.pop(sandbox_id, None)
+            self._warm_pool_identity.pop(sandbox_id, None)
             self._skip_health_check_warm_ids.discard(sandbox_id)
             for key in [k for k, sid in self._thread_boxes.items() if sid == sandbox_id]:
                 self._thread_boxes.pop(key, None)
@@ -335,6 +359,7 @@ class BoxliteProvider(WarmPoolLifecycleMixin[BoxliteBox], SandboxProvider):
                     box_runtime.stop()
                     continue
                 self._warm_pool[sandbox_id] = (wrapped, now)
+                self._warm_pool_identity[sandbox_id] = None
                 adopted += 1
             logger.info("Adopted existing BoxLite box %s (%s) into warm pool", sandbox_id, name)
 
@@ -348,6 +373,7 @@ class BoxliteProvider(WarmPoolLifecycleMixin[BoxliteBox], SandboxProvider):
             box = self._create_box(sandbox_id)
             with self._lock:
                 self._boxes[box.id] = box
+                self._active_box_identity[box.id] = None
             return box.id
 
         key = self._thread_key(thread_id, user_id)
@@ -358,17 +384,42 @@ class BoxliteProvider(WarmPoolLifecycleMixin[BoxliteBox], SandboxProvider):
                 existing = self._thread_boxes.get(key)
                 if existing is not None and existing in self._boxes:
                     return existing
+                if sandbox_id in self._boxes:
+                    owner = self._active_box_identity.get(sandbox_id)
+                    if owner != key:
+                        raise SandboxIdentityCollisionError(
+                            sandbox_id,
+                            owner,
+                            key,
+                        )
+                    self._thread_boxes[key] = sandbox_id
+                    return sandbox_id
 
-            reclaimed = self._reclaim_warm_pool(sandbox_id)
+            reclaimed = self._reclaim_warm_pool(sandbox_id, key)
             if reclaimed is not None:
                 with self._lock:
                     self._thread_boxes[key] = reclaimed
                 return reclaimed
 
             box = self._create_box(sandbox_id)
+            conflict: tuple[str, str] | None | object = _NO_ACTIVE_IDENTITY
             with self._lock:
-                self._boxes[box.id] = box
-                self._thread_boxes[key] = box.id
+                if box.id in self._boxes:
+                    conflict = self._active_box_identity.get(box.id)
+                    if conflict == key:
+                        self._thread_boxes[key] = box.id
+                else:
+                    self._boxes[box.id] = box
+                    self._active_box_identity[box.id] = key
+                    self._thread_boxes[key] = box.id
+            if conflict is not _NO_ACTIVE_IDENTITY:
+                box.close()
+                if conflict != key:
+                    raise SandboxIdentityCollisionError(
+                        sandbox_id,
+                        conflict,
+                        key,
+                    )
             return box.id
 
     def _create_box(self, sandbox_id: str) -> BoxliteBox:
@@ -410,15 +461,19 @@ class BoxliteProvider(WarmPoolLifecycleMixin[BoxliteBox], SandboxProvider):
         close_box: BoxliteBox | None = None
         with self._lock:
             box = self._boxes.pop(sandbox_id, None)
-            for key in [k for k, sid in self._thread_boxes.items() if sid == sandbox_id]:
+            released_keys = [k for k, sid in self._thread_boxes.items() if sid == sandbox_id]
+            for key in released_keys:
                 self._thread_boxes.pop(key, None)
+            active_identity = self._active_box_identity.pop(sandbox_id, None)
             if box is None:
                 return
             if self._shutdown_called:
                 close_box = box
                 self._skip_health_check_warm_ids.discard(sandbox_id)
+                self._warm_pool_identity.pop(sandbox_id, None)
             else:
                 self._warm_pool[sandbox_id] = (box, time.time())
+                self._warm_pool_identity[sandbox_id] = released_keys[0] if released_keys else active_identity
                 self._skip_health_check_warm_ids.add(sandbox_id)
 
         if close_box is not None:
@@ -427,7 +482,7 @@ class BoxliteProvider(WarmPoolLifecycleMixin[BoxliteBox], SandboxProvider):
         else:
             logger.info("Released sandbox %s to warm pool (VM still running)", sandbox_id)
 
-    def _reclaim_warm_pool(self, sandbox_id: str) -> str | None:
+    def _reclaim_warm_pool(self, sandbox_id: str, expected_key: tuple[str, str]) -> str | None:
         """Try to reclaim a warm-pool box by sandbox_id.
 
         Returns sandbox_id on success, None if not found or dead.
@@ -440,6 +495,14 @@ class BoxliteProvider(WarmPoolLifecycleMixin[BoxliteBox], SandboxProvider):
         with self._lock:
             if sandbox_id not in self._warm_pool:
                 return None
+            # Startup-adopted entries have unknown identity until their first reclaim.
+            stored_key = self._warm_pool_identity.get(sandbox_id)
+            if stored_key is not None and stored_key != expected_key:
+                raise SandboxIdentityCollisionError(
+                    sandbox_id,
+                    stored_key,
+                    expected_key,
+                )
             box, released_at = self._warm_pool[sandbox_id]
             skip_eligible = sandbox_id in self._skip_health_check_warm_ids
 
@@ -449,10 +512,21 @@ class BoxliteProvider(WarmPoolLifecycleMixin[BoxliteBox], SandboxProvider):
             # health-check round trip, but never return an adapter that this
             # process already knows is closed.
             with self._lock:
+                current = self._warm_pool.get(sandbox_id)
+                stored_key = self._warm_pool_identity.get(sandbox_id)
+                if stored_key is not None and stored_key != expected_key:
+                    raise SandboxIdentityCollisionError(
+                        sandbox_id,
+                        stored_key,
+                        expected_key,
+                    )
+                if current is None or current[0] is not box:
+                    return None
                 warm_entry = self._warm_pool.pop(sandbox_id, None)
                 if warm_entry is None:
                     return None  # Raced with another thread
                 self._skip_health_check_warm_ids.discard(sandbox_id)
+                self._warm_pool_identity.pop(sandbox_id, None)
                 box, _ = warm_entry
                 if box.is_closed:
                     logger.warning("Warm-pool box %s was closed before skipped health check reclaim", sandbox_id)
@@ -460,6 +534,7 @@ class BoxliteProvider(WarmPoolLifecycleMixin[BoxliteBox], SandboxProvider):
                 else:
                     close_box = None
                     self._boxes[sandbox_id] = box
+                    self._active_box_identity[sandbox_id] = expected_key
             if close_box is not None:
                 close_box.close()
                 return None
@@ -476,26 +551,60 @@ class BoxliteProvider(WarmPoolLifecycleMixin[BoxliteBox], SandboxProvider):
             if "ok" not in result:
                 logger.warning("Warm pool box %s health check failed: %s", sandbox_id, result)
                 with self._lock:
-                    warm_entry = self._warm_pool.pop(sandbox_id, None)
+                    current = self._warm_pool.get(sandbox_id)
+                    stored_key = self._warm_pool_identity.get(sandbox_id)
+                    if current is not None and current[0] is not box and stored_key is not None and stored_key != expected_key:
+                        raise SandboxIdentityCollisionError(
+                            sandbox_id,
+                            stored_key,
+                            expected_key,
+                        )
+                    warm_entry = self._warm_pool.pop(sandbox_id, None) if current is not None and current[0] is box else None
+                    if warm_entry is not None:
+                        self._warm_pool_identity.pop(sandbox_id, None)
                 if warm_entry is not None:
                     self._destroy_warm_entry(sandbox_id, warm_entry[0], reason="health_check_failed")
                 return None
+        except SandboxIdentityCollisionError:
+            raise
         except Exception as e:
             logger.warning("Warm pool box %s health check error: %s", sandbox_id, e)
             with self._lock:
-                warm_entry = self._warm_pool.pop(sandbox_id, None)
+                current = self._warm_pool.get(sandbox_id)
+                stored_key = self._warm_pool_identity.get(sandbox_id)
+                if current is not None and current[0] is not box and stored_key is not None and stored_key != expected_key:
+                    raise SandboxIdentityCollisionError(
+                        sandbox_id,
+                        stored_key,
+                        expected_key,
+                    )
+                warm_entry = self._warm_pool.pop(sandbox_id, None) if current is not None and current[0] is box else None
+                if warm_entry is not None:
+                    self._warm_pool_identity.pop(sandbox_id, None)
             if warm_entry is not None:
                 self._destroy_warm_entry(sandbox_id, warm_entry[0], reason="health_check_failed")
             return None
 
         # Promote from warm pool to active
         with self._lock:
+            current = self._warm_pool.get(sandbox_id)
+            stored_key = self._warm_pool_identity.get(sandbox_id)
+            if stored_key is not None and stored_key != expected_key:
+                raise SandboxIdentityCollisionError(
+                    sandbox_id,
+                    stored_key,
+                    expected_key,
+                )
+            if current is None or current[0] is not box:
+                return None
             warm_entry = self._warm_pool.pop(sandbox_id, None)
             if warm_entry is None:
                 return None  # Raced with another thread
             self._skip_health_check_warm_ids.discard(sandbox_id)
+            self._warm_pool_identity.pop(sandbox_id, None)
             box, _ = warm_entry
             self._boxes[sandbox_id] = box
+            self._active_box_identity[sandbox_id] = expected_key
 
         logger.info("Reclaimed warm-pool box %s", sandbox_id)
         return sandbox_id
@@ -513,8 +622,10 @@ class BoxliteProvider(WarmPoolLifecycleMixin[BoxliteBox], SandboxProvider):
             now = time.time()
             for sandbox_id, box in self._boxes.items():
                 self._warm_pool.setdefault(sandbox_id, (box, now))
+                self._warm_pool_identity.setdefault(sandbox_id, self._active_box_identity.get(sandbox_id))
                 self._skip_health_check_warm_ids.discard(sandbox_id)
             self._boxes.clear()
+            self._active_box_identity.clear()
             self._thread_boxes.clear()
             self._acquire_locks.clear()
 
@@ -531,6 +642,8 @@ class BoxliteProvider(WarmPoolLifecycleMixin[BoxliteBox], SandboxProvider):
             warm = [box for box, _ in self._warm_pool.values()]
             self._boxes.clear()
             self._warm_pool.clear()
+            self._active_box_identity.clear()
+            self._warm_pool_identity.clear()
             self._thread_boxes.clear()
             self._acquire_locks.clear()
             self._skip_health_check_warm_ids.clear()

backend/packages/harness/deerflow/config/extensions_config.py

@@ -315,7 +315,7 @@ class ExtensionsConfig(BaseModel):
         skill_config = self.skills.get(skill_name)
         if skill_config is None:
             # Default to enabled for all skill categories
-            return skill_category in ("public", "custom", "legacy")
+            return skill_category in ("public", "custom", "legacy", "integrations")
         return skill_config.enabled
 
 

backend/packages/harness/deerflow/config/paths.py

@@ -12,6 +12,7 @@ VIRTUAL_PATH_PREFIX = "/mnt/user-data"
 
 _SAFE_THREAD_ID_RE = re.compile(r"^[A-Za-z0-9_\-]+$")
 _SAFE_USER_ID_RE = re.compile(r"^[A-Za-z0-9_\-]+$")
+_SAFE_INTEGRATION_ID_RE = re.compile(r"^[A-Za-z0-9_.\-]+$")
 _UNSAFE_USER_ID_CHAR_RE = re.compile(r"[^A-Za-z0-9_\-]")
 _SAFE_USER_ID_DIGEST_HEX_LEN = 16
 
@@ -37,6 +38,18 @@ def _validate_user_id(user_id: str) -> str:
     return user_id
 
 
+def _validate_integration_id(integration_id: str) -> str:
+    """Validate an integration ID before using it in filesystem paths."""
+    if not _SAFE_INTEGRATION_ID_RE.match(integration_id):
+        raise ValueError(f"Invalid integration_id {integration_id!r}: only alphanumeric characters, dots, hyphens, and underscores are allowed.")
+    # The charset allows dots for names like ``some.integration``; reject the
+    # bare ``.``/``..`` path components so a future caller cannot escape the
+    # per-integration namespace via ``_join_host_path(..., integration_id, ...)``.
+    if integration_id in {".", ".."}:
+        raise ValueError(f"Invalid integration_id {integration_id!r}: '.' and '..' are not allowed.")
+    return integration_id
+
+
 def make_safe_user_id(raw: str) -> str:
     """Normalize an external identity into the user-id charset (``[A-Za-z0-9_-]``).
 
@@ -236,6 +249,15 @@ class Paths:
         """
         return self.user_skills_dir(user_id) / "custom"
 
+    def integration_skills_dir(self) -> Path:
+        """Globally installed managed integration skills.
+
+        Layout: ``{base_dir}/integrations/skills/{provider}/{skill}/``. The
+        package contents are shared and read-only; credentials and enabled
+        state remain user-scoped elsewhere under ``users/{user_id}``.
+        """
+        return self.base_dir / "integrations" / "skills"
+
     def thread_dir(self, thread_id: str, *, user_id: str | None = None) -> Path:
         """
         Host path for a thread's data.
@@ -325,6 +347,22 @@ class Paths:
         """Host path for the ACP workspace mount source."""
         return _join_host_path(self.host_thread_dir(thread_id, user_id=user_id), "acp-workspace")
 
+    def host_user_custom_skills_dir(self, user_id: str) -> str:
+        """Host path for a user's custom skills directory, preserving Windows path syntax."""
+        return _join_host_path(self._host_base_dir_str(), "users", _validate_user_id(user_id), "skills", "custom")
+
+    def host_integration_skills_dir(self) -> str:
+        """Host path for globally installed managed integration skills."""
+        return _join_host_path(self._host_base_dir_str(), "integrations", "skills")
+
+    def host_user_integration_config_dir(self, user_id: str, integration_id: str) -> str:
+        """Host path for a user's managed integration runtime config directory."""
+        return _join_host_path(self._host_base_dir_str(), "users", _validate_user_id(user_id), "integrations", _validate_integration_id(integration_id), "config")
+
+    def host_user_integration_data_dir(self, user_id: str, integration_id: str) -> str:
+        """Host path for a user's managed integration runtime data directory."""
+        return _join_host_path(self._host_base_dir_str(), "users", _validate_user_id(user_id), "integrations", _validate_integration_id(integration_id), "data")
+
     def ensure_thread_dirs(self, thread_id: str, *, user_id: str | None = None) -> None:
         """Create all standard sandbox directories for a thread.
 

backend/packages/harness/deerflow/config/summarization_config.py

@@ -28,7 +28,10 @@ class SummarizationConfig(BaseModel):
     )
     model_name: str | None = Field(
         default=None,
-        description="Model name to use for summarization (None = use a lightweight model)",
+        description="Model name to use for summarization. None = summarize with the model the run "
+        "actually executes with (the lead run's model, a subagent's own model, or a thread's "
+        "custom-agent model), not config.models[0]. When set, that model generates and the run's "
+        "own model is used as a fallback if the configured summary provider fails.",
     )
     trigger: ContextSize | list[ContextSize] | None = Field(
         default=None,

backend/packages/harness/deerflow/integrations/__init__.py

@@ -0,0 +1 @@
+"""First-party integration installers and status helpers."""

backend/packages/harness/deerflow/mcp/tools.py

@@ -70,7 +70,10 @@ def _local_path_from_uri(uri: str, *, base_dir: Path | None = None) -> Path | No
     """
     if not uri:
         return None
-    parsed = urlparse(uri)
+    try:
+        parsed = urlparse(uri)
+    except ValueError:
+        return None
     if parsed.scheme == "file":
         raw = unquote(parsed.path)
     elif parsed.scheme == "":

backend/packages/harness/deerflow/persistence/migrations/versions/0008_thread_operation_kind.py

@@ -0,0 +1,36 @@
+"""thread operation kind.
+
+Revision ID: 0008_thread_operation_kind
+Revises: 0007_scheduled_run_active_index
+Create Date: 2026-07-24
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0008_thread_operation_kind"
+down_revision: str | Sequence[str] | None = "0007_scheduled_run_active_index"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def upgrade() -> None:
+    from deerflow.persistence.migrations._helpers import safe_add_column
+
+    safe_add_column(
+        "runs",
+        sa.Column(
+            "operation_kind",
+            sa.String(length=32),
+            nullable=False,
+            server_default=sa.text("'run'"),
+        ),
+    )
+
+
+def downgrade() -> None:
+    op.drop_column("runs", "operation_kind")

backend/packages/harness/deerflow/persistence/migrations/versions/0009_feedback_tags.py

@@ -1,7 +1,7 @@
 """feedback tags.
 
-Revision ID: 0008_feedback_tags
-Revises: 0007_scheduled_run_active_index
+Revision ID: 0009_feedback_tags
+Revises: 0008_thread_operation_kind
 Create Date: 2026-07-23
 """
 
@@ -12,8 +12,8 @@ from collections.abc import Sequence
 import sqlalchemy as sa
 from alembic import op
 
-revision: str = "0008_feedback_tags"
-down_revision: str | Sequence[str] | None = "0007_scheduled_run_active_index"
+revision: str = "0009_feedback_tags"
+down_revision: str | Sequence[str] | None = "0008_thread_operation_kind"
 branch_labels: str | Sequence[str] | None = None
 depends_on: str | Sequence[str] | None = None
 

backend/packages/harness/deerflow/persistence/run/model.py

@@ -19,6 +19,7 @@ class RunRow(Base):
     user_id: Mapped[str | None] = mapped_column(String(64), index=True)
     status: Mapped[str] = mapped_column(String(20), default="pending")
     # "pending" | "running" | "success" | "error" | "timeout" | "interrupted"
+    operation_kind: Mapped[str] = mapped_column(String(32), nullable=False, default="run", server_default=text("'run'"))
 
     model_name: Mapped[str | None] = mapped_column(String(128))
     multitask_strategy: Mapped[str] = mapped_column(String(20), default="reject")

backend/packages/harness/deerflow/persistence/run/sql.py

@@ -92,6 +92,7 @@ class RunRepository(RunStore):
         user_id: str | None | _AutoSentinel = AUTO,
         model_name: str | None = None,
         status="pending",
+        operation_kind: str = "run",
         multitask_strategy="reject",
         metadata=None,
         kwargs=None,
@@ -118,6 +119,7 @@ class RunRepository(RunStore):
             "user_id": resolved_user_id,
             "model_name": self._normalize_model_name(model_name),
             "status": status,
+            "operation_kind": operation_kind,
             "multitask_strategy": multitask_strategy,
             "metadata_json": self._safe_json(metadata) or {},
             "kwargs_json": self._safe_json(kwargs) or {},
@@ -160,7 +162,7 @@ class RunRepository(RunStore):
         limit=100,
     ):
         resolved_user_id = resolve_user_id(user_id, method_name="RunRepository.list_by_thread")
-        stmt = select(RunRow).where(RunRow.thread_id == thread_id)
+        stmt = select(RunRow).where(RunRow.thread_id == thread_id, RunRow.operation_kind == "run")
         if resolved_user_id is not None:
             stmt = stmt.where(RunRow.user_id == resolved_user_id)
         stmt = stmt.order_by(RunRow.created_at.desc()).limit(limit)
@@ -178,6 +180,7 @@ class RunRepository(RunStore):
         source = RunRow.metadata_json["regenerate_from_run_id"].as_string()
         stmt = select(source).where(
             RunRow.thread_id == thread_id,
+            RunRow.operation_kind == "run",
             RunRow.status == "success",
             source.is_not(None),
             source != "",
@@ -198,7 +201,7 @@ class RunRepository(RunStore):
         if not run_ids:
             return {}
         resolved_user_id = resolve_user_id(user_id, method_name="RunRepository.get_many_by_thread")
-        stmt = select(RunRow).where(RunRow.thread_id == thread_id, RunRow.run_id.in_(run_ids))
+        stmt = select(RunRow).where(RunRow.thread_id == thread_id, RunRow.operation_kind == "run", RunRow.run_id.in_(run_ids))
         if resolved_user_id is not None:
             stmt = stmt.where(RunRow.user_id == resolved_user_id)
         async with self._sf() as session:
@@ -221,6 +224,20 @@ class RunRepository(RunStore):
             await session.commit()
             return result.rowcount != 0
 
+    async def start_run(self, run_id: str) -> bool:
+        """Start only a still-pending run; cancelled rows must not be resurrected."""
+        async with self._sf() as session:
+            result = await session.execute(
+                update(RunRow)
+                .where(
+                    RunRow.run_id == run_id,
+                    RunRow.status == "pending",
+                )
+                .values(status="running", updated_at=datetime.now(UTC))
+            )
+            await session.commit()
+            return result.rowcount != 0
+
     async def update_model_name(self, run_id, model_name):
         async with self._sf() as session:
             await session.execute(update(RunRow).where(RunRow.run_id == run_id).values(model_name=self._normalize_model_name(model_name), updated_at=datetime.now(UTC)))
@@ -242,6 +259,10 @@ class RunRepository(RunStore):
             await session.delete(row)
             await session.commit()
 
+    async def delete_thread_operation(self, run_id: str, *, user_id: str | None) -> None:
+        """Release a reservation using its captured owner, not request context."""
+        await self.delete(run_id, user_id=user_id)
+
     async def list_pending(self, *, before=None):
         if before is None:
             before_dt = datetime.now(UTC)
@@ -249,7 +270,7 @@ class RunRepository(RunStore):
             before_dt = before
         else:
             before_dt = datetime.fromisoformat(before)
-        stmt = select(RunRow).where(RunRow.status == "pending", RunRow.created_at <= before_dt).order_by(RunRow.created_at.asc())
+        stmt = select(RunRow).where(RunRow.operation_kind == "run", RunRow.status == "pending", RunRow.created_at <= before_dt).order_by(RunRow.created_at.asc())
         async with self._sf() as session:
             result = await session.execute(stmt)
             return [self._row_to_dict(r) for r in result.scalars()]
@@ -294,7 +315,8 @@ class RunRepository(RunStore):
     ) -> bool:
         """Update status + token usage + convenience fields on run completion.
 
-        Returns ``False`` when no run row matched the requested ``run_id``.
+        Returns ``False`` when the row is missing or already has a conflicting
+        terminal outcome.
         """
         values: dict[str, Any] = {
             "status": status,
@@ -315,8 +337,20 @@ class RunRepository(RunStore):
             values["first_human_message"] = first_human_message[:2000]
         if error is not None:
             values["error"] = error
+        allowed_sources = ["pending", "running"]
+        if status not in allowed_sources:
+            allowed_sources.append(status)
+        if status == "error" and "interrupted" not in allowed_sources:
+            allowed_sources.append("interrupted")
         async with self._sf() as session:
-            result = await session.execute(update(RunRow).where(RunRow.run_id == run_id).values(**values))
+            result = await session.execute(
+                update(RunRow)
+                .where(
+                    RunRow.run_id == run_id,
+                    RunRow.status.in_(tuple(allowed_sources)),
+                )
+                .values(**values)
+            )
             await session.commit()
             return result.rowcount != 0
 
@@ -378,6 +412,7 @@ class RunRepository(RunStore):
         statuses = ("success", "error", "running") if include_active else ("success", "error")
         _completed = RunRow.status.in_(statuses)
         _thread = RunRow.thread_id == thread_id
+        _run_operation = RunRow.operation_kind == "run"
 
         stmt = select(
             RunRow.model_name,
@@ -388,7 +423,7 @@ class RunRepository(RunStore):
             RunRow.subagent_tokens,
             RunRow.middleware_tokens,
             RunRow.token_usage_by_model,
-        ).where(_thread, _completed)
+        ).where(_thread, _run_operation, _completed)
 
         async with self._sf() as session:
             rows = (await session.execute(stmt)).all()
@@ -510,13 +545,14 @@ class RunRepository(RunStore):
             result = await session.execute(stmt)
             return [self._row_to_dict(r) for r in result.scalars()]
 
-    async def create_run_atomic(
+    async def create_thread_operation_atomic(
         self,
         run_id: str,
         *,
         thread_id: str,
         owner_worker_id: str,
         lease_expires_at: str | None,
+        operation_kind: str = "run",
         multitask_strategy: str = "reject",
         assistant_id: str | None = None,
         user_id: str | None = None,
@@ -541,7 +577,7 @@ class RunRepository(RunStore):
         """
         from deerflow.runtime.runs.manager import ConflictError
 
-        resolved_user_id = resolve_user_id(user_id or AUTO, method_name="RunRepository.create_run_atomic")
+        resolved_user_id = resolve_user_id(user_id or AUTO, method_name="RunRepository.create_thread_operation_atomic")
         now = datetime.now(UTC)
         created = datetime.fromisoformat(created_at) if created_at else now
         lease_dt = datetime.fromisoformat(lease_expires_at) if lease_expires_at else None
@@ -553,6 +589,7 @@ class RunRepository(RunStore):
             "user_id": resolved_user_id,
             "model_name": self._normalize_model_name(model_name),
             "status": "pending",
+            "operation_kind": operation_kind,
             "multitask_strategy": multitask_strategy,
             "metadata_json": self._safe_json(metadata) or {},
             "kwargs_json": self._safe_json(kwargs) or {},
@@ -576,6 +613,7 @@ class RunRepository(RunStore):
                 )
                 result = await session.execute(stmt)
                 for row in result.scalars():
+                    lease_expired = False
                     if row.lease_expires_at is not None:
                         # SQLite drops tzinfo on read despite
                         # ``DateTime(timezone=True)`` (see ``_row_to_dict``).
@@ -588,6 +626,7 @@ class RunRepository(RunStore):
                         row_lease = row.lease_expires_at
                         if row_lease.tzinfo is None:
                             row_lease = row_lease.replace(tzinfo=UTC)
+                        lease_expired = row_lease < cutoff
                         if row_lease >= cutoff and row.owner_worker_id != owner_worker_id:
                             # Live run owned by another worker — we cannot
                             # interrupt it and the partial unique index would
@@ -595,6 +634,8 @@ class RunRepository(RunStore):
                             # ConflictError so the caller gets a clean signal
                             # instead of a retry loop on IntegrityError.
                             raise ConflictError(f"Thread {thread_id} already has an active run owned by another worker")
+                    if row.operation_kind != "run" and not lease_expired:
+                        raise ConflictError(f"Thread {thread_id} has an active checkpoint write")
                     row.status = "interrupted"
                     row.error = "Cancelled by newer run"
                     row.owner_worker_id = owner_worker_id

backend/packages/harness/deerflow/persistence/thread_meta/__init__.py

@@ -4,7 +4,7 @@ from __future__ import annotations
 
 from typing import TYPE_CHECKING
 
-from deerflow.persistence.thread_meta.base import InvalidMetadataFilterError, ThreadMetaStore
+from deerflow.persistence.thread_meta.base import THREAD_PINNED_METADATA_KEY, InvalidMetadataFilterError, ThreadMetaStore
 from deerflow.persistence.thread_meta.memory import MemoryThreadMetaStore
 from deerflow.persistence.thread_meta.model import ThreadMetaRow
 from deerflow.persistence.thread_meta.sql import ThreadMetaRepository
@@ -16,6 +16,7 @@ if TYPE_CHECKING:
 __all__ = [
     "InvalidMetadataFilterError",
     "MemoryThreadMetaStore",
+    "THREAD_PINNED_METADATA_KEY",
     "ThreadMetaRepository",
     "ThreadMetaRow",
     "ThreadMetaStore",

backend/packages/harness/deerflow/persistence/thread_meta/base.py

@@ -19,6 +19,11 @@ from typing import Any
 
 from deerflow.runtime.user_context import AUTO, _AutoSentinel
 
+# Cross-component metadata key. Keep in sync with
+# ``frontend/src/core/threads/utils.ts`` and
+# ``frontend/tests/e2e/utils/mock-api.ts``.
+THREAD_PINNED_METADATA_KEY = "deerflow_pinned"
+
 
 class InvalidMetadataFilterError(ValueError):
     """Raised when all client-supplied metadata filter keys are rejected."""
@@ -51,6 +56,12 @@ class ThreadMetaStore(abc.ABC):
         offset: int = 0,
         user_id: str | None | _AutoSentinel = AUTO,
     ) -> list[dict[str, Any]]:
+        """Search threads.
+
+        Results are ordered with pinned threads first
+        (``metadata.deerflow_pinned is True``), then by ``updated_at`` and
+        ``thread_id`` descending within each group.
+        """
         pass
 
     @abc.abstractmethod
@@ -62,12 +73,17 @@ class ThreadMetaStore(abc.ABC):
         pass
 
     @abc.abstractmethod
-    async def update_metadata(self, thread_id: str, metadata: dict, *, user_id: str | None | _AutoSentinel = AUTO) -> None:
+    async def update_metadata(self, thread_id: str, metadata: dict, *, touch: bool = True, user_id: str | None | _AutoSentinel = AUTO) -> None:
         """Merge ``metadata`` into the thread's metadata field.
 
         Existing keys are overwritten by the new values; keys absent from
         ``metadata`` are preserved. No-op if the thread does not exist
         or the owner check fails.
+
+        When ``touch`` is ``True`` (default) the row's ``updated_at`` is
+        refreshed so the change bumps recency ordering. Pass ``touch=False``
+        for metadata that is not conversation activity (e.g. pin/unpin) so the
+        thread keeps its place in ``updated_at``-sorted lists.
         """
         pass
 

backend/packages/harness/deerflow/persistence/thread_meta/memory.py

@@ -11,11 +11,12 @@ from typing import Any
 
 from langgraph.store.base import BaseStore
 
-from deerflow.persistence.thread_meta.base import ThreadMetaStore
+from deerflow.persistence.thread_meta.base import THREAD_PINNED_METADATA_KEY, ThreadMetaStore
 from deerflow.runtime.user_context import AUTO, _AutoSentinel, resolve_user_id
 from deerflow.utils.time import coerce_iso, now_iso
 
 THREADS_NS: tuple[str, ...] = ("threads",)
+SEARCH_PAGE_SIZE = 500
 
 
 class MemoryThreadMetaStore(ThreadMetaStore):
@@ -75,6 +76,12 @@ class MemoryThreadMetaStore(ThreadMetaStore):
         offset: int = 0,
         user_id: str | None | _AutoSentinel = AUTO,
     ) -> list[dict[str, Any]]:
+        """Search threads by materializing matches, then sorting in Python.
+
+        The memory backend loads all matching rows in chunks before slicing so
+        it can mirror SQL's pinned-first ordering. Use the SQL store for
+        scalable paginated I/O.
+        """
         resolved_user_id = resolve_user_id(user_id, method_name="MemoryThreadMetaStore.search")
         filter_dict: dict[str, Any] = {}
         if metadata:
@@ -84,13 +91,25 @@ class MemoryThreadMetaStore(ThreadMetaStore):
         if resolved_user_id is not None:
             filter_dict["user_id"] = resolved_user_id
 
-        items = await self._store.asearch(
-            THREADS_NS,
-            filter=filter_dict or None,
-            limit=limit,
-            offset=offset,
-        )
-        return [self._item_to_dict(item) for item in items]
+        items = []
+        search_offset = 0
+        while True:
+            page = await self._store.asearch(
+                THREADS_NS,
+                filter=filter_dict or None,
+                limit=SEARCH_PAGE_SIZE,
+                offset=search_offset,
+            )
+            if not page:
+                break
+            items.extend(page)
+            if len(page) < SEARCH_PAGE_SIZE:
+                break
+            search_offset += len(page)
+
+        records = [self._item_to_dict(item) for item in items]
+        records.sort(key=self._sort_key, reverse=True)
+        return records[offset : offset + limit]
 
     async def check_access(self, thread_id: str, user_id: str, *, require_existing: bool = False) -> bool:
         item = await self._store.aget(THREADS_NS, thread_id)
@@ -117,14 +136,15 @@ class MemoryThreadMetaStore(ThreadMetaStore):
         record["updated_at"] = now_iso()
         await self._store.aput(THREADS_NS, thread_id, record)
 
-    async def update_metadata(self, thread_id: str, metadata: dict, *, user_id: str | None | _AutoSentinel = AUTO) -> None:
+    async def update_metadata(self, thread_id: str, metadata: dict, *, touch: bool = True, user_id: str | None | _AutoSentinel = AUTO) -> None:
         record = await self._get_owned_record(thread_id, user_id, "MemoryThreadMetaStore.update_metadata")
         if record is None:
             return
         merged = dict(record.get("metadata") or {})
         merged.update(metadata)
         record["metadata"] = merged
-        record["updated_at"] = now_iso()
+        if touch:
+            record["updated_at"] = now_iso()
         await self._store.aput(THREADS_NS, thread_id, record)
 
     async def update_owner(self, thread_id: str, owner_user_id: str, *, user_id: str | None | _AutoSentinel = AUTO) -> None:
@@ -157,3 +177,9 @@ class MemoryThreadMetaStore(ThreadMetaStore):
             "created_at": coerce_iso(val.get("created_at", "")),
             "updated_at": coerce_iso(val.get("updated_at", "")),
         }
+
+    @staticmethod
+    def _sort_key(record: dict[str, Any]) -> tuple[bool, str, str]:
+        metadata = record.get("metadata")
+        pinned = isinstance(metadata, dict) and metadata.get(THREAD_PINNED_METADATA_KEY) is True
+        return (pinned, str(record.get("updated_at") or ""), str(record.get("thread_id") or ""))

backend/packages/harness/deerflow/persistence/thread_meta/sql.py

@@ -6,11 +6,12 @@ import logging
 from datetime import UTC, datetime
 from typing import Any
 
-from sqlalchemy import select, update
+from sqlalchemy import case, select, update
 from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+from sqlalchemy.orm.attributes import flag_modified
 
 from deerflow.persistence.json_compat import json_match
-from deerflow.persistence.thread_meta.base import InvalidMetadataFilterError, ThreadMetaStore
+from deerflow.persistence.thread_meta.base import THREAD_PINNED_METADATA_KEY, InvalidMetadataFilterError, ThreadMetaStore
 from deerflow.persistence.thread_meta.model import ThreadMetaRow
 from deerflow.runtime.user_context import AUTO, _AutoSentinel, resolve_user_id
 from deerflow.utils.time import coerce_iso
@@ -123,7 +124,15 @@ class ThreadMetaRepository(ThreadMetaStore):
         context. Pass ``user_id=None`` to bypass (migration/CLI).
         """
         resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.search")
-        stmt = select(ThreadMetaRow).order_by(ThreadMetaRow.updated_at.desc(), ThreadMetaRow.thread_id.desc())
+        pinned_order = case(
+            (json_match(ThreadMetaRow.metadata_json, THREAD_PINNED_METADATA_KEY, True), 1),
+            else_=0,
+        )
+        stmt = select(ThreadMetaRow).order_by(
+            pinned_order.desc(),
+            ThreadMetaRow.updated_at.desc(),
+            ThreadMetaRow.thread_id.desc(),
+        )
         if resolved_user_id is not None:
             stmt = stmt.where(ThreadMetaRow.user_id == resolved_user_id)
         if status:
@@ -190,6 +199,7 @@ class ThreadMetaRepository(ThreadMetaStore):
         thread_id: str,
         metadata: dict,
         *,
+        touch: bool = True,
         user_id: str | None | _AutoSentinel = AUTO,
     ) -> None:
         """Merge ``metadata`` into ``metadata_json``.
@@ -197,6 +207,9 @@ class ThreadMetaRepository(ThreadMetaStore):
         Read-modify-write inside a single session/transaction so concurrent
         callers see consistent state. No-op if the row does not exist or
         the user_id check fails.
+
+        ``touch`` refreshes ``updated_at`` (default); pass ``touch=False`` to
+        preserve recency ordering for metadata-only changes such as pin/unpin.
         """
         resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.update_metadata")
         async with self._sf() as session:
@@ -208,7 +221,14 @@ class ThreadMetaRepository(ThreadMetaStore):
             merged = dict(row.metadata_json or {})
             merged.update(metadata)
             row.metadata_json = merged
-            row.updated_at = datetime.now(UTC)
+            if touch:
+                row.updated_at = datetime.now(UTC)
+            else:
+                # ``updated_at`` has an ``onupdate`` hook that fires on any row
+                # UPDATE unless the column has an explicit SET value. Mark the
+                # current value dirty so SQLAlchemy emits it in SET, skips the
+                # hook, and preserves recency ordering.
+                flag_modified(row, "updated_at")
             await session.commit()
 
     async def update_owner(

backend/packages/harness/deerflow/runtime/__init__.py

@@ -7,14 +7,14 @@ directly from ``deerflow.runtime``.
 
 from .checkpoint_state import CheckpointStateAccessor, build_state_mutation_graph
 from .checkpointer import checkpointer_context, get_checkpointer, make_checkpointer, reset_checkpointer
-from .runs import ORPHAN_RECOVERY_STOP_REASON, STARTUP_ORPHAN_RECOVERY_ERROR, CancelOutcome, ConflictError, DisconnectMode, RunContext, RunManager, RunRecord, RunStatus, UnsupportedStrategyError, run_agent
+from .runs import ORPHAN_RECOVERY_STOP_REASON, STARTUP_ORPHAN_RECOVERY_ERROR, CancelOutcome, ConflictError, DisconnectMode, RunContext, RunManager, RunRecord, RunStatus, ThreadOperationKind, UnsupportedStrategyError, run_agent
 from .serialization import serialize, serialize_channel_values, serialize_channel_values_for_api, serialize_lc_object, serialize_messages_tuple, strip_data_url_image_blocks
 from .store import get_store, make_store, reset_store, store_context
 
 # NOTE: ``RedisStreamBridge`` is intentionally not re-exported — ``redis`` is an
 # optional extra and importing it here would load ``redis.asyncio`` in every
 # process. Import it from ``deerflow.runtime.stream_bridge.redis`` when needed.
-from .stream_bridge import END_SENTINEL, HEARTBEAT_SENTINEL, MemoryStreamBridge, StreamBridge, StreamEvent, make_stream_bridge
+from .stream_bridge import END_SENTINEL, HEARTBEAT_SENTINEL, MemoryStreamBridge, StreamBridge, StreamEvent, StreamGap, StreamItem, make_stream_bridge
 
 __all__ = [
     # checkpoint state
@@ -34,6 +34,7 @@ __all__ = [
     "RunManager",
     "RunRecord",
     "RunStatus",
+    "ThreadOperationKind",
     "STARTUP_ORPHAN_RECOVERY_ERROR",
     "UnsupportedStrategyError",
     "run_agent",
@@ -55,5 +56,7 @@ __all__ = [
     "MemoryStreamBridge",
     "StreamBridge",
     "StreamEvent",
+    "StreamGap",
+    "StreamItem",
     "make_stream_bridge",
 ]

backend/packages/harness/deerflow/runtime/context_compaction.py

@@ -2,15 +2,19 @@
 
 from __future__ import annotations
 
+import asyncio
+import logging
 from dataclasses import dataclass
 from types import SimpleNamespace
 
 from langgraph.types import Overwrite
 
-from deerflow.agents.middlewares.summarization_middleware import DeerFlowSummarizationMiddleware, create_summarization_middleware
+from deerflow.agents.middlewares.summarization_middleware import DeerFlowSummarizationMiddleware, SummaryGenerationError, create_summarization_middleware
 from deerflow.config.app_config import AppConfig, get_app_config
 from deerflow.runtime.checkpoint_state import CheckpointStateAccessor
 
+logger = logging.getLogger(__name__)
+
 
 class ContextCompactionDisabled(RuntimeError):
     """Raised when manual compaction is requested while summarization is disabled."""
@@ -38,13 +42,60 @@ def _create_compaction_middleware(
     *,
     app_config: AppConfig,
     keep: tuple[str, int | float] | None,
+    run_model_name: str | None = None,
 ) -> DeerFlowSummarizationMiddleware:
-    middleware = create_summarization_middleware(app_config=app_config, keep=keep)
+    middleware = create_summarization_middleware(app_config=app_config, keep=keep, run_model_name=run_model_name)
     if middleware is None:
         raise ContextCompactionDisabled("Context compaction is disabled.")
     return middleware
 
 
+def _safe_load_agent_config(agent_name: str, user_id: str | None):
+    """Load a custom agent's config, returning ``None`` on any failure.
+
+    A missing / unparseable agent config must not fail compaction; the run model is a
+    best-effort optimization and the default is a safe fallback. The caller runs this
+    off the event loop via ``asyncio.to_thread``, so the strict blocking-IO detector
+    does not flag the filesystem read and the broad ``except`` here cannot mask a
+    ``BlockingError`` raised on the loop.
+    """
+    from deerflow.config.agents_config import load_agent_config
+
+    try:
+        return load_agent_config(agent_name, user_id=user_id)
+    except Exception:
+        logger.warning("Could not load agent config for %r; using the default model for summarization", agent_name, exc_info=True)
+        return None
+
+
+async def _aresolve_thread_model_name(
+    model_name: str | None,
+    agent_name: str | None,
+    user_id: str | None,
+    app_config: AppConfig,
+) -> str | None:
+    """Resolve the model a thread should summarize with, mirroring lead resolution.
+
+    Precedence matches ``lead_agent._resolve_model_name``: an explicit request model
+    override (validated against configured models) wins, else the thread's custom-agent
+    configured model, else ``config.models[0]``. Manual ``/compact`` does not execute
+    the agent, so there is no live runtime carrying the selected model — the caller
+    (route / client) supplies it as ``model_name`` the same way a normal run submits
+    ``context.model_name``. The custom-agent config read happens only when no request
+    model was supplied, runs off the event loop, and passes the owning ``user_id`` so
+    the per-user agent directory resolves.
+    """
+    default = app_config.models[0].name if getattr(app_config, "models", None) else None
+    candidate = model_name
+    if not candidate and agent_name:
+        agent_config = await asyncio.to_thread(_safe_load_agent_config, agent_name, user_id)
+        if agent_config and agent_config.model:
+            candidate = agent_config.model
+    if candidate and app_config.get_model_config(candidate):
+        return candidate
+    return default
+
+
 async def compact_thread_context(
     accessor: CheckpointStateAccessor,
     thread_id: str,
@@ -53,11 +104,13 @@ async def compact_thread_context(
     force: bool = True,
     user_id: str | None = None,
     agent_name: str | None = None,
+    model_name: str | None = None,
     app_config: AppConfig | None = None,
 ) -> ThreadCompactionResult:
     """Summarize old messages in a thread and write a compacted checkpoint."""
     resolved_app_config = app_config or get_app_config()
-    middleware = _create_compaction_middleware(app_config=resolved_app_config, keep=keep)
+    run_model_name = await _aresolve_thread_model_name(model_name, agent_name, user_id, resolved_app_config)
+    middleware = _create_compaction_middleware(app_config=resolved_app_config, keep=keep, run_model_name=run_model_name)
 
     read_config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
     snapshot = await accessor.aget(read_config)
@@ -80,7 +133,18 @@ async def compact_thread_context(
     if agent_name:
         runtime_context["agent_name"] = agent_name
     runtime = SimpleNamespace(context=runtime_context)
-    result = await middleware.acompact_state(state, runtime, force=force)  # type: ignore[arg-type]
+    try:
+        # ``raise_on_failure`` is independent of ``force``: a manual caller always wants
+        # a generation failure surfaced (even a force=False call that met the threshold),
+        # so it must not collapse into the force=False "nothing to compact" branch below.
+        result = await middleware.acompact_state(state, runtime, force=force, raise_on_failure=True)  # type: ignore[arg-type]
+    except SummaryGenerationError as exc:
+        # A compressible thread whose summary LLM failed (after the run-model fallback)
+        # is a real failure, distinct from "nothing to compact". Route it to the
+        # already-consumed ContextCompactionFailed path (HTTP 500 -> frontend error
+        # toast) instead of a compacted=False result that reads as "does not need
+        # compaction".
+        raise ContextCompactionFailed("summary generation failed") from exc
     if result is None:
         return ThreadCompactionResult(thread_id=thread_id, compacted=False, reason="not_enough_messages")
 

backend/packages/harness/deerflow/runtime/events/store/base.py

@@ -51,6 +51,26 @@ class RunEventStore(abc.ABC):
         Returns complete records with seq assigned.
         """
 
+    @abc.abstractmethod
+    async def put_if_absent(
+        self,
+        *,
+        thread_id: str,
+        run_id: str,
+        event_type: str,
+        category: str,
+        content: str | dict = "",
+        metadata: dict | None = None,
+        created_at: str | None = None,
+    ) -> tuple[dict, bool]:
+        """Write one event unless this run already has the same event type.
+
+        The check and write must be serialized with ordinary writers for the
+        thread. Returns ``(record, created)``. This is the durability primitive
+        used by terminal run receipts, whose recovery path may safely retry
+        after a worker crash.
+        """
+
     @abc.abstractmethod
     async def list_messages(
         self,

backend/packages/harness/deerflow/runtime/events/store/db.py

@@ -194,6 +194,59 @@ class DbRunEventStore(RunEventStore):
                         rows.append(row)
                 return [self._row_to_dict(r) for r in rows]
 
+    async def put_if_absent(
+        self,
+        *,
+        thread_id,
+        run_id,
+        event_type,
+        category,
+        content="",
+        metadata=None,
+        created_at=None,
+    ):
+        """Idempotently insert a run-scoped singleton event.
+
+        ``_max_seq_for_thread`` takes the same PostgreSQL advisory lock used by
+        every normal writer (and the in-process lock covers SQLite), so the
+        existence check cannot race another ``put_if_absent`` or journal write.
+        Terminal delivery receipts use this method on both the worker and
+        recovery paths; ordinary event types remain append-only.
+        """
+        content, metadata = self._truncate_trace(category, content, metadata)
+        db_content, metadata = self._content_to_db(content, metadata)
+        user_id = self._user_id_from_context()
+        async with self._get_write_lock(thread_id):
+            async with self._sf() as session:
+                async with session.begin():
+                    max_seq = await self._max_seq_for_thread(session, thread_id)
+                    stmt = (
+                        select(RunEventRow)
+                        .where(
+                            RunEventRow.thread_id == thread_id,
+                            RunEventRow.run_id == run_id,
+                            RunEventRow.event_type == event_type,
+                        )
+                        .order_by(RunEventRow.seq.asc())
+                        .limit(1)
+                    )
+                    existing = await session.scalar(stmt)
+                    if existing is not None:
+                        return self._row_to_dict(existing), False
+                    row = RunEventRow(
+                        thread_id=thread_id,
+                        run_id=run_id,
+                        user_id=user_id,
+                        event_type=event_type,
+                        category=category,
+                        content=db_content,
+                        event_metadata=metadata,
+                        seq=(max_seq or 0) + 1,
+                        created_at=datetime.fromisoformat(created_at) if created_at else datetime.now(UTC),
+                    )
+                    session.add(row)
+                return self._row_to_dict(row), True
+
     async def list_messages(
         self,
         thread_id,

backend/packages/harness/deerflow/runtime/events/store/jsonl.py

@@ -178,6 +178,36 @@ class JsonlRunEventStore(RunEventStore):
             results.extend(records)
         return results
 
+    async def put_if_absent(
+        self,
+        *,
+        thread_id,
+        run_id,
+        event_type,
+        category,
+        content="",
+        metadata=None,
+        created_at=None,
+    ):
+        async with self._get_write_lock(thread_id):
+            existing = await asyncio.to_thread(self._read_run_events, thread_id, run_id)
+            for event in existing:
+                if event.get("event_type") == event_type:
+                    return event, False
+            await self._ensure_seq_loaded(thread_id)
+            record = {
+                "thread_id": thread_id,
+                "run_id": run_id,
+                "event_type": event_type,
+                "category": category,
+                "content": content,
+                "metadata": metadata or {},
+                "seq": self._next_seq(thread_id),
+                "created_at": created_at or datetime.now(UTC).isoformat(),
+            }
+            await asyncio.to_thread(self._write_record, record)
+            return record, True
+
     async def _write_batch_async(self, thread_id: str, batch: list[dict[str, Any]]) -> list[dict[str, Any]]:
         async with self._get_write_lock(thread_id):
             await self._ensure_seq_loaded(thread_id)

backend/packages/harness/deerflow/runtime/events/store/memory.py

@@ -93,6 +93,35 @@ class MemoryRunEventStore(RunEventStore):
             results.append(record)
         return results
 
+    async def put_if_absent(
+        self,
+        *,
+        thread_id,
+        run_id,
+        event_type,
+        category,
+        content="",
+        metadata=None,
+        created_at=None,
+    ):
+        # No await occurs between the lookup and append, so this is atomic for
+        # the backend's documented single-event-loop concurrency model.
+        for event in self._events_by_run.get(thread_id, {}).get(run_id, []):
+            if event["event_type"] == event_type:
+                return event, False
+        return (
+            self._put_one(
+                thread_id=thread_id,
+                run_id=run_id,
+                event_type=event_type,
+                category=category,
+                content=content,
+                metadata=metadata,
+                created_at=created_at,
+            ),
+            True,
+        )
+
     async def list_messages(self, thread_id, *, limit=50, before_seq=None, after_seq=None, user_id: str | None | _AutoSentinel = AUTO):
         # ``messages`` is messages-only and seq-sorted, so the seq window is a
         # contiguous slice located with bisect (O(log m)) rather than a full scan.

backend/packages/harness/deerflow/runtime/journal.py

@@ -91,7 +91,7 @@ def build_branch_history_seed_events(
     messages: Sequence[Any],
     *,
     thread_id: str,
-    run_id: str,
+    run_id_prefix: str,
     parent_thread_id: str,
 ) -> list[dict]:
     """Serialize a branch checkpoint's messages into run-event message rows.
@@ -104,6 +104,18 @@ def build_branch_history_seed_events(
     checkpoint snapshot the branch was created from keeps the feed
     consistent with what the branch actually contains.
 
+    Rows are grouped into one synthetic run per inherited turn
+    (``{run_id_prefix}-{n}``), a new turn starting at every persisted human
+    message — the same boundary a real run has, since a run begins with a
+    human input (including the allowlisted hidden ``ask_clarification``
+    reply, which resumes as its own run). ``run_id`` is a *turn* identity to
+    the feed's consumers, not merely a provenance tag: regenerating the last
+    inherited answer resolves that row's ``run_id`` as the superseded source
+    (``_find_target_run_id``) and ``GET /messages/page`` then drops **every**
+    row carrying it. One shared id for the whole seed therefore deleted the
+    complete inherited history on the branch's first regenerate (#4458); one
+    id per turn confines the drop to the turn actually regenerated.
+
     Mirrors RunJournal's message-event contract so seeded rows are
     indistinguishable from journaled ones except by the ``branch_seed``
     marker: same event types, ``category="message"``, ``content=
@@ -125,6 +137,8 @@ def build_branch_history_seed_events(
     events: list[dict] = []
     created_at = datetime.now(UTC).isoformat()
     seed_metadata = {"branch_seed": True, "branch_parent_thread_id": parent_thread_id}
+    # Messages ahead of the first human turn (none in practice) stay in turn 0.
+    turn_index = 0
     for raw_message in messages:
         message = _coerce_seed_message(raw_message)
         if not isinstance(message, BaseMessage):
@@ -132,6 +146,7 @@ def build_branch_history_seed_events(
         if isinstance(message, HumanMessage):
             if not _should_persist_human_input_message(message):
                 continue
+            turn_index += 1
             event_type = "llm.human.input"
             content = restore_original_human_message(message).model_dump()
             metadata: dict[str, Any] = {"caller": "lead_agent", **seed_metadata}
@@ -149,7 +164,7 @@ def build_branch_history_seed_events(
         events.append(
             {
                 "thread_id": thread_id,
-                "run_id": run_id,
+                "run_id": f"{run_id_prefix}-{turn_index}",
                 "event_type": event_type,
                 "category": "message",
                 "content": content,
@@ -163,6 +178,12 @@ def build_branch_history_seed_events(
 class RunJournal(BaseCallbackHandler):
     """LangChain callback handler that captures events to RunEventStore."""
 
+    # Every callback only updates in-memory run state or schedules async IO.
+    # Keeping callbacks on the run's event-loop thread serializes mutations
+    # from parallel tool calls and prevents cancelled executor callbacks from
+    # racing terminal delivery recording and flush.
+    run_inline = True
+
     def __init__(
         self,
         run_id: str,
@@ -227,6 +248,11 @@ class RunJournal(BaseCallbackHandler):
         self._current_run_tool_call_names: dict[str, str] = {}
         self._persisted_tool_message_identities: set[str] = set()
 
+        # Artifact-production tracking for the terminal run.delivery event
+        # (#4272 slice 1). Deduped by (path, tool_name); insertion order kept.
+        self._produced_artifacts: list[tuple[str, str | None]] = []
+        self._produced_artifact_keys: set[tuple[str, str | None]] = set()
+
     # -- Lifecycle callbacks --
 
     @staticmethod
@@ -474,11 +500,26 @@ class RunJournal(BaseCallbackHandler):
             elif isinstance(output, Command):
                 cmd = cast(Command, output)
                 messages = cmd.update.get("messages", [])
+                # A non-empty ``artifacts`` update is only produced on the
+                # success path (e.g. present_files returns an error ToolMessage
+                # without touching state when validation fails), so its
+                # presence is the artifact-production signal (#4272 slice 1).
+                artifacts = cmd.update.get("artifacts")
+                artifact_tool_names: set[str] = set()
                 for message in messages:
                     if isinstance(message, BaseMessage):
                         self._persist_tool_result_message(message)
+                        if artifacts and isinstance(message, ToolMessage):
+                            tool_call_id = getattr(message, "tool_call_id", None)
+                            if isinstance(tool_call_id, str):
+                                tool_name = self._current_run_tool_call_names.get(tool_call_id)
+                                if tool_name:
+                                    artifact_tool_names.add(tool_name)
                     else:
                         logger.warning(f"on_tool_end {run_id}: command update message is not BaseMessage: {type(message)}")
+                if artifacts:
+                    artifact_tool_name = next(iter(artifact_tool_names)) if len(artifact_tool_names) == 1 else None
+                    self._record_produced_artifacts(artifacts, artifact_tool_name)
             else:
                 logger.warning(f"on_tool_end {run_id}: output is not ToolMessage: {type(output)}")
         finally:
@@ -769,6 +810,44 @@ class RunJournal(BaseCallbackHandler):
         )
         self._memory_context_recorded = True
 
+    def _record_produced_artifacts(self, artifacts: Any, tool_name: str | None) -> None:
+        """Accumulate produced artifact paths, deduped by (path, tool_name)."""
+        if not isinstance(artifacts, list):
+            return
+        for path in artifacts:
+            if not isinstance(path, str) or not path:
+                continue
+            key = (path, tool_name)
+            if key not in self._produced_artifact_keys:
+                self._produced_artifact_keys.add(key)
+                self._produced_artifacts.append(key)
+
+    def get_delivery_content(self) -> dict[str, Any]:
+        """Return the terminal delivery fact accumulated for this run.
+
+        This is a fact record, not a verdict: runs that produced no artifacts
+        emit ``presented: 0``.
+        """
+        by_tool: dict[str, list[str]] = {}
+        paths: list[str] = []
+        for path, tool_name in self._produced_artifacts:
+            paths.append(path)
+            if tool_name:
+                by_tool.setdefault(tool_name, []).append(path)
+        return {"presented": len(paths), "paths": paths, "by_tool": by_tool}
+
+    def record_delivery(self) -> None:
+        """Buffer the terminal ``run.delivery`` event for this run (#4272 slice 1).
+
+        Kept for direct journal users. The worker uses the event store's
+        idempotent singleton write so crash recovery can safely backfill it.
+        """
+        self._put(
+            event_type="run.delivery",
+            category="outputs",
+            content=self.get_delivery_content(),
+        )
+
     async def flush(self) -> None:
         """Force flush remaining buffer. Called in worker's finally block."""
         if self._pending_flush_tasks:

backend/packages/harness/deerflow/runtime/runs/__init__.py

@@ -1,7 +1,7 @@
 """Run lifecycle management for LangGraph Platform API compatibility."""
 
 from .manager import ORPHAN_RECOVERY_STOP_REASON, STARTUP_ORPHAN_RECOVERY_ERROR, CancelOutcome, ConflictError, RunManager, RunRecord, UnsupportedStrategyError
-from .schemas import DisconnectMode, RunStatus
+from .schemas import DisconnectMode, RunStatus, ThreadOperationKind
 from .worker import RunContext, run_agent
 
 __all__ = [
@@ -13,6 +13,7 @@ __all__ = [
     "RunManager",
     "RunRecord",
     "RunStatus",
+    "ThreadOperationKind",
     "STARTUP_ORPHAN_RECOVERY_ERROR",
     "UnsupportedStrategyError",
     "run_agent",

backend/packages/harness/deerflow/runtime/runs/manager.py

@@ -7,7 +7,8 @@ import logging
 import socket
 import sqlite3
 import uuid
-from collections.abc import Awaitable, Callable
+from collections.abc import AsyncIterator, Awaitable, Callable
+from contextlib import asynccontextmanager
 from dataclasses import dataclass, field
 from datetime import UTC, datetime, timedelta
 from enum import StrEnum
@@ -19,10 +20,11 @@ from deerflow.runtime.user_context import AUTO, _AutoSentinel, resolve_user_id
 from deerflow.utils.time import is_lease_expired
 from deerflow.utils.time import now_iso as _now_iso
 
-from .schemas import DisconnectMode, RunStatus
+from .schemas import DisconnectMode, RunStatus, ThreadOperationKind
 
 if TYPE_CHECKING:
     from deerflow.config.run_ownership_config import RunOwnershipConfig
+    from deerflow.runtime.events.store.base import RunEventStore
     from deerflow.runtime.runs.store.base import RunStore
 
 logger = logging.getLogger(__name__)
@@ -158,6 +160,7 @@ class RunRecord:
     assistant_id: str | None
     status: RunStatus
     on_disconnect: DisconnectMode
+    operation_kind: ThreadOperationKind = ThreadOperationKind.run
     multitask_strategy: str = "reject"
     metadata: dict = field(default_factory=dict)
     kwargs: dict = field(default_factory=dict)
@@ -165,6 +168,8 @@ class RunRecord:
     created_at: str = ""
     updated_at: str = ""
     task: asyncio.Task | None = field(default=None, repr=False)
+    # Serializes startup if an admitted run is ever handed to more than one worker path.
+    start_lock: asyncio.Lock = field(default_factory=asyncio.Lock, repr=False)
     abort_event: asyncio.Event = field(default_factory=asyncio.Event, repr=False)
     abort_action: str = "interrupt"
     error: str | None = None
@@ -185,9 +190,24 @@ class RunRecord:
     finalizing: bool = False
     owner_worker_id: str | None = None
     lease_expires_at: str | None = None
+    # Process-local fencing signal. Once set, this worker must not perform
+    # further durable run/thread finalization because its lease ownership is
+    # either known to be lost or could not be confirmed before expiry.
+    ownership_lost: bool = False
     stop_reason: str | None = None
 
 
+class RunStartOutcome(StrEnum):
+    """Result of the pending-to-running startup barrier."""
+
+    started = "started"
+    cancelled = "cancelled"
+
+
+class RunStartupError(RuntimeError):
+    """Raised when durable startup cannot be resolved safely."""
+
+
 OrphanRecoveryCallback = Callable[[list[RunRecord]], Awaitable[None]]
 
 
@@ -206,6 +226,7 @@ class RunManager:
         persistence_retry_policy: PersistenceRetryPolicy | None = None,
         worker_id: str | None = None,
         run_ownership_config: RunOwnershipConfig | None = None,
+        event_store: RunEventStore | None = None,
         on_orphans_recovered: OrphanRecoveryCallback | None = None,
     ) -> None:
         self._runs: dict[str, RunRecord] = {}
@@ -219,6 +240,7 @@ class RunManager:
         self._persistence_retry_policy = persistence_retry_policy or PersistenceRetryPolicy()
         self._worker_id = worker_id or _generate_worker_id()
         self._run_ownership_config = run_ownership_config
+        self._event_store = event_store
         self._on_orphans_recovered = on_orphans_recovered
         self._heartbeat_task: asyncio.Task | None = None
         self._heartbeat_stop: asyncio.Event | None = None
@@ -260,6 +282,7 @@ class RunManager:
             "thread_id": record.thread_id,
             "assistant_id": record.assistant_id,
             "status": record.status.value,
+            "operation_kind": record.operation_kind.value,
             "multitask_strategy": record.multitask_strategy,
             "metadata": record.metadata or {},
             "kwargs": record.kwargs or {},
@@ -346,6 +369,13 @@ class RunManager:
 
     async def _persist_status(self, record: RunRecord, status: RunStatus, *, error: str | None = None, stop_reason: str | None = None) -> bool:
         """Best-effort persist a status transition to the backing store."""
+        if record.ownership_lost:
+            logger.warning(
+                "Skipped status update to %s for run %s after lease ownership was lost",
+                status.value,
+                record.run_id,
+            )
+            return False
         if self._store is None:
             return True
         row_recovery_payload = self._store_put_payload(record, error=error, stop_reason=stop_reason)
@@ -365,12 +395,25 @@ class RunManager:
                 existing = await self._store.get(record.run_id)
                 if existing is not None:
                     existing_status = existing.get("status")
+                    if existing_status == status.value:
+                        logger.info(
+                            "Run %s status update to %s was already persisted",
+                            record.run_id,
+                            status.value,
+                        )
+                        return True
                     if existing_status == "error":
                         logger.warning(
                             "Run %s status update to %s skipped: store row already at error (peer takeover)",
                             record.run_id,
                             status.value,
                         )
+                        if self.heartbeat_enabled and not record.store_only:
+                            await self._mark_ownership_lost(
+                                record,
+                                reason="A peer terminalized the run before this worker could persist its outcome.",
+                                require_active=False,
+                            )
                     else:
                         logger.info(
                             "Run %s status update to %s skipped: store row already at %s (local cancel/completion race)",
@@ -398,6 +441,7 @@ class RunManager:
             assistant_id=row.get("assistant_id"),
             status=RunStatus(row.get("status") or RunStatus.pending.value),
             on_disconnect=DisconnectMode(row.get("on_disconnect") or DisconnectMode.cancel.value),
+            operation_kind=ThreadOperationKind(row.get("operation_kind") or ThreadOperationKind.run.value),
             multitask_strategy=row.get("multitask_strategy") or "reject",
             metadata=row.get("metadata") or {},
             kwargs=row.get("kwargs") or {},
@@ -426,8 +470,12 @@ class RunManager:
     async def update_run_completion(self, run_id: str, **kwargs) -> None:
         """Persist token usage and completion data to the backing store."""
         row_recovery_payload: dict[str, Any] | None = None
+        record: RunRecord | None = None
         async with self._lock:
             record = self._runs.get(run_id)
+            if record is not None and record.ownership_lost:
+                logger.warning("Skipped completion persistence for run %s after lease ownership was lost", run_id)
+                return
             if record is not None:
                 for key, value in kwargs.items():
                     if key == "status":
@@ -445,6 +493,22 @@ class RunManager:
                 lambda: self._store.update_run_completion(run_id, **kwargs),
             )
             if updated is False:
+                existing = await self._store.get(run_id)
+                requested_status = kwargs.get("status")
+                if existing is not None and existing.get("status") != requested_status:
+                    existing_status = existing.get("status")
+                    logger.warning(
+                        "Run completion update for %s skipped because store row is already at %s",
+                        run_id,
+                        existing_status,
+                    )
+                    if existing_status == "error" and record is not None and self.heartbeat_enabled:
+                        await self._mark_ownership_lost(
+                            record,
+                            reason="A peer terminalized the run before completion data was persisted.",
+                            require_active=False,
+                        )
+                    return
                 if row_recovery_payload is None:
                     logger.warning("Failed to recreate missing run %s for completion persistence", run_id)
                     return
@@ -466,7 +530,7 @@ class RunManager:
         async with self._lock:
             record = self._runs.get(run_id)
             if record is not None:
-                should_persist = record.status == RunStatus.running
+                should_persist = record.status == RunStatus.running and not record.ownership_lost
             if record is not None and should_persist:
                 for key, value in kwargs.items():
                     if hasattr(record, key) and value is not None:
@@ -493,7 +557,7 @@ class RunManager:
 
         Note: this method assumes no active run exists for the thread. It
         persists via ``store.put`` (upsert) rather than the atomic
-        ``create_run_atomic`` primitive, so a concurrent insert for the
+        ``create_thread_operation_atomic`` primitive, so a concurrent insert for the
         same thread will hit the partial unique index and surface as a
         raw ``IntegrityError`` instead of a ``ConflictError``. Production
         callers should use :meth:`create_or_reject`.
@@ -586,7 +650,7 @@ class RunManager:
             limit: Maximum number of runs to return.
         """
         async with self._lock:
-            memory_records = self._thread_records_locked(thread_id)
+            memory_records = [record for record in self._thread_records_locked(thread_id) if record.operation_kind == ThreadOperationKind.run]
         if self._store is None:
             return sorted(memory_records, key=lambda r: r.created_at, reverse=True)[:limit]
         records_by_id = {record.run_id: record for record in memory_records}
@@ -626,7 +690,7 @@ class RunManager:
         """
         resolved_user_id = resolve_user_id(user_id, method_name="RunManager.list_successful_regenerate_sources")
         async with self._lock:
-            memory_records = [record for record in self._thread_records_locked(thread_id) if resolved_user_id is None or record.user_id == resolved_user_id]
+            memory_records = [record for record in self._thread_records_locked(thread_id) if record.operation_kind == ThreadOperationKind.run and (resolved_user_id is None or record.user_id == resolved_user_id)]
 
         sources = set(await self._store.list_successful_regenerate_sources(thread_id, user_id=resolved_user_id)) if self._store is not None else set()
         # _thread_records_locked preserves the insertion order of the thread
@@ -642,6 +706,69 @@ class RunManager:
                 sources.add(source)
         return sources
 
+    async def try_start(self, run_id: str) -> RunStartOutcome:
+        """Transition an uncancelled pending run to running before building the agent."""
+        async with self._lock:
+            record = self._runs.get(run_id)
+        if record is None:
+            raise RunStartupError(f"Cannot start unknown run {run_id}")
+
+        async with record.start_lock:
+            async with self._lock:
+                if record.abort_event.is_set() or record.status != RunStatus.pending:
+                    return RunStartOutcome.cancelled
+
+            if self._store is not None:
+                try:
+                    updated = await self._call_store_with_retry(
+                        "start_run",
+                        run_id,
+                        lambda: self._store.start_run(run_id),
+                    )
+                except Exception as exc:
+                    raise RunStartupError(f"Failed to start run {run_id}: {exc}") from exc
+                if updated is False:
+                    async with self._lock:
+                        if record.status == RunStatus.pending:
+                            record.status = RunStatus.interrupted
+                            record.abort_event.set()
+                            record.updated_at = _now_iso()
+                    return RunStartOutcome.cancelled
+
+            async with self._lock:
+                if record.abort_event.is_set() or record.status != RunStatus.pending:
+                    restore_status = record.status
+                    restore_error = record.error
+                    restore_stop_reason = record.stop_reason
+                else:
+                    record.status = RunStatus.running
+                    record.updated_at = _now_iso()
+                    logger.info("Run %s -> %s", run_id, RunStatus.running.value)
+                    return RunStartOutcome.started
+
+            if self._store is not None:
+                await self._persist_status(
+                    record,
+                    restore_status,
+                    error=restore_error,
+                    stop_reason=restore_stop_reason,
+                )
+            return RunStartOutcome.cancelled
+
+    async def fail_start_if_pending(self, run_id: str, *, error: str) -> bool:
+        """Mark an admitted run as failed if its worker task could not be attached."""
+        async with self._lock:
+            record = self._runs.get(run_id)
+            if record is None or record.status != RunStatus.pending:
+                return False
+            record.status = RunStatus.error
+            record.error = error
+            record.abort_event.set()
+            record.updated_at = _now_iso()
+
+        await self._persist_status(record, RunStatus.error, error=error)
+        return True
+
     async def get_many_by_thread(
         self,
         thread_id: str,
@@ -654,7 +781,9 @@ class RunManager:
             return {}
         resolved_user_id = resolve_user_id(user_id, method_name="RunManager.get_many_by_thread")
         async with self._lock:
-            records_by_id = {record.run_id: record for record in self._thread_records_locked(thread_id) if record.run_id in run_ids and (resolved_user_id is None or record.user_id == resolved_user_id)}
+            records_by_id = {
+                record.run_id: record for record in self._thread_records_locked(thread_id) if record.operation_kind == ThreadOperationKind.run and record.run_id in run_ids and (resolved_user_id is None or record.user_id == resolved_user_id)
+            }
         if self._store is None:
             return records_by_id
 
@@ -675,22 +804,86 @@ class RunManager:
                 logger.warning("Failed to map store row for run %s", run_id, exc_info=True)
         return records_by_id
 
-    async def set_status(self, run_id: str, status: RunStatus, *, error: str | None = None, stop_reason: str | None = None) -> None:
+    async def set_status(
+        self,
+        run_id: str,
+        status: RunStatus,
+        *,
+        error: str | None = None,
+        stop_reason: str | None = None,
+        persist: bool = True,
+    ) -> None:
         """Transition a run to a new status."""
         async with self._lock:
             record = self._runs.get(run_id)
             if record is None:
                 logger.warning("set_status called for unknown run %s", run_id)
                 return
+            if record.ownership_lost:
+                logger.warning(
+                    "Skipped local status transition to %s for run %s after lease ownership was lost",
+                    status.value,
+                    run_id,
+                )
+                return
             record.status = status
             record.updated_at = _now_iso()
             if error is not None:
                 record.error = error
             if stop_reason is not None:
                 record.stop_reason = stop_reason
-        await self._persist_status(record, status, error=error, stop_reason=stop_reason)
+        if persist:
+            persisted = await self._persist_status(record, status, error=error, stop_reason=stop_reason)
+            if not persisted and self.heartbeat_enabled and status == RunStatus.success and not record.ownership_lost:
+                await self._mark_ownership_lost(
+                    record,
+                    reason="Successful completion could not be confirmed in the durable run store.",
+                    require_active=False,
+                )
+        if record.ownership_lost:
+            return
         logger.info("Run %s -> %s", run_id, status.value)
 
+    async def persist_current_status(self, run_id: str) -> bool:
+        """Persist the status already staged on the in-memory run record."""
+        async with self._lock:
+            record = self._runs.get(run_id)
+            if record is None:
+                logger.warning("persist_current_status called for unknown run %s", run_id)
+                return False
+            status = record.status
+            error = record.error
+            stop_reason = record.stop_reason
+        persisted = await self._persist_status(record, status, error=error, stop_reason=stop_reason)
+        if not persisted and self.heartbeat_enabled and status == RunStatus.success and not record.ownership_lost:
+            await self._mark_ownership_lost(
+                record,
+                reason="Successful completion could not be confirmed in the durable run store.",
+                require_active=False,
+            )
+        return persisted
+
+    async def _ensure_delivery_receipt(self, record: RunRecord) -> bool:
+        """Idempotently persist a zero-delivery receipt during recovery."""
+        if self._event_store is None:
+            return True
+        try:
+            await self._event_store.put_if_absent(
+                thread_id=record.thread_id,
+                run_id=record.run_id,
+                event_type="run.delivery",
+                category="outputs",
+                content={"presented": 0, "paths": [], "by_tool": {}},
+            )
+            return True
+        except Exception:
+            logger.warning(
+                "Failed to backfill delivery receipt for recovered run %s; preserving its terminal status",
+                record.run_id,
+                exc_info=True,
+            )
+            return False
+
     async def set_finalizing(self, run_id: str, finalizing: bool) -> None:
         """Mark whether a run is performing post-cancel cleanup."""
         async with self._lock:
@@ -707,6 +900,7 @@ class RunManager:
         run_id: str,
         *,
         poll_interval: float = 0.01,
+        abort_event: asyncio.Event | None = None,
     ) -> None:
         """Wait until older same-thread runs have finished post-cancel cleanup."""
         while True:
@@ -723,7 +917,14 @@ class RunManager:
                 if not found_current or not prior_finalizing:
                     return
 
-            await asyncio.sleep(poll_interval)
+            if abort_event is None:
+                await asyncio.sleep(poll_interval)
+                continue
+            try:
+                await asyncio.wait_for(abort_event.wait(), timeout=poll_interval)
+            except TimeoutError:
+                continue
+            return
 
     async def has_later_run(self, thread_id: str, run_id: str) -> bool:
         """Return whether a newer in-memory run has been admitted for the thread."""
@@ -818,7 +1019,7 @@ class RunManager:
                 record.abort_event.set()
                 task_active = record.task is not None and not record.task.done()
                 record.finalizing = task_active
-                if task_active:
+                if task_active and record.status == RunStatus.running:
                     record.task.cancel()
                 record.status = RunStatus.interrupted
                 record.updated_at = _now_iso()
@@ -943,6 +1144,77 @@ class RunManager:
         multitask_strategy: str = "reject",
         model_name: str | None = None,
         user_id: str | None = None,
+    ) -> RunRecord:
+        """Atomically admit a normal agent run for a thread."""
+        return await self._admit_thread_operation(
+            thread_id,
+            assistant_id,
+            operation_kind=ThreadOperationKind.run,
+            on_disconnect=on_disconnect,
+            metadata=metadata,
+            kwargs=kwargs,
+            multitask_strategy=multitask_strategy,
+            model_name=model_name,
+            user_id=user_id,
+        )
+
+    async def _close_cancelled_admission(self, record: RunRecord) -> None:
+        """Terminalize an unseen replacement and confirm its durable state."""
+        await self.cancel(record.run_id)
+        if self._store is None:
+            return
+
+        stored = await self._call_store_with_retry(
+            "verify cancelled admission",
+            record.run_id,
+            lambda: self._store.get(record.run_id, user_id=record.user_id),
+        )
+        active_statuses = (RunStatus.pending.value, RunStatus.running.value)
+        if stored is not None and stored.get("status") in active_statuses:
+            # `_persist_status` is deliberately best-effort. This compensation
+            # path needs a strict second CAS attempt because the caller never
+            # receives the record and no worker can attach after it returns.
+            # A peer terminal transition wins the CAS and is preserved below.
+            await self._call_store_with_retry(
+                "terminalize cancelled admission",
+                record.run_id,
+                lambda: self._store.update_status(record.run_id, RunStatus.interrupted.value),
+            )
+            stored = await self._call_store_with_retry(
+                "verify terminal cancelled admission",
+                record.run_id,
+                lambda: self._store.get(record.run_id, user_id=record.user_id),
+            )
+            if stored is not None and stored.get("status") in active_statuses:
+                raise RuntimeError(f"Cancelled admission {record.run_id} remains active in the run store")
+
+        if stored is None:
+            async with self._lock:
+                if self._runs.get(record.run_id) is record:
+                    self._runs.pop(record.run_id, None)
+                    self._unindex_run_locked(record.run_id, record.thread_id)
+            return
+
+        stored_status = RunStatus(stored.get("status") or RunStatus.pending.value)
+        async with self._lock:
+            if self._runs.get(record.run_id) is record:
+                record.status = stored_status
+                record.error = stored.get("error")
+                record.stop_reason = stored.get("stop_reason")
+                record.updated_at = _now_iso()
+
+    async def _admit_thread_operation(
+        self,
+        thread_id: str,
+        assistant_id: str | None = None,
+        *,
+        operation_kind: ThreadOperationKind,
+        on_disconnect: DisconnectMode = DisconnectMode.cancel,
+        metadata: dict | None = None,
+        kwargs: dict | None = None,
+        multitask_strategy: str = "reject",
+        model_name: str | None = None,
+        user_id: str | None = None,
     ) -> RunRecord:
         """Atomically check for inflight runs and create a new one.
 
@@ -975,6 +1247,7 @@ class RunManager:
             assistant_id=assistant_id,
             status=RunStatus.pending,
             on_disconnect=on_disconnect,
+            operation_kind=operation_kind,
             multitask_strategy=multitask_strategy,
             metadata=metadata or {},
             kwargs=kwargs or {},
@@ -991,6 +1264,9 @@ class RunManager:
             #    store's partial unique index below).
             local_inflight = [r for r in self._thread_records_locked(thread_id) if r.status in (RunStatus.pending, RunStatus.running) or r.finalizing]
 
+            if multitask_strategy in ("interrupt", "rollback") and any(record.operation_kind != ThreadOperationKind.run for record in local_inflight):
+                raise ConflictError(f"Thread {thread_id} has an active checkpoint write")
+
             if multitask_strategy == "reject" and local_inflight:
                 raise ConflictError(f"Thread {thread_id} already has an active run")
 
@@ -1008,13 +1284,14 @@ class RunManager:
                 if multitask_strategy == "reject":
                     try:
                         await self._call_store_with_retry(
-                            "create_run_atomic",
+                            "create_thread_operation_atomic",
                             run_id,
-                            lambda: self._store.create_run_atomic(
+                            lambda: self._store.create_thread_operation_atomic(
                                 run_id=run_id,
                                 thread_id=thread_id,
                                 owner_worker_id=self._worker_id,
                                 lease_expires_at=lease_expires_at,
+                                operation_kind=operation_kind.value,
                                 multitask_strategy="reject",
                                 assistant_id=assistant_id,
                                 user_id=user_id,
@@ -1039,13 +1316,14 @@ class RunManager:
                     for attempt in range(max_retries):
                         try:
                             await self._call_store_with_retry(
-                                "create_run_atomic",
+                                "create_thread_operation_atomic",
                                 run_id,
-                                lambda: self._store.create_run_atomic(
+                                lambda: self._store.create_thread_operation_atomic(
                                     run_id=run_id,
                                     thread_id=thread_id,
                                     owner_worker_id=self._worker_id,
                                     lease_expires_at=lease_expires_at,
+                                    operation_kind=operation_kind.value,
                                     multitask_strategy=multitask_strategy,
                                     assistant_id=assistant_id,
                                     user_id=user_id,
@@ -1068,7 +1346,7 @@ class RunManager:
                                 # worker won the race for this thread.
                                 raise ConflictError(f"Thread {thread_id} already has an active run") from exc
                             raise
-                    # ``create_run_atomic`` already marked any claimed store
+                    # ``create_thread_operation_atomic`` already marked any claimed store
                     # rows as interrupted in the same transaction; no extra
                     # store write is needed for them.
 
@@ -1093,13 +1371,92 @@ class RunManager:
                     interrupted_records.append(r)
 
         # Outside the lock: persist interrupted status for locally-cancelled
-        # runs. Store-side claimed rows are already finalised.
-        for interrupted_record in interrupted_records:
-            await self._persist_status(interrupted_record, RunStatus.interrupted)
+        # runs. Store-side claimed rows are already finalised. Cancellation at
+        # this point happens after the replacement was admitted, so close that
+        # new run before propagating cancellation to the caller.
+        try:
+            for interrupted_record in interrupted_records:
+                await self._persist_status(interrupted_record, RunStatus.interrupted)
+        except asyncio.CancelledError:
+            cleanup = asyncio.create_task(self._close_cancelled_admission(record))
+            cleanup.set_name(f"deerflow-close-cancelled-admission-{record.run_id}")
+            while not cleanup.done():
+                try:
+                    await asyncio.shield(cleanup)
+                except asyncio.CancelledError:
+                    pass
+                except Exception:
+                    break
+            try:
+                cleanup.result()
+            except asyncio.CancelledError:
+                logger.error("Cancelled admission cleanup task was itself cancelled for run %s", record.run_id)
+            except Exception:
+                logger.exception("Failed to close run %s after admission was cancelled", record.run_id)
+            raise
 
         logger.info("Run created: run_id=%s thread_id=%s", run_id, thread_id)
         return record
 
+    @asynccontextmanager
+    async def reserve_thread_operation(
+        self,
+        thread_id: str,
+        *,
+        kind: ThreadOperationKind,
+        user_id: str | None = None,
+    ) -> AsyncIterator[None]:
+        """Hold exclusive durable admission for a non-run thread operation.
+
+        The reservation is a short-lived pending row, so the same durable
+        uniqueness constraint used by ``create_or_reject`` closes both sides of
+        the race across Gateway workers.
+        """
+        if kind == ThreadOperationKind.run:
+            raise ValueError("Normal runs must be admitted with create_or_reject()")
+        record = await self._admit_thread_operation(
+            thread_id,
+            operation_kind=kind,
+            multitask_strategy="reject",
+            user_id=user_id,
+        )
+        try:
+            reservation_task = asyncio.current_task()
+            if reservation_task is None:
+                raise RuntimeError("Thread operation reservation requires an active asyncio task")
+            lease_lost = True
+            async with self._lock:
+                if self._runs.get(record.run_id) is record:
+                    record.task = reservation_task
+                    lease_lost = record.abort_event.is_set()
+            if lease_lost:
+                raise asyncio.CancelledError()
+            yield
+        except asyncio.CancelledError:
+            if record.abort_event.is_set():
+                raise ConflictError(f"Thread {thread_id} reservation lease was lost") from None
+            raise
+        finally:
+            try:
+                if self._store is not None:
+                    try:
+                        await self._call_store_with_retry(
+                            "release thread operation",
+                            record.run_id,
+                            lambda: self._store.delete_thread_operation(record.run_id, user_id=record.user_id),
+                        )
+                    except Exception:
+                        logger.warning(
+                            "Failed to release persisted thread operation %s; leaving it for orphan reconciliation",
+                            record.run_id,
+                            exc_info=True,
+                        )
+            finally:
+                async with self._lock:
+                    removed = self._runs.pop(record.run_id, None)
+                    if removed is not None:
+                        self._unindex_run_locked(record.run_id, removed.thread_id)
+
     async def reconcile_orphaned_inflight_runs(
         self,
         *,
@@ -1116,7 +1473,10 @@ class RunManager:
 
         Rows with a still-valid lease are skipped — they belong to another live
         worker. Rows with a NULL lease (pre-ownership data) are reclaimed as
-        well, matching the original single-worker recovery behaviour.
+        well, matching the original single-worker recovery behaviour. The
+        candidate scan is only an optimization: each row is claimed with a
+        lease-aware conditional update so a heartbeat renewal after the scan
+        always wins over reconciliation.
         """
         if self._store is None:
             return []
@@ -1170,7 +1530,14 @@ class RunManager:
             record.error = error
             record.stop_reason = stop_reason
             record.updated_at = now
-            recovered.append(record)
+            if record.operation_kind == ThreadOperationKind.run:
+                # The atomic takeover above must win before writing a zero-delivery
+                # receipt; otherwise a stale scan could race a heartbeat renewal and
+                # permanently overwrite a live run's later detailed receipt. The
+                # receipt remains best-effort, matching normal terminal delivery
+                # when its event store is unavailable.
+                await self._ensure_delivery_receipt(record)
+                recovered.append(record)
 
         if recovered:
             logger.warning("Recovered %d orphaned inflight run(s) as error", len(recovered))
@@ -1179,7 +1546,7 @@ class RunManager:
     async def has_inflight(self, thread_id: str) -> bool:
         """Return ``True`` if *thread_id* has a pending or running run."""
         async with self._lock:
-            return any(r.status in (RunStatus.pending, RunStatus.running) or r.finalizing for r in self._thread_records_locked(thread_id))
+            return any(r.operation_kind == ThreadOperationKind.run and (r.status in (RunStatus.pending, RunStatus.running) or r.finalizing) for r in self._thread_records_locked(thread_id))
 
     async def cleanup(self, run_id: str, *, delay: float = 300) -> None:
         """Remove a run record after an optional delay."""
@@ -1218,6 +1585,61 @@ class RunManager:
         """
         return self._run_ownership_config.grace_seconds if self._run_ownership_config else 10
 
+    @staticmethod
+    def _parse_lease_deadline(lease_expires_at: str | None) -> datetime | None:
+        """Parse the last durably confirmed lease expiry.
+
+        Missing or malformed deadlines are unsafe in heartbeat mode: the local
+        worker has no bounded interval during which it can prove ownership.
+        """
+        if lease_expires_at is None:
+            return None
+        try:
+            deadline = datetime.fromisoformat(lease_expires_at)
+        except (TypeError, ValueError):
+            return None
+        if deadline.tzinfo is None:
+            deadline = deadline.replace(tzinfo=UTC)
+        return deadline
+
+    async def _mark_ownership_lost(
+        self,
+        record: RunRecord,
+        *,
+        reason: str,
+        require_active: bool = True,
+    ) -> bool:
+        """Fence one local run and cancel its execution task.
+
+        No store write is attempted here: once the last confirmed lease has
+        expired, this worker is no longer authorized to publish a terminal
+        outcome. A peer reconciler owns durable terminalization.
+        """
+        task_to_cancel: asyncio.Task | None = None
+        async with self._lock:
+            current = self._runs.get(record.run_id)
+            if current is not record:
+                return False
+            if require_active:
+                if record.status not in (RunStatus.pending, RunStatus.running):
+                    return False
+                if record.task is not None and record.task.done():
+                    return False
+            if record.ownership_lost:
+                return True
+            record.ownership_lost = True
+            record.abort_event.set()
+            record.status = RunStatus.error
+            record.error = reason
+            record.updated_at = _now_iso()
+            if record.task is not None and not record.task.done() and record.task is not asyncio.current_task():
+                task_to_cancel = record.task
+
+        if task_to_cancel is not None:
+            task_to_cancel.cancel()
+        logger.error("Run %s lost lease ownership; local execution was fenced: %s", record.run_id, reason)
+        return True
+
     async def start_heartbeat(self) -> None:
         """Start the background lease-renewal task.
 
@@ -1294,17 +1716,22 @@ class RunManager:
                 self._schedule_orphan_reconciliation()
 
     async def _renew_leases(self) -> None:
-        """Renew the lease on every locally-owned active run."""
+        """Renew locally-owned leases, failing closed at their deadlines.
+
+        ``RunRecord.lease_expires_at`` advances only after a successful durable
+        renewal, so it is the last confirmed ownership deadline. Transient
+        exceptions are tolerated before that deadline; a call that blocks or
+        keeps failing through it fences the local run.
+        """
         if self._store is None or self._run_ownership_config is None:
             return
         lease_seconds = self._run_ownership_config.lease_seconds
-        new_expiry = (datetime.now(UTC) + timedelta(seconds=lease_seconds)).isoformat()
 
         async with self._lock:
             # Renew any pending/running run owned by this worker unless its
             # background task has already completed. A pending run whose task
             # has not been spawned yet (``task is None``) is still live from
-            # this worker's perspective — between ``create_run_atomic``
+            # this worker's perspective — between ``create_thread_operation_atomic``
             # inserting the row and the worker layer spawning the agent task
             # there is a brief window. If we drop those records here and the
             # window stretches past ``lease_seconds`` (e.g. event-loop
@@ -1314,17 +1741,34 @@ class RunManager:
             active_runs = [(rid, record) for rid, record in self._runs.items() if record.status in (RunStatus.pending, RunStatus.running) and record.owner_worker_id == self._worker_id and (record.task is None or not record.task.done())]
 
         for run_id, record in active_runs:
-            try:
-                updated = await self._call_store_with_retry(
-                    "update_lease",
-                    run_id,
-                    lambda: self._store.update_lease(
-                        run_id,
-                        owner_worker_id=self._worker_id,
-                        lease_expires_at=new_expiry,
-                    ),
+            confirmed_deadline = self._parse_lease_deadline(record.lease_expires_at)
+            if confirmed_deadline is None or confirmed_deadline <= datetime.now(UTC):
+                await self._mark_ownership_lost(
+                    record,
+                    reason="Lease ownership could not be confirmed before the last confirmed lease expired.",
                 )
+                continue
+
+            remaining = (confirmed_deadline - datetime.now(UTC)).total_seconds()
+            new_expiry = (datetime.now(UTC) + timedelta(seconds=lease_seconds)).isoformat()
+            try:
+                async with asyncio.timeout(remaining):
+                    updated = await self._call_store_with_retry(
+                        "update_lease",
+                        run_id,
+                        lambda: self._store.update_lease(
+                            run_id,
+                            owner_worker_id=self._worker_id,
+                            lease_expires_at=new_expiry,
+                        ),
+                    )
                 if updated:
+                    if confirmed_deadline <= datetime.now(UTC):
+                        await self._mark_ownership_lost(
+                            record,
+                            reason="Lease renewal completed after the last confirmed lease had already expired.",
+                        )
+                        continue
                     # Unsynced write is benign: ``lease_expires_at`` is the
                     # only field on an existing record this path mutates, so
                     # there is no concurrent writer to race against
@@ -1338,18 +1782,31 @@ class RunManager:
                     # or ``owner_worker_id`` changed). Stop the local task so
                     # we don't waste CPU or overwrite the takeover status on
                     # finalisation.
-                    logger.warning(
-                        "Run %s lease renewal failed (status=%s,owner=%s) – worker likely taken over; aborting local task",
-                        run_id,
-                        record.status.value,
-                        record.owner_worker_id,
-                    )
-                    record.abort_event.set()
-                    task_active = record.task is not None and not record.task.done()
-                    if task_active:
-                        record.task.cancel()
+                    async with self._lock:
+                        still_active = self._runs.get(run_id) is record and record.status in (RunStatus.pending, RunStatus.running) and record.owner_worker_id == self._worker_id and (record.task is None or not record.task.done())
+                    if still_active:
+                        logger.warning(
+                            "Run %s lease renewal failed (status=%s,owner=%s) – worker likely taken over; aborting local task",
+                            run_id,
+                            record.status.value,
+                            record.owner_worker_id,
+                        )
+                        await self._mark_ownership_lost(
+                            record,
+                            reason="The durable store rejected lease renewal for this worker.",
+                        )
             except Exception:
-                logger.warning("Failed to renew lease for run %s", run_id, exc_info=True)
+                if confirmed_deadline <= datetime.now(UTC):
+                    await self._mark_ownership_lost(
+                        record,
+                        reason="Lease ownership could not be confirmed before the last confirmed lease expired.",
+                    )
+                else:
+                    logger.warning(
+                        "Failed to renew lease for run %s before its confirmed deadline; will retry",
+                        run_id,
+                        exc_info=True,
+                    )
 
     async def _reconcile_orphans_periodic(self) -> None:
         """Sweep for expired leases owned by dead peers.

backend/packages/harness/deerflow/runtime/runs/schemas.py

@@ -3,6 +3,13 @@
 from enum import StrEnum
 
 
+class ThreadOperationKind(StrEnum):
+    """Kind of operation holding exclusive admission for a thread."""
+
+    run = "run"
+    checkpoint_write = "checkpoint_write"
+
+
 class RunStatus(StrEnum):
     """Lifecycle status of a single run."""
 

backend/packages/harness/deerflow/runtime/runs/store/base.py

@@ -25,6 +25,7 @@ class RunStore(abc.ABC):
         user_id: str | None = None,
         model_name: str | None = None,
         status: str = "pending",
+        operation_kind: str = "run",
         multitask_strategy: str = "reject",
         metadata: dict[str, Any] | None = None,
         kwargs: dict[str, Any] | None = None,
@@ -94,10 +95,27 @@ class RunStore(abc.ABC):
         """
         pass
 
+    @abc.abstractmethod
+    async def start_run(self, run_id: str) -> bool:
+        """Atomically transition a pending run to running.
+
+        Returns ``False`` when the row is missing or no longer pending.
+        """
+        pass
+
     @abc.abstractmethod
     async def delete(self, run_id: str) -> None:
         pass
 
+    async def delete_thread_operation(self, run_id: str, *, user_id: str | None) -> None:
+        """Release an admitted thread operation for its recorded owner.
+
+        The default keeps legacy stores compatible: older implementations only
+        accepted ``run_id``. User-aware stores should override this method so
+        cleanup never depends on ambient request context.
+        """
+        await self.delete(run_id)
+
     @abc.abstractmethod
     async def update_model_name(
         self,
@@ -128,7 +146,9 @@ class RunStore(abc.ABC):
     ) -> bool | None:
         """Persist final completion fields.
 
-        Returns ``False`` when the store can prove no row was updated.
+        Implementations must not replace a different terminal status. Returns
+        ``False`` when the row is missing or already has a conflicting terminal
+        outcome.
         """
         pass
 
@@ -215,7 +235,53 @@ class RunStore(abc.ABC):
         """Return active runs whose lease has expired (or is NULL for pre-ownership rows)."""
         pass
 
-    @abc.abstractmethod
+    async def create_thread_operation_atomic(
+        self,
+        run_id: str,
+        *,
+        thread_id: str,
+        owner_worker_id: str,
+        lease_expires_at: str | None,
+        operation_kind: str = "run",
+        multitask_strategy: str = "reject",
+        assistant_id: str | None = None,
+        user_id: str | None = None,
+        model_name: str | None = None,
+        metadata: dict[str, Any] | None = None,
+        kwargs: dict[str, Any] | None = None,
+        created_at: str | None = None,
+        grace_seconds: int = 10,
+    ) -> tuple[dict[str, Any], list[dict[str, Any]]]:
+        """Atomically create an active thread operation with cross-process uniqueness.
+
+        The default implementation preserves compatibility with stores that
+        still implement the former ``create_run_atomic`` interface. Legacy
+        stores support only normal run rows; internal operation kinds require
+        an implementation of this method.
+
+        Returns ``(new_run_dict, claimed_run_dicts)``.
+        Raises ``IntegrityError`` on conflict for ``reject`` strategy.
+        """
+        legacy_impl = type(self).create_run_atomic
+        if legacy_impl is RunStore.create_run_atomic:
+            raise NotImplementedError("RunStore must implement create_thread_operation_atomic() or create_run_atomic()")
+        if operation_kind != "run":
+            raise NotImplementedError("Legacy RunStore.create_run_atomic() cannot create non-run thread operations")
+        return await self.create_run_atomic(
+            run_id,
+            thread_id=thread_id,
+            owner_worker_id=owner_worker_id,
+            lease_expires_at=lease_expires_at,
+            multitask_strategy=multitask_strategy,
+            assistant_id=assistant_id,
+            user_id=user_id,
+            model_name=model_name,
+            metadata=metadata,
+            kwargs=kwargs,
+            created_at=created_at,
+            grace_seconds=grace_seconds,
+        )
+
     async def create_run_atomic(
         self,
         run_id: str,
@@ -232,9 +298,22 @@ class RunStore(abc.ABC):
         created_at: str | None = None,
         grace_seconds: int = 10,
     ) -> tuple[dict[str, Any], list[dict[str, Any]]]:
-        """Atomically create a run row with cross-process thread-uniqueness.
-
-        Returns ``(new_run_dict, claimed_run_dicts)``.
-        Raises ``IntegrityError`` on conflict for ``reject`` strategy.
-        """
-        pass
+        """Deprecated compatibility alias for normal-run admission."""
+        operation_impl = type(self).create_thread_operation_atomic
+        if operation_impl is RunStore.create_thread_operation_atomic:
+            raise NotImplementedError("RunStore must implement create_thread_operation_atomic() or create_run_atomic()")
+        return await self.create_thread_operation_atomic(
+            run_id,
+            thread_id=thread_id,
+            owner_worker_id=owner_worker_id,
+            lease_expires_at=lease_expires_at,
+            operation_kind="run",
+            multitask_strategy=multitask_strategy,
+            assistant_id=assistant_id,
+            user_id=user_id,
+            model_name=model_name,
+            metadata=metadata,
+            kwargs=kwargs,
+            created_at=created_at,
+            grace_seconds=grace_seconds,
+        )

backend/packages/harness/deerflow/runtime/runs/store/memory.py

@@ -41,6 +41,7 @@ class MemoryRunStore(RunStore):
         user_id=None,
         model_name=None,
         status="pending",
+        operation_kind="run",
         multitask_strategy="reject",
         metadata=None,
         kwargs=None,
@@ -58,6 +59,7 @@ class MemoryRunStore(RunStore):
             "user_id": user_id,
             "model_name": model_name,
             "status": status,
+            "operation_kind": operation_kind,
             "multitask_strategy": multitask_strategy,
             "metadata": metadata or {},
             "kwargs": kwargs or {},
@@ -85,7 +87,7 @@ class MemoryRunStore(RunStore):
         run_ids = self._runs_by_thread.get(thread_id)
         if not run_ids:
             return []
-        results = [run for run_id in run_ids if (run := self._runs.get(run_id)) is not None and (user_id is None or run.get("user_id") == user_id)]
+        results = [run for run_id in run_ids if (run := self._runs.get(run_id)) is not None and run.get("operation_kind", "run") == "run" and (user_id is None or run.get("user_id") == user_id)]
         results.sort(key=lambda r: r["created_at"], reverse=True)
         return results[:limit]
 
@@ -94,7 +96,7 @@ class MemoryRunStore(RunStore):
         sources: set[str] = set()
         for run_id in run_ids:
             run = self._runs.get(run_id)
-            if run is None or run.get("status") != "success":
+            if run is None or run.get("operation_kind", "run") != "run" or run.get("status") != "success":
                 continue
             if user_id is not None and run.get("user_id") != user_id:
                 continue
@@ -105,7 +107,7 @@ class MemoryRunStore(RunStore):
 
     async def get_many_by_thread(self, thread_id, run_ids, *, user_id=None):
         thread_run_ids = self._runs_by_thread.get(thread_id) or ()
-        return {run_id: run for run_id in thread_run_ids if run_id in run_ids and (run := self._runs.get(run_id)) is not None and (user_id is None or run.get("user_id") == user_id)}
+        return {run_id: run for run_id in thread_run_ids if run_id in run_ids and (run := self._runs.get(run_id)) is not None and run.get("operation_kind", "run") == "run" and (user_id is None or run.get("user_id") == user_id)}
 
     async def update_status(self, run_id, status, *, error=None, stop_reason=None):
         run = self._runs.get(run_id)
@@ -123,25 +125,40 @@ class MemoryRunStore(RunStore):
         run["updated_at"] = datetime.now(UTC).isoformat()
         return True
 
+    async def start_run(self, run_id) -> bool:
+        run = self._runs.get(run_id)
+        if run is None or run["status"] != "pending":
+            return False
+        run["status"] = "running"
+        run["updated_at"] = datetime.now(UTC).isoformat()
+        return True
+
     async def update_model_name(self, run_id, model_name):
         if run_id in self._runs:
             self._runs[run_id]["model_name"] = model_name
             self._runs[run_id]["updated_at"] = datetime.now(UTC).isoformat()
 
-    async def delete(self, run_id):
+    async def delete(self, run_id, *, user_id=None):
         run = self._runs.pop(run_id, None)
         if run is not None:
             self._unindex_run(run_id, run["thread_id"])
 
     async def update_run_completion(self, run_id, *, status, **kwargs):
-        if run_id in self._runs:
-            self._runs[run_id]["status"] = status
-            for key, value in kwargs.items():
-                if value is not None:
-                    self._runs[run_id][key] = value
-            self._runs[run_id]["updated_at"] = datetime.now(UTC).isoformat()
-            return True
-        return False
+        run = self._runs.get(run_id)
+        if run is None:
+            return False
+        current_status = run.get("status")
+        allowed_sources = {"pending", "running", status}
+        if status == "error":
+            allowed_sources.add("interrupted")
+        if current_status not in allowed_sources:
+            return False
+        run["status"] = status
+        for key, value in kwargs.items():
+            if value is not None:
+                run[key] = value
+        run["updated_at"] = datetime.now(UTC).isoformat()
+        return True
 
     async def update_run_progress(self, run_id, **kwargs):
         if run_id in self._runs and self._runs[run_id].get("status") == "running":
@@ -152,7 +169,7 @@ class MemoryRunStore(RunStore):
 
     async def list_pending(self, *, before=None):
         now = before or datetime.now(UTC).isoformat()
-        results = [r for r in self._runs.values() if r["status"] == "pending" and r["created_at"] <= now]
+        results = [r for r in self._runs.values() if r.get("operation_kind", "run") == "run" and r["status"] == "pending" and r["created_at"] <= now]
         results.sort(key=lambda r: r["created_at"])
         return results
 
@@ -167,7 +184,7 @@ class MemoryRunStore(RunStore):
         # Use the thread index for an O(runs-in-thread) lookup instead of
         # scanning every run in the process (mirrors ``list_by_thread``).
         run_ids = self._runs_by_thread.get(thread_id) or ()
-        completed = [run for run_id in run_ids if (run := self._runs.get(run_id)) is not None and run.get("status") in statuses]
+        completed = [run for run_id in run_ids if (run := self._runs.get(run_id)) is not None and run.get("operation_kind", "run") == "run" and run.get("status") in statuses]
         by_model: dict[str, dict] = {}
         for r in completed:
             usage_by_model = r.get("token_usage_by_model") or {}
@@ -288,13 +305,14 @@ class MemoryRunStore(RunStore):
         results.sort(key=lambda r: r["created_at"])
         return results
 
-    async def create_run_atomic(
+    async def create_thread_operation_atomic(
         self,
         run_id: str,
         *,
         thread_id: str,
         owner_worker_id: str,
         lease_expires_at: str | None,
+        operation_kind: str = "run",
         multitask_strategy: str = "reject",
         assistant_id: str | None = None,
         user_id: str | None = None,
@@ -330,6 +348,7 @@ class MemoryRunStore(RunStore):
                     continue
                 if r["status"] not in ("pending", "running"):
                     continue
+                lease_expired = False
                 existing_lease = r.get("lease_expires_at")
                 if existing_lease is not None:
                     try:
@@ -340,6 +359,7 @@ class MemoryRunStore(RunStore):
                         # raise ``TypeError``.
                         if lease_dt.tzinfo is None:
                             lease_dt = lease_dt.replace(tzinfo=UTC)
+                        lease_expired = lease_dt < cutoff
                         if lease_dt >= cutoff and r.get("owner_worker_id") != owner_worker_id:
                             # Live run owned by another worker — cannot
                             # interrupt, and the partial unique index would
@@ -349,6 +369,8 @@ class MemoryRunStore(RunStore):
                             raise ConflictError(f"Thread {thread_id} already has an active run owned by another worker")
                     except (ValueError, TypeError):
                         pass
+                if r.get("operation_kind", "run") != "run" and not lease_expired:
+                    raise ConflictError(f"Thread {thread_id} has an active checkpoint write")
                 candidates.append(r)
             for r in candidates:
                 r["status"] = "interrupted"
@@ -364,6 +386,7 @@ class MemoryRunStore(RunStore):
             "user_id": user_id,
             "model_name": model_name,
             "status": "pending",
+            "operation_kind": operation_kind,
             "multitask_strategy": multitask_strategy,
             "metadata": metadata or {},
             "kwargs": kwargs or {},

backend/packages/harness/deerflow/runtime/runs/worker.py

@@ -40,7 +40,13 @@ from deerflow.runtime.checkpoint_mode import (
     aensure_checkpoint_mode_compatible,
     inject_checkpoint_mode,
 )
-from deerflow.runtime.checkpoint_state import CheckpointStateAccessor, build_state_mutation_graph, graph_state_schema
+from deerflow.runtime.checkpoint_state import (
+    CheckpointStateAccessor,
+    build_state_mutation_graph,
+    graph_reducer_channels,
+    graph_state_schema,
+    graph_writable_channels,
+)
 from deerflow.runtime.context_keys import CURRENT_RUN_PRE_EXISTING_MESSAGE_IDS_KEY
 from deerflow.runtime.goal import (
     DEFAULT_MAX_GOAL_CONTINUATIONS,
@@ -75,7 +81,7 @@ from deerflow.utils.messages import message_to_text
 from deerflow.workspace_changes import capture_workspace_snapshot, record_workspace_changes
 from deerflow.workspace_changes.types import WorkspaceSnapshot
 
-from .manager import RunManager, RunRecord
+from .manager import RunManager, RunRecord, RunStartOutcome
 from .naming import resolve_root_run_name
 from .schemas import RunStatus
 
@@ -103,6 +109,57 @@ async def _checkpoint_thread_lock(thread_id: str) -> AsyncIterator[None]:
         yield
 
 
+_DELIVERY_RECEIPT_RETRY_DELAYS_SECONDS = (0.1, 0.5)
+
+
+async def _persist_delivery_receipt(
+    event_store: Any,
+    *,
+    thread_id: str,
+    run_id: str,
+    content: dict[str, Any],
+) -> bool:
+    """Persist a terminal receipt with short bounded retries.
+
+    The owning worker still knows the real terminal outcome and renews its
+    lease while this coroutine runs. Retrying here handles transient event
+    store failures without handing a successful run to orphan recovery, which
+    cannot reconstruct either the terminal status or the detailed receipt.
+    """
+    attempts = len(_DELIVERY_RECEIPT_RETRY_DELAYS_SECONDS) + 1
+    for attempt in range(attempts):
+        try:
+            await event_store.put_if_absent(
+                thread_id=thread_id,
+                run_id=run_id,
+                event_type="run.delivery",
+                category="outputs",
+                content=content,
+            )
+            return True
+        except Exception:
+            if attempt == attempts - 1:
+                logger.warning(
+                    "Failed to persist delivery receipt for run %s after %d attempts; preserving the real terminal status without a receipt",
+                    run_id,
+                    attempts,
+                    exc_info=True,
+                )
+                return False
+            delay = _DELIVERY_RECEIPT_RETRY_DELAYS_SECONDS[attempt]
+            logger.warning(
+                "Failed to persist delivery receipt for run %s (attempt %d/%d); retrying in %.1fs",
+                run_id,
+                attempt + 1,
+                attempts,
+                delay,
+                exc_info=True,
+            )
+            await asyncio.sleep(delay)
+
+    return False  # pragma: no cover - loop always returns
+
+
 # Keep this streaming policy separate from middleware write-authorization sets.
 _LARGE_FILE_TOOL_NAMES = frozenset({"str_replace", "write_file"})
 _LARGE_FILE_TOOL_BATCH_SIZE = 32
@@ -393,6 +450,7 @@ async def run_agent(
     event_store = ctx.event_store
     run_events_config = ctx.run_events_config
     thread_store = ctx.thread_store
+    terminal_status_kwargs = {"persist": False} if event_store is not None else {}
 
     run_id = record.run_id
     thread_id = record.thread_id
@@ -413,16 +471,54 @@ async def run_agent(
     accessor: CheckpointStateAccessor | None = None
     rollback_point: RollbackPoint | None = None
     journal = None
+    # Journal construction moved ahead of preflight so every terminal run can
+    # emit a receipt. Completion persistence keeps its prior boundary: before
+    # #4272 the journal did not exist until preflight had succeeded, so early
+    # checkpoint failures / cancellation while waiting did not write an empty
+    # completion snapshot into RunStore.
+    persist_completion = False
     # Buffers subagent step events for batched persistence (#3779); assigned once
     # streaming starts and flushed in the finally block. Pre-bound to None so the
     # finally is safe even if an exception fires before streaming begins.
     subagent_events: _SubagentEventBuffer | None = None
+    started = False
 
     try:
         normalized_stream_modes = normalize_stream_modes(stream_modes)
         requested_modes: set[str] = set(normalized_stream_modes)
         lg_modes = to_langgraph_stream_modes(normalized_stream_modes)
-        await run_manager.wait_for_prior_finalizing(thread_id, run_id)
+        # Initialize the run-scoped journal before any fallible or cancellable
+        # preflight work. Every terminal run with an event store must reach the
+        # shared finally block with a journal available for its run.delivery
+        # receipt, including checkpoint validation failures and cancellation
+        # while waiting for an earlier run to finish finalizing.
+        if event_store is not None:
+            from deerflow.runtime.journal import RunJournal
+
+            journal = RunJournal(
+                run_id=run_id,
+                thread_id=thread_id,
+                event_store=event_store,
+                track_token_usage=getattr(run_events_config, "track_token_usage", True),
+                progress_reporter=lambda snapshot: run_manager.update_run_progress(run_id, **snapshot),
+            )
+
+        await run_manager.wait_for_prior_finalizing(
+            thread_id,
+            run_id,
+            abort_event=record.abort_event,
+        )
+
+        start_outcome = await run_manager.try_start(run_id)
+        if start_outcome is not RunStartOutcome.started:
+            return
+        started = True
+
+        if not record.ownership_lost and thread_store is not None:
+            try:
+                await thread_store.update_status(thread_id, "running")
+            except Exception:
+                logger.debug("Failed to update thread_meta status for %s (non-fatal)", thread_id)
         mode = ctx.checkpoint_channel_mode
         inject_checkpoint_mode(config, mode)
         checkpoint_config = {
@@ -455,25 +551,7 @@ async def run_agent(
                     mode,
                 )
 
-        # Initialize RunJournal + write human_message event.
-        # These are inside the try block so any exception (e.g. a DB
-        # error writing the event) flows through the except/finally
-        # path that publishes an "end" event to the SSE bridge —
-        # otherwise a failure here would leave the stream hanging
-        # with no terminator.
-        if event_store is not None:
-            from deerflow.runtime.journal import RunJournal
-
-            journal = RunJournal(
-                run_id=run_id,
-                thread_id=thread_id,
-                event_store=event_store,
-                track_token_usage=getattr(run_events_config, "track_token_usage", True),
-                progress_reporter=lambda snapshot: run_manager.update_run_progress(run_id, **snapshot),
-            )
-
-        # 1. Mark running
-        await run_manager.set_status(run_id, RunStatus.running)
+        persist_completion = True
 
         if event_store is not None:
             workspace_changes_user_id = get_effective_user_id()
@@ -575,14 +653,38 @@ async def run_agent(
         # failure disables rollback: restoring an empty or partial message
         # history would silently truncate the thread.
         if checkpointer is not None:
-            try:
-                rollback_point = await _capture_rollback_point(accessor, checkpointer, checkpoint_config)
-            except Exception:
-                snapshot_capture_failed = True
-                logger.warning("Could not capture pre-run checkpoint snapshot for run %s", run_id, exc_info=True)
-            if rollback_point is not None:
-                pre_run_checkpoint_id = rollback_point.config.get("configurable", {}).get("checkpoint_id")
-                pre_existing_message_ids = _collect_pre_existing_message_ids({"messages": list(rollback_point.messages)})
+            # A previous successful run may still be persisting duration
+            # metadata after its active admission slot is released. Share its
+            # checkpoint lock so the rollback snapshot and any resume rewrite
+            # are one uninterrupted read/write sequence against the head.
+            async with _checkpoint_thread_lock(thread_id):
+                try:
+                    rollback_point = await _capture_rollback_point(accessor, checkpointer, checkpoint_config)
+                except Exception:
+                    snapshot_capture_failed = True
+                    logger.warning("Could not capture pre-run checkpoint snapshot for run %s", run_id, exc_info=True)
+                if rollback_point is not None:
+                    pre_run_checkpoint_id = rollback_point.config.get("configurable", {}).get("checkpoint_id")
+                    pre_existing_message_ids = _collect_pre_existing_message_ids({"messages": list(rollback_point.messages)})
+
+                # Resuming from an older checkpoint is a fork, and a delta fork
+                # materializes the abandoned sibling's writes back into state
+                # (#4458). Rewrite it as a linear head write *after* the rollback
+                # point is captured, so cancel-with-rollback still restores the
+                # real pre-run head rather than the rolled-back one.
+                resumed_messages = await _linearize_delta_checkpoint_resume(
+                    accessor=accessor,
+                    checkpointer=checkpointer,
+                    config=config,
+                    thread_id=thread_id,
+                    run_id=run_id,
+                )
+            if resumed_messages is not None:
+                # The graph now starts from the selected state, so the
+                # current-run message boundary is that state, not the head we
+                # captured for rollback.
+                pre_existing_message_ids = _collect_pre_existing_message_ids({"messages": list(resumed_messages)})
+                initial_runnable_config = RunnableConfig(**config)
 
         runtime_ctx[CURRENT_RUN_PRE_EXISTING_MESSAGE_IDS_KEY] = frozenset(pre_existing_message_ids)
         _install_runtime_context(config, runtime_ctx)
@@ -717,7 +819,12 @@ async def run_agent(
             await run_manager.set_finalizing(run_id, True)
             action = record.abort_action
             if action == "rollback":
-                await run_manager.set_status(run_id, RunStatus.error, error="Rolled back by user")
+                await run_manager.set_status(
+                    run_id,
+                    RunStatus.error,
+                    error="Rolled back by user",
+                    **terminal_status_kwargs,
+                )
                 try:
                     await _rollback_to_pre_run_checkpoint(
                         accessor=accessor,
@@ -731,13 +838,22 @@ async def run_agent(
                 except Exception:
                     logger.warning("Failed to rollback checkpoint for run %s", run_id, exc_info=True)
             else:
-                await run_manager.set_status(run_id, RunStatus.interrupted)
+                await run_manager.set_status(
+                    run_id,
+                    RunStatus.interrupted,
+                    **terminal_status_kwargs,
+                )
         elif llm_error_fallback_message or (journal is not None and journal.had_llm_error_fallback):
             error_msg = llm_error_fallback_message
             if error_msg is None and journal is not None:
                 error_msg = journal.llm_error_fallback_message
             error_msg = error_msg or "LLM provider failed after retries"
-            await run_manager.set_status(run_id, RunStatus.error, error=error_msg)
+            await run_manager.set_status(
+                run_id,
+                RunStatus.error,
+                error=error_msg,
+                **terminal_status_kwargs,
+            )
         else:
             runtime_context = runtime.context if isinstance(runtime.context, dict) else None
             # Guard middlewares that hard-stop a run by stripping tool_calls
@@ -747,6 +863,7 @@ async def run_agent(
             #   token_budget        -> "token_capped"
             #   safety_finish_reason -> "safety_capped"
             #   subagent_limit       -> "subagent_limit_capped"
+            #   model_length_finish_reason -> "model_length_capped"
             #
             # If more guards grow stop_reason semantics, consider a publish/
             # collect pattern (e.g. each guard middleware publishes its cap
@@ -754,13 +871,23 @@ async def run_agent(
             # collects the most severe / first / all reasons) instead of each
             # guard writing directly to the same key.
             stop_reason = runtime_context.get("stop_reason") if runtime_context is not None else None
-            await run_manager.set_status(run_id, RunStatus.success, stop_reason=stop_reason)
+            await run_manager.set_status(
+                run_id,
+                RunStatus.success,
+                stop_reason=stop_reason,
+                **terminal_status_kwargs,
+            )
 
     except asyncio.CancelledError:
         await run_manager.set_finalizing(run_id, True)
         action = record.abort_action
         if action == "rollback":
-            await run_manager.set_status(run_id, RunStatus.error, error="Rolled back by user")
+            await run_manager.set_status(
+                run_id,
+                RunStatus.error,
+                error="Rolled back by user",
+                **terminal_status_kwargs,
+            )
             try:
                 await _rollback_to_pre_run_checkpoint(
                     accessor=accessor,
@@ -774,13 +901,22 @@ async def run_agent(
             except Exception:
                 logger.warning("Run %s cancellation rollback failed", run_id, exc_info=True)
         else:
-            await run_manager.set_status(run_id, RunStatus.interrupted)
+            await run_manager.set_status(
+                run_id,
+                RunStatus.interrupted,
+                **terminal_status_kwargs,
+            )
             logger.info("Run %s was cancelled", run_id)
 
     except Exception as exc:
         error_msg = f"{exc}"
         logger.exception("Run %s failed: %s", run_id, error_msg)
-        await run_manager.set_status(run_id, RunStatus.error, error=error_msg)
+        await run_manager.set_status(
+            run_id,
+            RunStatus.error,
+            error=error_msg,
+            **terminal_status_kwargs,
+        )
         await bridge.publish(
             run_id,
             "error",
@@ -791,12 +927,18 @@ async def run_agent(
         )
 
     finally:
+        if record.ownership_lost:
+            logger.warning(
+                "Skipping durable finalization for run %s because this worker no longer owns its lease",
+                run_id,
+            )
+
         # Persist any subagent step events still buffered (#3779) — including on
         # abort/exception paths, where the stream loop broke before its own flush.
-        if subagent_events is not None:
+        if not record.ownership_lost and subagent_events is not None:
             await subagent_events.flush()
 
-        if event_store is not None and pre_run_workspace_snapshot is not None:
+        if not record.ownership_lost and event_store is not None and pre_run_workspace_snapshot is not None:
             try:
                 await record_workspace_changes(
                     event_store,
@@ -808,13 +950,35 @@ async def run_agent(
             except Exception:
                 logger.warning("Failed to record workspace changes for run %s", run_id, exc_info=True)
 
-        # Flush any buffered journal events and persist completion data
-        if journal is not None:
+        # Flush buffered journal events before the terminal receipt. The
+        # receipt uses a run-scoped idempotent write shared with recovery, then
+        # the staged terminal status is persisted. This ordering closes the
+        # crash window where a terminal run could otherwise outlive its receipt.
+        # A fenced worker leaves receipt recovery to the peer that claimed it.
+        if not record.ownership_lost and journal is not None:
             try:
                 await journal.flush()
             except Exception:
                 logger.warning("Failed to flush journal for run %s", run_id, exc_info=True)
 
+            await _persist_delivery_receipt(
+                event_store,
+                thread_id=thread_id,
+                run_id=run_id,
+                content=journal.get_delivery_content(),
+            )
+
+        if not record.ownership_lost and event_store is not None:
+            try:
+                # Even after bounded receipt retries are exhausted, persist the
+                # real worker outcome. Leaving a successful row inflight would
+                # let lease recovery rewrite it as an error with a synthetic
+                # zero receipt.
+                await run_manager.persist_current_status(run_id)
+            except Exception:
+                logger.warning("Failed to persist terminal status for run %s after delivery receipt attempts", run_id, exc_info=True)
+
+        if not record.ownership_lost and journal is not None and persist_completion:
             try:
                 # Persist token usage + convenience fields to RunStore
                 completion = journal.get_completion_data()
@@ -822,7 +986,7 @@ async def run_agent(
             except Exception:
                 logger.warning("Failed to persist run completion for %s (non-fatal)", run_id, exc_info=True)
 
-        if checkpointer is not None and record.status == RunStatus.interrupted:
+        if started and not record.ownership_lost and checkpointer is not None and record.status == RunStatus.interrupted:
             try:
                 await run_manager.wait_for_prior_finalizing(thread_id, run_id)
                 if not await run_manager.has_later_started_run(thread_id, run_id):
@@ -831,7 +995,7 @@ async def run_agent(
                 logger.debug("Failed to generate interrupted title for thread %s (non-fatal)", thread_id)
 
         # Sync title from checkpoint to threads_meta.display_name
-        if checkpointer is not None and thread_store is not None:
+        if started and not record.ownership_lost and checkpointer is not None and thread_store is not None:
             try:
                 ckpt_config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
                 ckpt_tuple = await checkpointer.aget_tuple(ckpt_config)
@@ -845,7 +1009,7 @@ async def run_agent(
 
         # Persist run duration to checkpoint metadata so history reads
         # don't need to correlate runs and events.
-        if checkpointer is not None and record.status == RunStatus.success:
+        if started and not record.ownership_lost and checkpointer is not None and record.status == RunStatus.success:
             try:
                 created = datetime.fromisoformat(record.created_at.replace("Z", "+00:00"))
                 updated = datetime.fromisoformat(record.updated_at.replace("Z", "+00:00"))
@@ -863,14 +1027,14 @@ async def run_agent(
                 logger.debug("Failed to persist run duration for thread %s run %s (non-fatal)", thread_id, run_id)
 
         # Update threads_meta status based on run outcome
-        if thread_store is not None:
+        if started and not record.ownership_lost and thread_store is not None:
             try:
                 final_status = "idle" if record.status == RunStatus.success else record.status.value
                 await thread_store.update_status(thread_id, final_status)
             except Exception:
                 logger.debug("Failed to update thread_meta status for %s (non-fatal)", thread_id)
 
-        if ctx.on_run_completed is not None:
+        if not record.ownership_lost and ctx.on_run_completed is not None:
             try:
                 await ctx.on_run_completed(record)
             except Exception:
@@ -1227,15 +1391,16 @@ async def _prepare_goal_continuation_input(
 
 @dataclass(frozen=True)
 class RollbackPoint:
-    """Materialized pre-run state used to fork the pre-run checkpoint lineage.
+    """Materialized pre-run state used to restore the thread after cancellation.
 
     Raw checkpoint blobs cannot reconstruct Delta-channel messages (their
-    checkpoints omit ``channel_values``), so rollback restores messages by
-    applying an ``Overwrite`` through a state-mutation graph anchored at the
-    pre-run checkpoint instead of cloning the raw blob.
+    checkpoints omit the materialized value), so rollback preserves those
+    messages plus delta mode's materialized non-message state in addition to
+    the raw pending writes.
     """
 
     config: dict[str, Any]
+    state_values: dict[str, Any]
     messages: tuple[Any, ...]
     metadata: dict[str, Any]
     pending_writes: tuple[tuple[str, str, Any], ...]
@@ -1257,8 +1422,9 @@ async def _capture_rollback_point(
     if not configurable.get("checkpoint_id"):
         return None
     checkpoint_tuple = await _call_checkpointer_method(checkpointer, "aget_tuple", "get_tuple", snapshot_config)
-    values = getattr(snapshot, "values", None) or {}
-    messages = values.get("messages") if isinstance(values, dict) else None
+    raw_values = getattr(snapshot, "values", None) or {}
+    messages = raw_values.get("messages") if isinstance(raw_values, dict) else None
+    state_values = copy.deepcopy({key: value for key, value in raw_values.items() if key != "messages"}) if accessor.mode == "delta" and isinstance(raw_values, dict) else {}
     return RollbackPoint(
         config={
             "configurable": {
@@ -1267,12 +1433,132 @@ async def _capture_rollback_point(
                 "checkpoint_id": configurable.get("checkpoint_id"),
             }
         },
+        state_values=state_values,
         messages=tuple(messages or ()),
         metadata=dict(getattr(snapshot, "metadata", None) or {}),
         pending_writes=tuple(getattr(checkpoint_tuple, "pending_writes", ()) or ()),
     )
 
 
+def _complete_state_replacement_values(
+    *,
+    mutation_graph: Any,
+    selected_values: dict[str, Any],
+    current_values: dict[str, Any],
+    run_id: str,
+    operation: str,
+) -> dict[str, Any]:
+    """Build a whole-state replacement through the graph's effective schema."""
+    writable_fields = graph_writable_channels(mutation_graph)
+    reducer_fields = graph_reducer_channels(mutation_graph)
+    if writable_fields is None or reducer_fields is None:
+        raise RuntimeError(f"Run {run_id} could not inspect the state schema for {operation}")
+
+    replacement_values: dict[str, Any] = {}
+    for field_name in writable_fields:
+        if field_name in selected_values:
+            replacement = copy.deepcopy(selected_values[field_name])
+        elif field_name in current_values:
+            # LangGraph has no public "unset channel" update. A fresh channel
+            # exposes its schema default when one exists (for example [] / {});
+            # optional and otherwise-unconstructible channels reset to None.
+            channel = mutation_graph.channels.get(field_name)
+            replacement = copy.deepcopy(channel.get()) if channel is not None and channel.is_available() else None
+        else:
+            continue
+        replacement_values[field_name] = Overwrite(replacement) if field_name in reducer_fields else replacement
+    return replacement_values
+
+
+async def _linearize_delta_checkpoint_resume(
+    *,
+    accessor: CheckpointStateAccessor,
+    checkpointer: Any,
+    config: dict[str, Any],
+    thread_id: str,
+    run_id: str,
+) -> list[Any] | None:
+    """Replace a delta-mode checkpoint fork with an equivalent linear write.
+
+    Resuming from an older checkpoint forks the lineage, and in ``delta`` mode
+    the fork's state cannot be materialized correctly: the delta history walk
+    collects **every** ``pending_writes`` entry stored on each on-path
+    ancestor, but a shared parent also carries the writes of the sibling child
+    that was abandoned. Those writes are replayed into the fork, so the run
+    starts from a message list that still contains the answer it was supposed
+    to replace — regenerating in a branched thread surfaced this as the old
+    assistant message reappearing beside the new one after a reload (#4458).
+    Reproduced on postgres, sqlite, and the in-memory saver; ``full`` mode is
+    unaffected because its checkpoints carry complete ``channel_values`` and
+    need no replay.
+
+    The upstream contract (`BaseCheckpointSaver.get_delta_channel_history` and
+    the savers overriding it) is where write-to-child ownership belongs, so
+    this does not reimplement it. Instead the fork is expressed as what it
+    means: materialize the requested checkpoint's state and write it with
+    replace semantics on the **current head**, which has no other children,
+    then run linearly. Every materialized channel is restored; channels that
+    exist only on the newer head are reset to their schema default (or
+    ``None`` when the channel has no constructible default). The abandoned
+    turn stays in checkpoint history as the rewritten head's ancestry.
+
+    Returns the materialized messages when the resume was linearized, or
+    ``None`` when there was nothing to do (full mode, no checkpoint selector,
+    a non-root namespace, or a selector that already names the head). Failures
+    propagate: silently falling back to the fork would persist the corrupted
+    history this exists to prevent. The worker call site holds
+    ``_checkpoint_thread_lock`` across rollback capture and this rewrite; do
+    not reacquire that non-reentrant lock inside this helper.
+    """
+    if checkpointer is None or accessor.mode != "delta":
+        return None
+    configurable = config.get("configurable")
+    if not isinstance(configurable, dict):
+        return None
+    checkpoint_id = configurable.get("checkpoint_id")
+    if not isinstance(checkpoint_id, str) or not checkpoint_id:
+        return None
+    if configurable.get("checkpoint_ns"):
+        # Subgraph namespaces have their own lineage; the Gateway only selects
+        # root checkpoints, so leave anything else untouched.
+        return None
+
+    head_config: dict[str, Any] = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
+    head = await accessor.aget(head_config)
+    if _checkpoint_id(head) == checkpoint_id:
+        # Selecting the head is already linear — no sibling can exist yet.
+        return None
+
+    source_config: dict[str, Any] = {"configurable": {"thread_id": thread_id, "checkpoint_ns": "", "checkpoint_id": checkpoint_id}}
+    snapshot = await accessor.aget(source_config)
+    values = getattr(snapshot, "values", None) or {}
+    messages = values.get("messages") if isinstance(values, dict) else None
+    if not isinstance(messages, list):
+        raise RuntimeError(f"Run {run_id} could not materialize resume checkpoint {checkpoint_id}")
+
+    # Write through the thread's effective schema so every application and
+    # middleware channel can be restored. Reducer channels need Overwrite to
+    # replace their already-aggregated value instead of merging it again.
+    mutation_graph = build_state_mutation_graph("checkpoint_resume", accessor.mode, graph_state_schema(getattr(accessor, "graph", None)))
+    selected_values = dict(values)
+    head_values = getattr(head, "values", None) or {}
+    head_values = dict(head_values) if isinstance(head_values, dict) else {}
+    replacement_values = _complete_state_replacement_values(
+        mutation_graph=mutation_graph,
+        selected_values=selected_values,
+        current_values=head_values,
+        run_id=run_id,
+        operation="checkpoint resume",
+    )
+
+    mutation_accessor = CheckpointStateAccessor.bind(mutation_graph, checkpointer, mode=accessor.mode)
+    await mutation_accessor.aupdate(head_config, replacement_values, as_node="checkpoint_resume")
+    configurable.pop("checkpoint_id", None)
+    configurable.pop("checkpoint_map", None)
+    logger.info("Run %s linearized a delta-mode resume of checkpoint %s onto thread %s", run_id, checkpoint_id, thread_id)
+    return list(messages)
+
+
 async def _rollback_to_pre_run_checkpoint(
     *,
     accessor: CheckpointStateAccessor | None,
@@ -1282,13 +1568,14 @@ async def _rollback_to_pre_run_checkpoint(
     rollback_point: RollbackPoint | None,
     snapshot_capture_failed: bool,
 ) -> None:
-    """Fork the pre-run checkpoint lineage with the pre-run messages restored.
+    """Restore the complete pre-run state after a cancelled run.
 
-    The fork is written through a state-only mutation graph (the synthetic
-    ``rollback_restore`` node must be registered for ``as_node`` and finishes
-    immediately so no agent nodes are scheduled). LangGraph owns the restored
-    checkpoint's source/step/channel versions/parent/timestamp; the parent
-    pointer back to the pre-run checkpoint is the audit trail.
+    Full mode forks the captured pre-run checkpoint and overwrites messages;
+    all other channels inherit from that parent. Delta mode cannot safely fork
+    once the cancelled path has attached writes to the same parent, so it
+    replaces every captured channel on the current head instead. Both writes
+    use a state-only mutation graph whose synthetic ``rollback_restore`` node
+    finishes immediately and schedules no agent work.
     """
     if checkpointer is None:
         logger.info("Run %s rollback requested but no checkpointer is configured", run_id)
@@ -1314,15 +1601,35 @@ async def _rollback_to_pre_run_checkpoint(
         logger.warning("Run %s rollback skipped: agent accessor unavailable", run_id)
         return
 
-    # The restored checkpoint inherits every channel from the pre-run fork;
-    # compile the mutation graph with the thread's effective schema so
-    # middleware-contributed channels survive (the base ThreadState fallback
-    # would silently drop them).
+    # Compile with the thread's effective schema so middleware-contributed
+    # channels survive (the base ThreadState fallback would silently drop
+    # them).
     mutation_graph = build_state_mutation_graph("rollback_restore", accessor.mode, graph_state_schema(getattr(accessor, "graph", None)))
     mutation_accessor = CheckpointStateAccessor.bind(mutation_graph, checkpointer, mode=accessor.mode)
+    if accessor.mode == "delta":
+        # A delta rollback fork has the same write-ownership problem as a
+        # checkpoint resume: the captured parent now carries writes from the
+        # cancelled sibling. Restore linearly on the current head instead.
+        restore_config: dict[str, Any] = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
+        current = await accessor.aget(restore_config)
+        raw_current_values = getattr(current, "values", None) or {}
+        current_values = dict(raw_current_values) if isinstance(raw_current_values, dict) else {}
+        selected_values = copy.deepcopy(rollback_point.state_values)
+        selected_values["messages"] = list(rollback_point.messages)
+        replacement_values = _complete_state_replacement_values(
+            mutation_graph=mutation_graph,
+            selected_values=selected_values,
+            current_values=current_values,
+            run_id=run_id,
+            operation="rollback",
+        )
+    else:
+        restore_config = rollback_point.config
+        replacement_values = {"messages": Overwrite(list(rollback_point.messages))}
+
     restored_config = await mutation_accessor.aupdate(
-        rollback_point.config,
-        {"messages": Overwrite(list(rollback_point.messages))},
+        restore_config,
+        replacement_values,
         as_node="rollback_restore",
     )
     if not isinstance(restored_config, dict):

backend/packages/harness/deerflow/runtime/stream_bridge/__init__.py

@@ -8,7 +8,7 @@ by :mod:`asyncio.Queue`.
 """
 
 from .async_provider import make_stream_bridge
-from .base import END_SENTINEL, HEARTBEAT_SENTINEL, StreamBridge, StreamEvent
+from .base import END_SENTINEL, HEARTBEAT_SENTINEL, StreamBridge, StreamEvent, StreamGap, StreamItem
 from .memory import MemoryStreamBridge
 
 # NOTE: ``RedisStreamBridge`` is intentionally NOT imported here. ``redis`` is an
@@ -25,5 +25,7 @@ __all__ = [
     "MemoryStreamBridge",
     "StreamBridge",
     "StreamEvent",
+    "StreamGap",
+    "StreamItem",
     "make_stream_bridge",
 ]

backend/packages/harness/deerflow/runtime/stream_bridge/base.py

@@ -30,8 +30,24 @@ class StreamEvent:
     data: Any
 
 
+@dataclass(frozen=True)
+class StreamGap:
+    """A subscriber cursor can no longer be replayed completely.
+
+    ``requested_event_id`` is the reconnect cursor, or the most recently
+    delivered event for a live subscriber that fell behind.  The retained
+    bounds let callers reload durable state and resume at the current tail
+    without mistaking a partial replay for a complete one.
+    """
+
+    requested_event_id: str | None
+    earliest_available_event_id: str
+    latest_available_event_id: str
+
+
 HEARTBEAT_SENTINEL = StreamEvent(id="", event="__heartbeat__", data=None)
 END_SENTINEL = StreamEvent(id="", event="__end__", data=None)
+type StreamItem = StreamEvent | StreamGap
 
 
 class StreamBridge(abc.ABC):
@@ -54,12 +70,13 @@ class StreamBridge(abc.ABC):
         *,
         last_event_id: str | None = None,
         heartbeat_interval: float = 15.0,
-    ) -> AsyncIterator[StreamEvent]:
+    ) -> AsyncIterator[StreamItem]:
         """Async iterator that yields events for *run_id* (consumer side).
 
         Yields :data:`HEARTBEAT_SENTINEL` when no event arrives within
         *heartbeat_interval* seconds.  Yields :data:`END_SENTINEL` once
-        the producer calls :meth:`publish_end`.
+        the producer calls :meth:`publish_end`. Yields :class:`StreamGap` and
+        stops when the subscriber has fallen behind retained history.
         """
 
     @abc.abstractmethod

backend/packages/harness/deerflow/runtime/stream_bridge/memory.py

@@ -4,14 +4,16 @@ from __future__ import annotations
 
 import asyncio
 import logging
+import re
 import time
 from collections.abc import AsyncIterator
 from dataclasses import dataclass, field
 from typing import Any
 
-from .base import END_SENTINEL, HEARTBEAT_SENTINEL, StreamBridge, StreamEvent
+from .base import END_SENTINEL, HEARTBEAT_SENTINEL, StreamBridge, StreamEvent, StreamGap, StreamItem
 
 logger = logging.getLogger(__name__)
+_MEMORY_STREAM_ID_RE = re.compile(r"\d+-(\d+)")
 
 
 @dataclass
@@ -56,25 +58,35 @@ class MemoryStreamBridge(StreamBridge):
         event, so it equals the event's absolute offset within the run. Returns
         ``None`` for ids that do not match the expected format.
         """
-        _, sep, seq_text = event_id.rpartition("-")
-        if not sep:
-            return None
-        try:
-            return int(seq_text)
-        except ValueError:
+        match = _MEMORY_STREAM_ID_RE.fullmatch(event_id)
+        if match is None:
             return None
+        return int(match.group(1))
 
-    def _resolve_start_offset(self, stream: _RunStream, last_event_id: str | None) -> int:
+    @staticmethod
+    def _make_gap(stream: _RunStream, requested_event_id: str | None) -> StreamGap:
+        return StreamGap(
+            requested_event_id=requested_event_id,
+            earliest_available_event_id=stream.events[0].id,
+            latest_available_event_id=stream.events[-1].id,
+        )
+
+    def _resolve_start_offset(self, stream: _RunStream, last_event_id: str | None) -> int | StreamGap:
         if last_event_id is None:
             return stream.start_offset
 
         # Event ids embed a per-run, monotonically increasing ``seq`` that equals
         # the event's absolute offset, so locate the event by arithmetic in O(1)
-        # rather than scanning the retained buffer. The id is verified at the
-        # computed index, so a stale/evicted/foreign/malformed id still falls back
-        # to replay-from-earliest — identical to the previous linear scan.
+        # rather than scanning the retained buffer. Retained ids are verified at
+        # the computed index. Once an id is below the retained watermark there is
+        # nothing left to verify its timestamp against, so even a numeric foreign
+        # id takes the conservative gap path; reloading durable state is safer
+        # than silently claiming a complete replay. Unknown ids at or above the
+        # watermark keep the legacy replay-from-earliest behavior.
         seq = self._parse_event_seq(last_event_id)
         if seq is not None:
+            if stream.events and seq < stream.start_offset:
+                return self._make_gap(stream, last_event_id)
             local_index = seq - stream.start_offset
             if 0 <= local_index < len(stream.events) and stream.events[local_index].id == last_event_id:
                 return stream.start_offset + local_index + 1
@@ -115,39 +127,57 @@ class MemoryStreamBridge(StreamBridge):
         *,
         last_event_id: str | None = None,
         heartbeat_interval: float = 15.0,
-    ) -> AsyncIterator[StreamEvent]:
+    ) -> AsyncIterator[StreamItem]:
         stream = self._get_or_create_stream(run_id)
         async with stream.condition:
-            next_offset = self._resolve_start_offset(stream, last_event_id)
+            start = self._resolve_start_offset(stream, last_event_id)
+            if isinstance(start, StreamGap):
+                gap = start
+                next_offset = stream.start_offset
+            else:
+                gap = None
+                next_offset = start
+
+        if gap is not None:
+            yield gap
+            return
+
+        cursor_event_id = last_event_id
 
         while True:
             async with stream.condition:
                 if next_offset < stream.start_offset:
                     logger.warning(
-                        "subscriber for run %s fell behind retained buffer; resuming from offset %s",
+                        "subscriber for run %s fell behind retained buffer at offset %s",
                         run_id,
-                        stream.start_offset,
+                        next_offset,
                     )
-                    next_offset = stream.start_offset
-
-                local_index = next_offset - stream.start_offset
-                if 0 <= local_index < len(stream.events):
-                    entry = stream.events[local_index]
-                    next_offset += 1
-                elif stream.ended:
-                    entry = END_SENTINEL
+                    entry: StreamItem = self._make_gap(stream, cursor_event_id)
+                    should_stop = True
                 else:
-                    try:
-                        await asyncio.wait_for(stream.condition.wait(), timeout=heartbeat_interval)
-                    except TimeoutError:
-                        entry = HEARTBEAT_SENTINEL
+                    should_stop = False
+
+                    local_index = next_offset - stream.start_offset
+                    if 0 <= local_index < len(stream.events):
+                        entry = stream.events[local_index]
+                        next_offset += 1
+                        cursor_event_id = entry.id
+                    elif stream.ended:
+                        entry = END_SENTINEL
                     else:
-                        continue
+                        try:
+                            await asyncio.wait_for(stream.condition.wait(), timeout=heartbeat_interval)
+                        except TimeoutError:
+                            entry = HEARTBEAT_SENTINEL
+                        else:
+                            continue
 
             if entry is END_SENTINEL:
                 yield END_SENTINEL
                 return
             yield entry
+            if should_stop:
+                return
 
     async def cleanup(self, run_id: str, *, delay: float = 0) -> None:
         if delay > 0:

backend/packages/harness/deerflow/runtime/stream_bridge/redis.py

@@ -27,7 +27,7 @@ except ImportError:  # pragma: no cover - only hit when the optional extra is mi
         "Or switch to stream_bridge.type: memory in config.yaml for single-process deployment."
     ) from None
 
-from .base import END_SENTINEL, HEARTBEAT_SENTINEL, StreamBridge, StreamEvent
+from .base import END_SENTINEL, HEARTBEAT_SENTINEL, StreamBridge, StreamEvent, StreamGap, StreamItem
 
 logger = logging.getLogger(__name__)
 
@@ -139,6 +139,30 @@ class RedisStreamBridge(StreamBridge):
             data=self._decode_data(payload.get("data")),
         )
 
+    @classmethod
+    def _is_end_entry(cls, fields: Mapping[Any, Any]) -> bool:
+        return cls._normalise_fields(fields).get("kind") == _KIND_END
+
+    @staticmethod
+    def _parse_stream_id(event_id: str) -> tuple[int, int] | None:
+        if _REDIS_STREAM_ID_RE.fullmatch(event_id) is None:
+            return None
+        milliseconds, separator, sequence = event_id.partition("-")
+        return int(milliseconds), int(sequence) if separator else 0
+
+    @classmethod
+    def _stream_id_lt(cls, left: str, right: str) -> bool:
+        left_parts = cls._parse_stream_id(left)
+        right_parts = cls._parse_stream_id(right)
+        return left_parts is not None and right_parts is not None and left_parts < right_parts
+
+    @classmethod
+    def _response_tail_id(cls, response: list[Any]) -> str | None:
+        for _stream_name, entries in reversed(response):
+            if entries:
+                return cls._decode(entries[-1][0])
+        return None
+
     async def publish(self, run_id: str, event: str, data: Any) -> None:
         key = self._stream_key(run_id)
         await self._xadd_retained(
@@ -178,28 +202,63 @@ class RedisStreamBridge(StreamBridge):
             return "0-0"
         return self._decode(event_id)
 
+    async def _read_retained_snapshot(
+        self,
+        key: str,
+        stream_id: str,
+    ) -> tuple[list[Any], list[Any], list[Any]]:
+        """Atomically read retained bounds and entries after ``stream_id``.
+
+        A blocking ``XREAD`` cannot participate in a Redis transaction. Live
+        subscribers therefore use this non-blocking atomic snapshot for
+        correctness, and a separate blocking read only as a wake-up signal.
+        This deliberately adds a three-command pipeline per poll (and a second
+        round trip while idle) so retained-bound checks cannot race with reads.
+        """
+        async with self._redis.pipeline(transaction=True) as pipe:
+            pipe.xrange(key, count=1)
+            pipe.xrevrange(key, count=1)
+            pipe.xread({key: stream_id}, count=_XREAD_COUNT)
+            earliest, latest, response = await pipe.execute()
+        return earliest, latest, response
+
     async def subscribe(
         self,
         run_id: str,
         *,
         last_event_id: str | None = None,
         heartbeat_interval: float = 15.0,
-    ) -> AsyncIterator[StreamEvent]:
+    ) -> AsyncIterator[StreamItem]:
         key = self._stream_key(run_id)
         stream_id = await self._resolve_start_stream_id(key, last_event_id)
+        gap_detection_enabled = last_event_id is not None and self._parse_stream_id(last_event_id) is not None
+        pending_initial_response: list[Any] | None = None
         block_ms = max(1, int(heartbeat_interval * 1000)) if heartbeat_interval > 0 else 1
         consecutive_errors = 0
 
         while True:
+            snapshot_stream_id = stream_id
+            pending_initial_tail_id = None
+            if pending_initial_response is not None:
+                pending_initial_tail_id = self._response_tail_id(pending_initial_response)
+                if pending_initial_tail_id is None:
+                    pending_initial_response = None
+                else:
+                    # The first blocking XREAD began against a proven-empty
+                    # stream. Its response is a provisional live baseline, not
+                    # yet client-visible data. Validate that baseline against
+                    # the retained watermark before yielding any of it.
+                    snapshot_stream_id = pending_initial_tail_id
+
             try:
-                response = await self._redis.xread({key: stream_id}, count=_XREAD_COUNT, block=block_ms)
+                earliest_entries, latest_entries, response = await self._read_retained_snapshot(key, snapshot_stream_id)
             except ResponseError:
                 # Last-Event-ID is client-controlled and validated before XREAD.
                 # If Redis still rejects the id, fail instead of resetting to
                 # 0-0, which would replay the whole retained buffer on reconnect.
                 logger.warning(
                     "Redis rejected stream id %r for stream bridge subscription",
-                    stream_id,
+                    snapshot_stream_id,
                     exc_info=True,
                 )
                 raise
@@ -218,21 +277,93 @@ class RedisStreamBridge(StreamBridge):
                 await asyncio.sleep(delay)
                 continue
             else:
-                consecutive_errors = 0
+                # A non-empty snapshot is forward progress. For an empty
+                # snapshot, keep any preceding wake-up failure count until the
+                # blocking XREAD itself succeeds; otherwise a permanently
+                # failing blocking read could retry forever because the
+                # non-blocking transaction succeeds between attempts.
+                if response:
+                    consecutive_errors = 0
 
-            if not response:
-                yield HEARTBEAT_SENTINEL
+            if earliest_entries and (gap_detection_enabled or pending_initial_tail_id is not None):
+                earliest_id = self._decode(earliest_entries[0][0])
+                if self._stream_id_lt(snapshot_stream_id, earliest_id):
+                    latest_id = self._decode(latest_entries[0][0])
+                    logger.warning(
+                        "subscriber for Redis stream %s fell behind retained history at %s",
+                        key,
+                        snapshot_stream_id,
+                    )
+                    yield StreamGap(
+                        requested_event_id=None if pending_initial_tail_id is not None else stream_id,
+                        earliest_available_event_id=earliest_id,
+                        latest_available_event_id=latest_id,
+                    )
+                    return
+
+            responses_to_process = []
+            if pending_initial_response is not None:
+                responses_to_process.append(pending_initial_response)
+                pending_initial_response = None
+            if response:
+                responses_to_process.append(response)
+
+            if not responses_to_process:
+                if latest_entries and self._decode(latest_entries[0][0]) == stream_id and self._is_end_entry(latest_entries[0][1]):
+                    yield END_SENTINEL
+                    return
+
+                try:
+                    wake_response = await self._redis.xread(
+                        {key: stream_id},
+                        count=_XREAD_COUNT,
+                        block=block_ms,
+                    )
+                except ResponseError:
+                    logger.warning(
+                        "Redis rejected stream id %r for stream bridge subscription",
+                        stream_id,
+                        exc_info=True,
+                    )
+                    raise
+                except RedisError:
+                    consecutive_errors += 1
+                    if consecutive_errors > _MAX_SUBSCRIBE_RETRIES:
+                        raise
+                    delay = min(2**consecutive_errors, heartbeat_interval)
+                    logger.warning(
+                        "Transient Redis error in stream bridge subscriber (retry %d/%d); backing off %.1fs",
+                        consecutive_errors,
+                        _MAX_SUBSCRIBE_RETRIES,
+                        delay,
+                        exc_info=True,
+                    )
+                    await asyncio.sleep(delay)
+                    continue
+                else:
+                    consecutive_errors = 0
+
+                if not wake_response:
+                    yield HEARTBEAT_SENTINEL
+                elif last_event_id is None and not gap_detection_enabled and not earliest_entries:
+                    # Do not discard the first wake-up from a proven-empty
+                    # stream. Holding it until the next atomic bounds check
+                    # gives no-cursor subscribers the same fell-behind signal
+                    # as Memory without changing malformed-cursor live tailing.
+                    pending_initial_response = wake_response
                 continue
 
-            for _stream_name, entries in response:
-                for event_id, fields in entries:
-                    event_id = self._decode(event_id)
-                    stream_id = event_id
-                    entry = self._entry_from_redis(event_id, fields)
-                    if entry is END_SENTINEL:
-                        yield END_SENTINEL
-                        return
-                    yield entry
+            for retained_response in responses_to_process:
+                for _stream_name, entries in retained_response:
+                    for event_id, fields in entries:
+                        event_id = self._decode(event_id)
+                        stream_id = event_id
+                        gap_detection_enabled = True
+                        entry = self._entry_from_redis(event_id, fields)
+                        if entry is END_SENTINEL:
+                            yield END_SENTINEL
+                            return
+                        yield entry
 
     async def cleanup(self, run_id: str, *, delay: float = 0) -> None:
         if delay > 0:

backend/packages/harness/deerflow/sandbox/local/local_sandbox_provider.py

@@ -136,6 +136,7 @@ class LocalSandboxProvider(SandboxProvider):
             _RESERVED_CONTAINER_PREFIXES = [
                 f"{container_path}/public",
                 f"{container_path}/custom",
+                f"{container_path}/integrations",
                 f"{container_path}/legacy",
                 _ACP_WORKSPACE_VIRTUAL_PREFIX,
                 _USER_DATA_VIRTUAL_PREFIX,
@@ -287,7 +288,9 @@ class LocalSandboxProvider(SandboxProvider):
             config = get_app_config()
             skills_container_path = config.skills.container_path
             user_custom_path = paths.user_custom_skills_dir(effective_user_id)
+            integrations_path = paths.integration_skills_dir()
             user_custom_path.mkdir(parents=True, exist_ok=True)
+            integrations_path.mkdir(parents=True, exist_ok=True)
 
             mappings.append(
                 PathMapping(
@@ -296,6 +299,13 @@ class LocalSandboxProvider(SandboxProvider):
                     read_only=True,
                 )
             )
+            mappings.append(
+                PathMapping(
+                    container_path=f"{skills_container_path}/integrations",
+                    local_path=str(integrations_path),
+                    read_only=True,
+                )
+            )
         except Exception as exc:
             logger.warning("Could not setup per-thread custom skills mount: %s", exc, exc_info=True)
 

backend/packages/harness/deerflow/sandbox/middleware.py

@@ -9,7 +9,7 @@ from langchain.agents.middleware import AgentMiddleware
 from langchain_core.messages import ToolMessage
 from langgraph.prebuilt.tool_node import ToolCallRequest
 from langgraph.runtime import Runtime
-from langgraph.types import Command
+from langgraph.types import Command, Overwrite
 
 from deerflow.agents.thread_state import SandboxStateField, ThreadDataState
 from deerflow.runtime.user_context import resolve_runtime_user_id
@@ -25,6 +25,24 @@ class SandboxMiddlewareState(AgentState):
     thread_data: NotRequired[ThreadDataState | None]
 
 
+def _unwrap_sandbox(sandbox: object) -> tuple[object, bool]:
+    """Unwrap an ``Overwrite``-wrapped sandbox channel value, if present.
+
+    Fork-restored checkpoints can deliver the sandbox channel still wrapped in
+    ``langgraph.types.Overwrite`` (the rollback restore applies replace-style
+    writes through a state-mutation graph in delta checkpoint mode). Reading
+    ``sandbox["sandbox_id"]`` on the wrapper itself crashes with ``TypeError:
+    'Overwrite' object is not subscriptable``, so unwrap before use.
+
+    Returns ``(value, fork_restored)``. The wrapped form replays the parent
+    thread's sandbox state, so callers must not treat the sandbox as owned by
+    this run (e.g. release it).
+    """
+    if isinstance(sandbox, Overwrite):
+        return sandbox.value, True
+    return sandbox, False
+
+
 class SandboxMiddleware(AgentMiddleware[SandboxMiddlewareState]):
     """Create a sandbox environment and assign it to an agent.
 
@@ -99,9 +117,14 @@ class SandboxMiddleware(AgentMiddleware[SandboxMiddlewareState]):
 
     @override
     def after_agent(self, state: SandboxMiddlewareState, runtime: Runtime) -> dict | None:
-        sandbox = state.get("sandbox")
+        sandbox, fork_restored = _unwrap_sandbox(state.get("sandbox"))
         if sandbox is not None:
             sandbox_id = sandbox["sandbox_id"]
+            if fork_restored:
+                # The wrapped value replays the parent thread's sandbox state;
+                # releasing it here would evict the parent's warm sandbox.
+                logger.info(f"Not releasing fork-restored sandbox {sandbox_id}")
+                return None
             logger.info(f"Releasing sandbox {sandbox_id}")
             get_sandbox_provider().release(sandbox_id)
             return None
@@ -117,9 +140,14 @@ class SandboxMiddleware(AgentMiddleware[SandboxMiddlewareState]):
 
     @override
     async def aafter_agent(self, state: SandboxMiddlewareState, runtime: Runtime) -> dict | None:
-        sandbox = state.get("sandbox")
+        sandbox, fork_restored = _unwrap_sandbox(state.get("sandbox"))
         if sandbox is not None:
             sandbox_id = sandbox["sandbox_id"]
+            if fork_restored:
+                # The wrapped value replays the parent thread's sandbox state;
+                # releasing it here would evict the parent's warm sandbox.
+                logger.info(f"Not releasing fork-restored sandbox {sandbox_id}")
+                return None
             logger.info(f"Releasing sandbox {sandbox_id}")
             await self._release_sandbox_async(sandbox_id)
             return None

backend/packages/harness/deerflow/sandbox/tools.py

@@ -163,6 +163,7 @@ def _extract_skill_name_from_skills_path(path: str) -> str | None:
     /mnt/skills/public/bootstrap/SKILL.md → "bootstrap"
     /mnt/skills/custom/my-skill/SKILL.md → "my-skill"
     /mnt/skills/legacy/my-skill/references/... → "my-skill"
+    /mnt/skills/integrations/lark-cli/lark-doc/SKILL.md → "lark-doc"
     /mnt/skills/public/bootstrap/ → "bootstrap"
     Returns None if the path doesn't contain a recognizable skill name pattern.
     """
@@ -173,16 +174,22 @@ def _extract_skill_name_from_skills_path(path: str) -> str | None:
     relative = path[len(skills_prefix) :].lstrip("/")
     if not relative:
         return None
-    # Expected patterns: "public/<name>/...", "custom/<name>/...", "legacy/<name>/..."
+    # Expected patterns: "public/<name>/...", "custom/<name>/...",
+    # "legacy/<name>/...", "integrations/<provider>/<name>/..."
     # or "<name>/..." (direct skill access). Empty segments are dropped so a
     # directory entry ("public/", as `ls` emits for dirs) is still recognized as
     # a category root rather than yielding an empty skill name.
     parts = [part for part in relative.split("/") if part]
     if len(parts) >= 2 and parts[0] in ("public", "custom", "legacy"):
         return parts[1]
-    if len(parts) == 1 and parts[0] in ("public", "custom", "legacy"):
+    if len(parts) >= 3 and parts[0] == "integrations":
+        return parts[2]
+    if len(parts) == 1 and parts[0] in ("public", "custom", "legacy", "integrations"):
         # Category root like /mnt/skills/custom — not a skill path.
         return None
+    if len(parts) == 2 and parts[0] == "integrations":
+        # Provider root like /mnt/skills/integrations/lark-cli.
+        return None
     if len(parts) >= 1:
         # Direct path like /mnt/skills/my-skill/SKILL.md
         return parts[0]
@@ -215,6 +222,8 @@ def _is_disabled_skill_path(path: str, *, user_id: str | None = None) -> bool:
             category = "custom"
         elif relative.startswith("legacy/"):
             category = "legacy"
+        elif relative.startswith("integrations/"):
+            category = "integrations"
         else:
             # Try to infer from storage
             effective_uid = user_id or get_effective_user_id()
@@ -307,7 +316,7 @@ def _resolve_skills_path(path: str) -> str:
 
     relative = path[len(skills_container) :].lstrip("/")
 
-    # Per-user custom skills: resolve to user-specific directory.
+    # Per-user custom and globally managed integration skills.
     # ``skill_manage_tool`` writes custom skills to the per-user directory,
     # and ``LocalSandboxProvider._build_thread_path_mappings`` mounts
     # ``/mnt/skills/custom`` to that same per-user dir.  Without this
@@ -327,6 +336,22 @@ def _resolve_skills_path(path: str) -> str:
             return str(user_custom_dir / custom_relative)
         return str(user_custom_dir)
 
+    if relative == "integrations" or relative.startswith("integrations/"):
+        from deerflow.config.paths import get_paths
+
+        paths = get_paths()
+        integrations_dir = paths.integration_skills_dir()
+        integrations_relative = relative[len("integrations") :].lstrip("/")
+        if not integrations_relative:
+            return str(integrations_dir)
+        # Defense-in-depth: even though _reject_path_traversal runs upstream for
+        # sandbox callers, confirm the resolved path stays within the global
+        # integration dir so a lexical ``../`` cannot escape it here.
+        resolved = (integrations_dir / integrations_relative).resolve()
+        if not resolved.is_relative_to(integrations_dir.resolve()):
+            raise PermissionError("Access denied: path traversal detected")
+        return str(integrations_dir / integrations_relative)
+
     return _join_path_preserving_style(skills_host, relative)
 
 
@@ -758,11 +783,11 @@ def mask_local_paths_in_output(output: str, thread_data: ThreadDataState | None)
     """Mask host absolute paths from local sandbox output using virtual paths.
 
     Handles user-data paths (per-thread), skills paths (global + per-user
-    custom), and ACP workspace paths (per-thread).
+    custom + managed integrations), and ACP workspace paths (per-thread).
     """
     # Build the ordered (host_base, virtual_base) source list. Order is
     # preserved from the original implementation: skills, then per-user
-    # custom skills, then ACP workspace, then user-data mappings (longest
+    # custom/integration skills, then ACP workspace, then user-data mappings (longest
     # host path first). Custom mount host paths are masked by
     # LocalSandbox._reverse_resolve_paths_in_output().
     sources: list[tuple[str, str]] = []
@@ -782,9 +807,13 @@ def mask_local_paths_in_output(output: str, thread_data: ThreadDataState | None)
 
         user_id = get_effective_user_id()
         user_custom_dir = get_paths().user_custom_skills_dir(user_id)
+        integrations_dir = get_paths().integration_skills_dir()
         if user_custom_dir.exists():
             skills_container = _get_skills_container_path()
             sources.append((str(user_custom_dir), f"{skills_container}/custom"))
+        if integrations_dir.exists():
+            skills_container = _get_skills_container_path()
+            sources.append((str(integrations_dir), f"{skills_container}/integrations"))
     except Exception:
         pass
 
@@ -1697,6 +1726,30 @@ def _github_env_from_runtime(runtime: Runtime) -> dict[str, str] | None:
     return {"GH_TOKEN": token, "GITHUB_TOKEN": token}
 
 
+_LARK_CLI_COMMAND_RE = re.compile(r"(?<![A-Za-z0-9_.-])lark-cli(?![A-Za-z0-9_.-])")
+
+
+def _lark_cli_env_from_runtime(runtime: Runtime, command: str, *, sandbox_paths: bool) -> dict[str, str] | None:
+    """Expose Settings-page Lark auth to sandbox ``lark-cli`` commands.
+
+    Settings authorizes ``lark-cli`` under DeerFlow's per-user integration
+    config/data directories. Agent conversations invoke ``lark-cli`` through the
+    sandbox, so lark commands must receive those same directories or they see an
+    unrelated unauthenticated profile. Keep this scoped to commands that
+    actually call ``lark-cli`` so ordinary bash calls do not switch AIO into the
+    env-bearing execution path.
+    """
+    if not _LARK_CLI_COMMAND_RE.search(command):
+        return None
+    try:
+        from deerflow.integrations.lark_cli import lark_cli_env_overlay
+
+        return lark_cli_env_overlay(resolve_runtime_user_id(runtime), sandbox_paths=sandbox_paths)
+    except Exception:
+        logger.warning("Could not build Lark CLI env overlay; running command without managed auth", exc_info=True)
+        return None
+
+
 @tool("bash", parse_docstring=True)
 def bash_tool(runtime: Runtime, description: str, command: str) -> str:
     """Execute a bash command in a Linux environment.
@@ -1723,8 +1776,11 @@ def bash_tool(runtime: Runtime, description: str, command: str) -> str:
         injected_env = read_active_secrets(getattr(runtime, "context", None)) or None
         identity_prefix = _channel_identity_prefix(runtime)
         github_env = _github_env_from_runtime(runtime)
+        lark_cli_env = _lark_cli_env_from_runtime(runtime, command, sandbox_paths=not is_local_sandbox(runtime))
         if github_env:
             injected_env = {**(injected_env or {}), **github_env}
+        if lark_cli_env:
+            injected_env = {**(injected_env or {}), **lark_cli_env}
         if is_local_sandbox(runtime):
             if not is_host_bash_allowed():
                 return f"Error: {LOCAL_HOST_BASH_DISABLED_MESSAGE}"

backend/packages/harness/deerflow/skills/storage/user_scoped_skill_storage.py

@@ -8,6 +8,7 @@ Layout::
 
     <host_root>/public/<name>/SKILL.md            ← global, read-only
     <user_custom_root>/<name>/SKILL.md             ← per-user, read-write
+    <user_integrations_root>/<provider>/<name>/SKILL.md  ← per-user, read-only
     <user_custom_root>/.history/<name>.jsonl       ← per-user history
     <user_skills_root>/_skill_states.json          ← per-user enabled state
     <global_custom_root>/<name>/SKILL.md           ← legacy fallback, read-only
@@ -82,6 +83,7 @@ class UserScopedSkillStorage(LocalSkillStorage):
         self._user_id = _validate_user_id(user_id)
         paths = get_paths()
         self._user_custom_root: Path = paths.user_custom_skills_dir(self._user_id)
+        self._integrations_root: Path = paths.integration_skills_dir()
         self._user_skills_root: Path = paths.user_skills_dir(self._user_id)
         self._global_custom_root: Path = self._host_root / SkillCategory.CUSTOM.value
         self._skill_states_file: Path = self._user_skills_root / "_skill_states.json"
@@ -256,8 +258,11 @@ class UserScopedSkillStorage(LocalSkillStorage):
         is_global_custom_fallback = (self._global_custom_root / normalized_name / SKILL_MD_FILE).exists()
         if is_global_public:
             raise ValueError(f"'{name}' is a built-in skill. Use the skill_manage tool to create your own version — it will shadow the built-in one.")
+        is_integration = any((candidate / SKILL_MD_FILE).exists() for candidate in self._integrations_root.glob(f"*/{normalized_name}") if candidate.is_dir())
         if is_global_custom_fallback:
             raise ValueError(f"'{name}' is a legacy shared skill (not editable). To customise it, create your own version with the same name — it will shadow the shared one.")
+        if is_integration:
+            raise ValueError(f"'{name}' is a managed integration skill and cannot be edited. Create a custom skill with another name if you need a modified workflow.")
         raise FileNotFoundError(f"Custom skill '{name}' not found.")
 
     def _iter_skill_files(self) -> Iterable[tuple[SkillCategory, Path, Path]]:
@@ -271,7 +276,17 @@ class UserScopedSkillStorage(LocalSkillStorage):
                 dir_names.clear()
                 yield SkillCategory.PUBLIC, public_path, Path(current_root) / SKILL_MD_FILE
 
-        # 2. Custom skills: prefer user-level directory
+        # 2. Managed integration skills: globally installed, read-only. Their
+        # enabled state is still merged from this user's _skill_states.json.
+        integration_path = self._integrations_root
+        if integration_path.exists() and integration_path.is_dir():
+            for current_root, dir_names, file_names in os.walk(integration_path, followlinks=True):
+                dir_names[:] = sorted(name for name in dir_names if not name.startswith("."))
+                if SKILL_MD_FILE not in file_names:
+                    continue
+                yield SkillCategory.INTEGRATION, integration_path, Path(current_root) / SKILL_MD_FILE
+
+        # 3. Custom skills: prefer user-level directory
         user_custom_exists = False
         user_custom_path = self._user_custom_root
         if user_custom_path.exists() and user_custom_path.is_dir():
@@ -283,7 +298,7 @@ class UserScopedSkillStorage(LocalSkillStorage):
                 user_custom_exists = True
                 yield SkillCategory.CUSTOM, user_custom_path, Path(current_root) / SKILL_MD_FILE
 
-        # 3. Fallback: if user has no custom skills, load from global custom
+        # 4. Fallback: if user has no custom skills, load from global custom
         #    as LEGACY (read-only) so legacy skills are visible but not
         #    editable/deletable by the user. LEGACY skills are mounted at
         #    /mnt/skills/legacy/<name>/ in the sandbox so their supporting
@@ -370,6 +385,10 @@ class UserScopedSkillStorage(LocalSkillStorage):
         """Host path to this user's custom skills root directory."""
         return self._user_custom_root
 
+    def get_user_integrations_root(self) -> Path:
+        """Host path to this user's managed integration skills root directory."""
+        return self._user_integrations_root
+
     # ------------------------------------------------------------------
     # Path validation — accept per-user custom root as well as global root
     # ------------------------------------------------------------------
@@ -382,7 +401,7 @@ class UserScopedSkillStorage(LocalSkillStorage):
         would reject them.  This override allows both roots.
         """
         resolved_file = skill_file.resolve()
-        for allowed_root in (self._host_root.resolve(), self._user_custom_root.resolve()):
+        for allowed_root in (self._host_root.resolve(), self._user_custom_root.resolve(), self._user_integrations_root.resolve()):
             try:
                 resolved_file.relative_to(allowed_root)
                 return resolved_file

backend/packages/harness/deerflow/skills/types.py

@@ -12,6 +12,7 @@ class SkillCategory(StrEnum):
 
     - ``PUBLIC``: built-in skill bundled with the platform, read-only.
     - ``CUSTOM``: user-authored skill that can be edited or deleted.
+    - ``INTEGRATION``: managed third-party integration skill, read-only.
     - ``LEGACY``: global custom skill from before user-isolation migration,
       presented as read-only (visible but not editable/deletable). These
       skills are mounted at ``/mnt/skills/legacy/<name>/`` in the sandbox.
@@ -19,6 +20,7 @@ class SkillCategory(StrEnum):
 
     PUBLIC = "public"
     CUSTOM = "custom"
+    INTEGRATION = "integrations"
     LEGACY = "legacy"
 
 

backend/tests/_replay_fixture.py

@@ -156,7 +156,10 @@ def drive_gateway(app, *, prompt: str, context: dict) -> list[dict]:
         body = {
             "assistant_id": "lead_agent",
             "input": {"messages": [{"role": "user", "content": prompt}]},
-            "config": {"recursion_limit": 50},
+            # Keep replay close to the Gateway default. A tighter limit can
+            # produce false golden drift when protocol-neutral middleware adds
+            # graph steps without changing the streamed SSE contract.
+            "config": {"recursion_limit": 100},
             "context": context,
             "stream_mode": ["values"],
         }

backend/tests/blocking_io/test_aio_sandbox_get.py

@@ -82,6 +82,8 @@ def _make_provider(tmp_path: Path):
     provider._thread_locks = {}
     provider._last_activity = {}
     provider._warm_pool = {}
+    provider._active_sandbox_identity = {}
+    provider._warm_pool_identity = {}
     provider._local_teardown = set()
     provider._acquire_epoch = {}
     provider._acquire_epoch_counter = 0

backend/tests/blocking_io/test_integrations_router.py

@@ -0,0 +1,130 @@
+"""Regression anchors: integrations router must not block the event loop.
+
+The Lark integration handlers are async FastAPI route handlers, but the work
+they dispatch includes zip reads, filesystem staging, manifest writes, and
+``lark-cli`` subprocess calls. Those phases must stay behind
+``asyncio.to_thread``; if a future refactor runs them inline, the strict
+Blockbuster gate raises ``BlockingError`` and these anchors fail.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import os
+import zipfile
+from pathlib import Path
+from types import SimpleNamespace
+
+import pytest
+
+from app.gateway.routers import integrations
+from deerflow.config import paths as paths_module
+from deerflow.integrations import lark_cli
+
+pytestmark = pytest.mark.asyncio
+
+
+def _skill_content(name: str) -> str:
+    return f"---\nname: {name}\ndescription: {name} integration skill\n---\n\n# {name}\n"
+
+
+def _build_lark_archive(archive: Path) -> None:
+    archive.parent.mkdir(parents=True, exist_ok=True)
+    with zipfile.ZipFile(archive, "w") as zf:
+        for skill_name in lark_cli.LARK_SKILL_NAMES:
+            zf.writestr(f"cli-1.0.65/skills/{skill_name}/SKILL.md", _skill_content(skill_name))
+            zf.writestr(f"cli-1.0.65/skills/{skill_name}/references/readme.md", f"# {skill_name}\n")
+
+
+def _write_stub_lark_cli(path: Path) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_text(
+        """#!/bin/sh
+if [ "$1" = "--version" ]; then
+  echo "v1.0.65"
+  exit 0
+fi
+
+if [ "$1" = "auth" ] && [ "$2" = "login" ]; then
+  echo "{}"
+  exit 0
+fi
+
+if [ "$1" = "auth" ] && [ "$2" = "status" ]; then
+  echo '{"identities":{"user":{"userName":"Alice"}}}'
+  exit 0
+fi
+
+echo "{}"
+exit 0
+""",
+        encoding="utf-8",
+    )
+    path.chmod(0o755)
+
+
+def _reset_paths(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setenv("DEER_FLOW_HOME", str(tmp_path / "home"))
+    monkeypatch.setattr(paths_module, "_paths", None)
+
+
+async def _config(tmp_path: Path) -> SimpleNamespace:
+    skills_root = tmp_path / "skills"
+    await asyncio.to_thread((skills_root / "public").mkdir, parents=True, exist_ok=True)
+    await asyncio.to_thread((skills_root / "custom").mkdir, parents=True, exist_ok=True)
+    return SimpleNamespace(
+        skills=SimpleNamespace(
+            get_skills_path=lambda: skills_root,
+            container_path="/mnt/skills",
+            use="deerflow.skills.storage.local_skill_storage:LocalSkillStorage",
+        )
+    )
+
+
+async def test_lark_install_route_does_not_block_event_loop(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+    _reset_paths(tmp_path, monkeypatch)
+    config = await _config(tmp_path)
+    archive = tmp_path / "fixtures" / "lark-cli.zip"
+    await asyncio.to_thread(_build_lark_archive, archive)
+
+    async def _allow_admin(*_args, **_kwargs) -> None:
+        return None
+
+    refresh_calls = 0
+
+    async def _refresh_cache() -> None:
+        nonlocal refresh_calls
+        refresh_calls += 1
+
+    monkeypatch.setenv(lark_cli.LARK_CLI_SOURCE_ARCHIVE_ENV, str(archive))
+    monkeypatch.setattr(lark_cli, "probe_lark_cli", lambda: lark_cli.LarkCliProbe(available=True, path="/usr/bin/lark-cli", version="v1.0.65"))
+    monkeypatch.setattr(lark_cli, "probe_lark_auth", lambda _user_id, **_kwargs: lark_cli.LarkAuthProbe(status="not_configured", message="not configured"))
+    monkeypatch.setattr(integrations, "get_effective_user_id", lambda: "loop-user")
+    monkeypatch.setattr(integrations, "require_admin_user", _allow_admin)
+    monkeypatch.setattr(integrations, "refresh_skills_system_prompt_cache_async", _refresh_cache, raising=False)
+
+    response = await integrations.install_lark(request=None, config=config)
+
+    assert response.success is True
+    assert refresh_calls == 1
+    install_root = await asyncio.to_thread(lark_cli.lark_integration_root, "loop-user")
+    assert await asyncio.to_thread((install_root / "lark-doc" / "SKILL.md").exists)
+
+
+async def test_lark_auth_complete_route_does_not_block_event_loop(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+    _reset_paths(tmp_path, monkeypatch)
+    config = await _config(tmp_path)
+    cli_path = tmp_path / "bin" / "lark-cli"
+    await asyncio.to_thread(_write_stub_lark_cli, cli_path)
+
+    monkeypatch.setenv("PATH", f"{cli_path.parent}{os.pathsep}{os.environ.get('PATH', '')}")
+    monkeypatch.setattr(integrations, "get_effective_user_id", lambda: "loop-user")
+
+    response = await integrations.complete_lark_browser_auth(
+        request=None,
+        body=integrations.LarkAuthCompleteRequest(device_code="device-code"),
+        config=config,
+    )
+
+    assert response.status.cli.available is True
+    assert response.status.cli.version == "v1.0.65"

backend/tests/blocking_io/test_jsonl_run_event_store.py

@@ -38,7 +38,7 @@ async def test_jsonl_run_event_store_async_api_does_not_block_event_loop(tmp_pat
     thread_dir.mkdir(parents=True, exist_ok=True)
     (thread_dir / "r0.jsonl").write_text('{"seq": 1, "category": "message", "run_id": "r0"}\n', encoding="utf-8")
 
-    # writes: put + put_batch
+    # writes: put + put_batch + idempotent singleton insert
     record = await store.put(thread_id="t1", run_id="r1", event_type="message", category="message", content="hi")
     assert record["seq"] >= 2
     batch = await store.put_batch(
@@ -48,6 +48,23 @@ async def test_jsonl_run_event_store_async_api_does_not_block_event_loop(tmp_pat
         ]
     )
     assert len(batch) == 2
+    singleton, created = await store.put_if_absent(
+        thread_id="t1",
+        run_id="r1",
+        event_type="run.delivery",
+        category="outputs",
+        content={"presented": 0, "paths": [], "by_tool": {}},
+    )
+    duplicate, duplicate_created = await store.put_if_absent(
+        thread_id="t1",
+        run_id="r1",
+        event_type="run.delivery",
+        category="outputs",
+        content={"presented": 99},
+    )
+    assert created is True
+    assert duplicate_created is False
+    assert duplicate == singleton
 
     # reads: list_messages / list_events / list_messages_by_run / count_messages.
     # list_events is exercised both without and with the event_types filter so

backend/tests/blocking_io/test_run_journal_callbacks.py

@@ -0,0 +1,39 @@
+"""Regression anchor: inline RunJournal callbacks stay event-loop safe."""
+
+from __future__ import annotations
+
+from uuid import uuid4
+
+import pytest
+from langchain_core.callbacks.manager import ahandle_event
+from langchain_core.messages import AIMessage, ToolMessage
+from langgraph.types import Command
+
+from deerflow.runtime.events.store.memory import MemoryRunEventStore
+from deerflow.runtime.journal import RunJournal
+
+pytestmark = pytest.mark.asyncio
+
+
+async def test_inline_tool_callback_does_not_block_event_loop() -> None:
+    journal = RunJournal("run-1", "thread-1", MemoryRunEventStore(), flush_threshold=100)
+    journal._remember_current_run_tool_calls(
+        AIMessage(content="", tool_calls=[{"id": "call-1", "name": "present_files", "args": {}}]),
+        caller="lead_agent",
+    )
+    command = Command(
+        update={
+            "artifacts": ["/mnt/user-data/outputs/report.md"],
+            "messages": [ToolMessage("Successfully presented files", tool_call_id="call-1")],
+        }
+    )
+
+    await ahandle_event(
+        [journal],
+        "on_tool_end",
+        "ignore_agent",
+        command,
+        run_id=uuid4(),
+    )
+
+    assert journal.get_delivery_content()["paths"] == ["/mnt/user-data/outputs/report.md"]

backend/tests/blocking_io/test_sandbox_release.py

@@ -75,6 +75,7 @@ def _make_provider_with_active_sandbox(tmp_path: Path, sandbox_id: str):
     provider = AioSandboxProvider.__new__(AioSandboxProvider)
     provider._lock = threading.Lock()
     provider._sandboxes = {sandbox_id: MagicMock()}
+    provider._active_sandbox_identity = {sandbox_id: ("default", "thread-1")}
     provider._sandbox_infos = {
         sandbox_id: SandboxInfo(
             sandbox_id=sandbox_id,
@@ -87,6 +88,7 @@ def _make_provider_with_active_sandbox(tmp_path: Path, sandbox_id: str):
     provider._thread_locks = {}
     provider._last_activity = {sandbox_id: 1.0}
     provider._warm_pool = {}
+    provider._warm_pool_identity = {}
     provider._unowned_since = {}
     provider._local_teardown = set()
     provider._acquire_epoch = {}

backend/tests/test_aio_sandbox.py

@@ -7,6 +7,47 @@ from unittest.mock import MagicMock, patch
 import pytest
 
 
+def test_local_sandbox_client_bypasses_environment_proxy():
+    """Local sandbox API calls must not inherit HTTP_PROXY (#3441)."""
+    from deerflow.community.aio_sandbox.aio_sandbox import AioSandbox
+
+    sentinel_httpx = MagicMock()
+    with (
+        patch("deerflow.community.aio_sandbox.aio_sandbox.httpx.Client", return_value=sentinel_httpx) as client_cls,
+        patch("deerflow.community.aio_sandbox.aio_sandbox.AioSandboxClient") as sdk_cls,
+    ):
+        AioSandbox(id="test-sandbox", base_url="http://host.docker.internal:8080")
+
+    client_cls.assert_called_once_with(timeout=600, follow_redirects=True, trust_env=False)
+    sdk_cls.assert_called_once_with(
+        base_url="http://host.docker.internal:8080",
+        timeout=600,
+        httpx_client=sentinel_httpx,
+    )
+
+
+@pytest.mark.parametrize(
+    "base_url",
+    [
+        "https://sandbox.example.com",
+        "http://8.8.8.8:8080",
+        "http://[2606:4700:4700::1111]:8080",
+    ],
+)
+def test_external_sandbox_client_keeps_environment_proxy_support(base_url: str):
+    """Externally hosted sandbox URLs retain the SDK's default proxy behavior."""
+    from deerflow.community.aio_sandbox.aio_sandbox import AioSandbox
+
+    with (
+        patch("deerflow.community.aio_sandbox.aio_sandbox.httpx.Client") as client_cls,
+        patch("deerflow.community.aio_sandbox.aio_sandbox.AioSandboxClient") as sdk_cls,
+    ):
+        AioSandbox(id="test-sandbox", base_url=base_url)
+
+    client_cls.assert_not_called()
+    sdk_cls.assert_called_once_with(base_url=base_url, timeout=600)
+
+
 @pytest.fixture()
 def sandbox():
     """Create an AioSandbox with a mocked client."""

backend/tests/test_aio_sandbox_provider.py

@@ -1,7 +1,10 @@
 """Tests for AioSandboxProvider mount helpers."""
 
 import asyncio
+import contextlib
+import hashlib
 import importlib
+import stat
 from types import SimpleNamespace
 from unittest.mock import MagicMock, patch
 
@@ -10,6 +13,11 @@ import pytest
 from deerflow.config.paths import Paths, join_host_path
 from deerflow.runtime.user_context import reset_current_user, set_current_user
 
+_LEGACY_COLLIDING_IDENTITIES = (
+    ("user-9721", "thread-9721"),
+    ("user-94361", "thread-94361"),
+)
+
 # ── ensure_thread_dirs ───────────────────────────────────────────────────────
 
 
@@ -61,6 +69,8 @@ def _make_provider(tmp_path):
         provider = aio_mod.AioSandboxProvider.__new__(aio_mod.AioSandboxProvider)
         provider._config = {"idle_timeout": 600, "replicas": 3}
         provider._sandboxes = {}
+        provider._active_sandbox_identity = {}
+        provider._warm_pool_identity = {}
         provider._local_teardown = set()
         provider._acquire_epoch = {}
         provider._acquire_epoch_counter = 0
@@ -119,6 +129,108 @@ def test_get_thread_mounts_uses_explicit_user_id(tmp_path, monkeypatch):
     assert container_paths["/mnt/user-data/outputs"] == str(tmp_path / "users" / "ou-user" / "threads" / "thread-4" / "user-data" / "outputs")
 
 
+def test_get_lark_cli_runtime_mounts_uses_user_auth_dirs(tmp_path, monkeypatch):
+    """Sandbox lark-cli commands must read the same auth dirs as Settings."""
+    aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
+    lark_cli = importlib.import_module("deerflow.integrations.lark_cli")
+    monkeypatch.setattr(aio_mod, "get_paths", lambda: Paths(base_dir=tmp_path))
+    monkeypatch.setattr(aio_mod, "get_effective_user_id", lambda: "default")
+    runtime_dir = tmp_path / "integrations" / "lark-cli" / "sandbox-cli"
+    runtime_dir.mkdir(parents=True)
+
+    mounts = aio_mod.AioSandboxProvider._get_lark_cli_runtime_mounts(user_id="alice")
+    container_paths = {container_path: (host_path, read_only) for host_path, container_path, read_only in mounts}
+
+    assert container_paths[lark_cli.LARK_CLI_SANDBOX_CONFIG_DIR] == (
+        str(tmp_path / "users" / "alice" / "integrations" / "lark-cli" / "config"),
+        True,
+    )
+    assert container_paths[lark_cli.LARK_CLI_SANDBOX_DATA_DIR] == (
+        str(tmp_path / "users" / "alice" / "integrations" / "lark-cli" / "data"),
+        False,
+    )
+    assert stat.S_IMODE((tmp_path / "users" / "alice" / "integrations" / "lark-cli" / "config").stat().st_mode) == 0o700
+    assert stat.S_IMODE((tmp_path / "users" / "alice" / "integrations" / "lark-cli" / "data").stat().st_mode) == 0o700
+    assert container_paths["/mnt/integrations/lark-cli/runtime"] == (
+        str(runtime_dir),
+        True,
+    )
+
+
+def test_get_user_skill_mounts_mounts_only_global_integrations(tmp_path, monkeypatch):
+    aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
+    skills_root = tmp_path / "skills"
+    (skills_root / "public").mkdir(parents=True)
+    config = SimpleNamespace(
+        skills=SimpleNamespace(
+            get_skills_path=lambda: skills_root,
+            container_path="/mnt/skills",
+        )
+    )
+    monkeypatch.setattr(aio_mod, "get_app_config", lambda: config)
+    monkeypatch.setattr(aio_mod, "get_paths", lambda: Paths(base_dir=tmp_path / "home"))
+
+    alice = {container: host for host, container, _read_only in aio_mod.AioSandboxProvider._get_user_skill_mounts(user_id="alice")}
+    bob = {container: host for host, container, _read_only in aio_mod.AioSandboxProvider._get_user_skill_mounts(user_id="bob")}
+
+    assert set(alice) == {"/mnt/skills/integrations"}
+    assert set(bob) == {"/mnt/skills/integrations"}
+    assert alice["/mnt/skills/integrations"] == bob["/mnt/skills/integrations"]
+    assert alice["/mnt/skills/integrations"] == str(tmp_path / "home" / "integrations" / "skills")
+
+
+def test_get_extra_mounts_provisioner_payload_has_unique_container_paths(tmp_path, monkeypatch, provisioner_module):
+    """Full AIO mount composition must not send duplicate paths to provisioner."""
+    aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
+    lark_cli = importlib.import_module("deerflow.integrations.lark_cli")
+    remote_backend = importlib.import_module("deerflow.community.aio_sandbox.remote_backend")
+    skills_root = tmp_path / "skills"
+    (skills_root / "public").mkdir(parents=True)
+    home = tmp_path / "home"
+    config = SimpleNamespace(
+        skills=SimpleNamespace(
+            get_skills_path=lambda: skills_root,
+            container_path="/mnt/skills",
+        )
+    )
+    runtime_dir = home / "integrations" / "lark-cli" / "sandbox-cli"
+    runtime_dir.mkdir(parents=True)
+
+    monkeypatch.setattr(aio_mod, "get_app_config", lambda: config)
+    monkeypatch.setattr(aio_mod, "get_paths", lambda: Paths(base_dir=home))
+    monkeypatch.setattr(aio_mod, "get_effective_user_id", lambda: "default")
+    monkeypatch.setattr(aio_mod, "user_should_see_legacy_skills", lambda *_args, **_kwargs: False)
+
+    provider = _make_provider(tmp_path)
+    mounts = provider._get_extra_mounts("thread-1", user_id="alice")
+    container_paths = [container for _host, container, _read_only in mounts]
+
+    assert len(container_paths) == len(set(container_paths))
+    assert "/mnt/skills/custom" in container_paths
+    assert "/mnt/skills/integrations" in container_paths
+    assert lark_cli.LARK_CLI_SANDBOX_CONFIG_DIR in container_paths
+    assert lark_cli.LARK_CLI_SANDBOX_DATA_DIR in container_paths
+    assert lark_cli.LARK_CLI_SANDBOX_RUNTIME_DIR in container_paths
+
+    payload = remote_backend._provisioner_extra_mounts_payload(mounts)
+    payload_paths = [str(item["container_path"]) for item in payload]
+    assert len(payload_paths) == len(set(payload_paths))
+
+    provisioner_module.DEER_FLOW_HOST_BASE_DIR = str(home)
+    validated = provisioner_module._validated_extra_mounts([provisioner_module.ExtraMount(**item) for item in payload])
+    validated_paths = [mount.container_path for mount in validated]
+
+    assert len(validated_paths) == len(set(validated_paths))
+    assert set(validated_paths) == {
+        "/mnt/acp-workspace",
+        "/mnt/skills/custom",
+        "/mnt/skills/integrations",
+        lark_cli.LARK_CLI_SANDBOX_CONFIG_DIR,
+        lark_cli.LARK_CLI_SANDBOX_DATA_DIR,
+        lark_cli.LARK_CLI_SANDBOX_RUNTIME_DIR,
+    }
+
+
 def test_join_host_path_preserves_windows_drive_letter_style():
     base = r"C:\Users\demo\deer-flow\backend\.deer-flow"
 
@@ -405,6 +517,7 @@ def test_remote_backend_create_forwards_effective_user_id(monkeypatch):
         "thread_id": "thread-42",
         "user_id": "user-7",
         "include_legacy_skills": True,
+        "provision_lark_cli_runtime": False,
     }
 
 
@@ -435,6 +548,62 @@ def test_remote_backend_create_prefers_explicit_user_id(monkeypatch):
     assert posted["json"]["include_legacy_skills"] is False
 
 
+def test_create_sandbox_requests_runtime_when_lark_installed(tmp_path, monkeypatch):
+    """The provider must request lark-cli runtime provisioning when Lark is installed."""
+    aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
+    provider = _make_provider(tmp_path)
+    provider._config = {"replicas": 3}
+    provider._thread_locks = {}
+    provider._warm_pool = {}
+    provider._sandbox_infos = {}
+    provider._thread_sandboxes = {}
+    provider._last_activity = {}
+    provider._lock = aio_mod.threading.Lock()
+
+    captured: dict = {}
+
+    def _create(thread_id, sandbox_id, *, extra_mounts=None, user_id=None, provision_lark_cli_runtime=False):
+        captured["provision_lark_cli_runtime"] = provision_lark_cli_runtime
+        return aio_mod.SandboxInfo(sandbox_id=sandbox_id, sandbox_url="http://sandbox")
+
+    provider._backend = SimpleNamespace(create=_create, destroy=MagicMock(), discover=MagicMock(return_value=None))
+    monkeypatch.setattr(aio_mod, "wait_for_sandbox_ready", lambda *_a, **_k: True)
+    monkeypatch.setattr(provider, "_get_extra_mounts", lambda *_a, **_k: [])
+    monkeypatch.setattr(aio_mod.AioSandboxProvider, "_lark_integration_active", staticmethod(lambda user_id=None: True))
+    monkeypatch.setattr(provider, "_register_created_sandbox", lambda *a, **k: "sandbox-lark")
+
+    provider._create_sandbox("thread-lark", "sandbox-lark", user_id="alice")
+    assert captured["provision_lark_cli_runtime"] is True
+
+
+def test_create_sandbox_skips_runtime_when_lark_absent(tmp_path, monkeypatch):
+    """No runtime provisioning request when the Lark skill pack is not installed."""
+    aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
+    provider = _make_provider(tmp_path)
+    provider._config = {"replicas": 3}
+    provider._thread_locks = {}
+    provider._warm_pool = {}
+    provider._sandbox_infos = {}
+    provider._thread_sandboxes = {}
+    provider._last_activity = {}
+    provider._lock = aio_mod.threading.Lock()
+
+    captured: dict = {}
+
+    def _create(thread_id, sandbox_id, *, extra_mounts=None, user_id=None, provision_lark_cli_runtime=False):
+        captured["provision_lark_cli_runtime"] = provision_lark_cli_runtime
+        return aio_mod.SandboxInfo(sandbox_id=sandbox_id, sandbox_url="http://sandbox")
+
+    provider._backend = SimpleNamespace(create=_create, destroy=MagicMock(), discover=MagicMock(return_value=None))
+    monkeypatch.setattr(aio_mod, "wait_for_sandbox_ready", lambda *_a, **_k: True)
+    monkeypatch.setattr(provider, "_get_extra_mounts", lambda *_a, **_k: [])
+    monkeypatch.setattr(aio_mod.AioSandboxProvider, "_lark_integration_active", staticmethod(lambda user_id=None: False))
+    monkeypatch.setattr(provider, "_register_created_sandbox", lambda *a, **k: "sandbox-nolark")
+
+    provider._create_sandbox("thread-nolark", "sandbox-nolark", user_id="alice")
+    assert captured["provision_lark_cli_runtime"] is False
+
+
 # ── Sandbox client teardown (#2872) ──────────────────────────────────────────
 
 
@@ -723,3 +892,101 @@ def test_create_sandbox_evicts_oldest_warm_replica_via_shared_lifecycle(tmp_path
     assert "warm-oldest" not in provider._warm_pool
     assert provider._warm_pool == {"warm-newest": (newest_info, 20.0)}
     assert provider._sandbox_infos["created"] is created_info
+
+
+def _make_tenant_isolation_provider(tmp_path, monkeypatch):
+    aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
+    provider = _make_provider(tmp_path)
+    provider._lock = aio_mod.threading.Lock()
+    provider._sandboxes = {}
+    provider._sandbox_infos = {}
+    provider._thread_sandboxes = {}
+    provider._thread_locks = {}
+    provider._last_activity = {}
+    provider._warm_pool = {}
+    provider._active_sandbox_identity = {}
+    provider._warm_pool_identity = {}
+    provider._shutdown_called = False
+    provider._config = {"replicas": 3, "idle_timeout": 0}
+
+    create_calls = []
+
+    def _create(thread_id, sandbox_id, **kwargs):
+        create_calls.append((thread_id, sandbox_id, kwargs.get("user_id")))
+        return aio_mod.SandboxInfo(
+            sandbox_id=sandbox_id,
+            sandbox_url=f"http://sandbox-{len(create_calls)}.local",
+            container_name=f"deer-flow-sandbox-{sandbox_id}",
+        )
+
+    provider._backend = SimpleNamespace(
+        create=MagicMock(side_effect=_create),
+        destroy=MagicMock(),
+        discover=MagicMock(return_value=None),
+        is_alive=MagicMock(return_value=True),
+        list_running=MagicMock(return_value=[]),
+    )
+    provider._claim_ownership = MagicMock(return_value=True)
+    provider._held_teardown_lease = lambda _sandbox_id: contextlib.nullcontext()
+
+    monkeypatch.setattr(aio_mod, "get_paths", lambda: Paths(base_dir=tmp_path))
+    monkeypatch.setattr(
+        aio_mod.AioSandboxProvider,
+        "_get_extra_mounts",
+        lambda self, thread_id, *, user_id=None: [],
+    )
+    monkeypatch.setattr(
+        aio_mod,
+        "wait_for_sandbox_ready",
+        lambda _url, timeout=60: True,
+    )
+    return provider, create_calls, aio_mod
+
+
+def test_aio_wider_id_separates_known_legacy_collision():
+    aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider")
+    identity_a, identity_b = _LEGACY_COLLIDING_IDENTITIES
+    user_a, thread_a = identity_a
+    user_b, thread_b = identity_b
+
+    old_a = hashlib.sha256(f"{user_a}:{thread_a}".encode()).hexdigest()[:8]
+    old_b = hashlib.sha256(f"{user_b}:{thread_b}".encode()).hexdigest()[:8]
+
+    assert old_a == old_b
+    assert aio_mod.AioSandboxProvider._deterministic_sandbox_id(
+        thread_a,
+        user_a,
+    ) != aio_mod.AioSandboxProvider._deterministic_sandbox_id(
+        thread_b,
+        user_b,
+    )
+
+
+def test_aio_forced_collision_never_overwrites_active_tenant(
+    tmp_path,
+    monkeypatch,
+):
+    provider, create_calls, aio_mod = _make_tenant_isolation_provider(
+        tmp_path,
+        monkeypatch,
+    )
+    monkeypatch.setattr(
+        aio_mod.AioSandboxProvider,
+        "_deterministic_sandbox_id",
+        staticmethod(lambda thread_id, user_id: "deadbeefdeadbeef"),
+    )
+
+    sandbox_id = provider.acquire("thread-a", user_id="user-a")
+    info_a = provider._sandbox_infos[sandbox_id]
+    provider.release(sandbox_id)
+
+    assert sandbox_id in provider._warm_pool
+
+    with pytest.raises(aio_mod.SandboxIdentityCollisionError):
+        provider.acquire("thread-b", user_id="user-b")
+
+    assert provider._warm_pool[sandbox_id][0] is info_a
+    provider._backend.destroy.assert_not_called()
+    assert len(create_calls) == 1
+    assert provider.acquire("thread-a", user_id="user-a") == sandbox_id
+    assert provider._sandbox_infos[sandbox_id] is info_a

backend/tests/test_aio_sandbox_readiness.py

@@ -8,11 +8,20 @@ from deerflow.community.aio_sandbox import backend as readiness
 
 
 class _FakeAsyncClient:
-    def __init__(self, *, responses: list[object], calls: list[str], timeout: float, request_timeouts: list[float] | None = None) -> None:
+    def __init__(
+        self,
+        *,
+        responses: list[object],
+        calls: list[str],
+        timeout: float,
+        request_timeouts: list[float] | None = None,
+        trust_env: bool = True,
+    ) -> None:
         self._responses = responses
         self._calls = calls
         self._timeout = timeout
         self._request_timeouts = request_timeouts
+        self.trust_env = trust_env
 
     async def __aenter__(self) -> _FakeAsyncClient:
         return self
@@ -41,17 +50,65 @@ class _FakeLoop:
         return value
 
 
+@pytest.mark.parametrize(
+    ("sandbox_url", "expected"),
+    [
+        ("http://localhost:8080", False),
+        ("http://127.0.0.1:8080", False),
+        ("http://[::1]:8080", False),
+        ("http://host.docker.internal:8080", False),
+        ("http://host.containers.internal:8080", False),
+        ("http://k3s:30001", False),
+        ("http://10.0.0.8:8080", False),
+        ("http://8.8.8.8:8080", True),
+        ("http://[2606:4700:4700::1111]:8080", True),
+        ("https://sandbox.example.com", True),
+    ],
+)
+def test_sandbox_http_trust_env_only_uses_proxy_for_external_urls(sandbox_url: str, expected: bool) -> None:
+    assert readiness.sandbox_http_trust_env(sandbox_url) is expected
+
+
+def test_wait_for_sandbox_ready_bypasses_environment_proxy_for_docker_host(monkeypatch: pytest.MonkeyPatch) -> None:
+    sessions: list[object] = []
+
+    class FakeSession:
+        trust_env = True
+
+        def __enter__(self):
+            sessions.append(self)
+            return self
+
+        def __exit__(self, *_exc_info) -> None:
+            return None
+
+        def get(self, url: str, *, timeout: float):
+            assert url == "http://host.docker.internal:8080/v1/sandbox"
+            assert timeout == 5
+            return SimpleNamespace(status_code=200)
+
+    monkeypatch.setattr(readiness.requests, "Session", FakeSession)
+
+    assert readiness.wait_for_sandbox_ready("http://host.docker.internal:8080", timeout=1) is True
+    assert len(sessions) == 1
+    assert sessions[0].trust_env is False
+
+
 @pytest.mark.anyio
 async def test_wait_for_sandbox_ready_async_uses_nonblocking_polling(monkeypatch: pytest.MonkeyPatch) -> None:
     calls: list[str] = []
     sleeps: list[float] = []
+    clients: list[_FakeAsyncClient] = []
 
-    def fake_client(*, timeout: float):
-        return _FakeAsyncClient(
+    def fake_client(*, timeout: float, trust_env: bool):
+        client = _FakeAsyncClient(
             responses=[SimpleNamespace(status_code=503), SimpleNamespace(status_code=200)],
             calls=calls,
             timeout=timeout,
+            trust_env=trust_env,
         )
+        clients.append(client)
+        return client
 
     async def fake_sleep(delay: float) -> None:
         sleeps.append(delay)
@@ -65,6 +122,7 @@ async def test_wait_for_sandbox_ready_async_uses_nonblocking_polling(monkeypatch
 
     assert calls == ["http://sandbox/v1/sandbox", "http://sandbox/v1/sandbox"]
     assert sleeps == [0.05]
+    assert clients[0].trust_env is False
 
 
 @pytest.mark.anyio
@@ -72,11 +130,12 @@ async def test_wait_for_sandbox_ready_async_retries_request_errors(monkeypatch:
     calls: list[str] = []
     sleeps: list[float] = []
 
-    def fake_client(*, timeout: float):
+    def fake_client(*, timeout: float, trust_env: bool):
         return _FakeAsyncClient(
             responses=[readiness.httpx.ConnectError("not ready"), SimpleNamespace(status_code=200)],
             calls=calls,
             timeout=timeout,
+            trust_env=trust_env,
         )
 
     async def fake_sleep(delay: float) -> None:
@@ -97,12 +156,13 @@ async def test_wait_for_sandbox_ready_async_clamps_request_and_sleep_to_deadline
     request_timeouts: list[float] = []
     sleeps: list[float] = []
 
-    def fake_client(*, timeout: float):
+    def fake_client(*, timeout: float, trust_env: bool):
         return _FakeAsyncClient(
             responses=[SimpleNamespace(status_code=503)],
             calls=calls,
             timeout=timeout,
             request_timeouts=request_timeouts,
+            trust_env=trust_env,
         )
 
     async def fake_sleep(delay: float) -> None:

backend/tests/test_auth.py

@@ -478,6 +478,317 @@ def test_update_user_raises_when_row_concurrently_deleted(tmp_path):
     asyncio.run(_run())
 
 
+# ── Email case-insensitivity (account collision invariant) ──────────────────
+#
+# Regression coverage for the case-collision gap: local registration normalises
+# email through ``EmailStr`` (lowercases only the domain) while OIDC lowercases
+# the whole address, and the repo lookup used to be case-sensitive, so
+# ``Victim@x.com`` and ``victim@x.com`` became two separate accounts — defeating
+# the invariant that a local account blocks an SSO login on the same email
+# (flagged on PR #3506, fixed there only OIDC-side). The repo now canonicalises
+# to lowercase on write and matches case-insensitively on read.
+
+
+def test_email_lookup_is_case_insensitive(tmp_path):
+    """A user registered with mixed case resolves for any-case lookup."""
+    import asyncio
+
+    from app.gateway.auth.repositories.sqlite import SQLiteUserRepository
+
+    async def _run() -> None:
+        from deerflow.persistence.engine import close_engine, get_session_factory, init_engine
+
+        url = f"sqlite+aiosqlite:///{tmp_path}/scratch.db"
+        await init_engine("sqlite", url=url, sqlite_dir=str(tmp_path))
+        try:
+            repo = SQLiteUserRepository(get_session_factory())
+            created = await repo.create_user(User(email="Victim@x.com", password_hash="h", system_role="user"))
+            # Stored canonical (lowercase) and reflected back on the returned object.
+            assert created.email == "victim@x.com"
+
+            for variant in ("victim@x.com", "VICTIM@X.COM", "Victim@x.com"):
+                found = await repo.get_user_by_email(variant)
+                assert found is not None, f"lookup missed {variant!r}"
+                assert str(found.id) == str(created.id)
+        finally:
+            await close_engine()
+
+    asyncio.run(_run())
+
+
+def test_create_user_rejects_email_differing_only_in_case(tmp_path):
+    """The second case-variant registration collides on the canonical email."""
+    import asyncio
+
+    from app.gateway.auth.repositories.sqlite import SQLiteUserRepository
+
+    async def _run() -> None:
+        from deerflow.persistence.engine import close_engine, get_session_factory, init_engine
+
+        url = f"sqlite+aiosqlite:///{tmp_path}/scratch.db"
+        await init_engine("sqlite", url=url, sqlite_dir=str(tmp_path))
+        try:
+            repo = SQLiteUserRepository(get_session_factory())
+            await repo.create_user(User(email="Victim@x.com", password_hash="h", system_role="user"))
+            with pytest.raises(ValueError):
+                await repo.create_user(User(email="victim@x.com", password_hash="h", system_role="user"))
+            assert await repo.count_users() == 1
+        finally:
+            await close_engine()
+
+    asyncio.run(_run())
+
+
+def test_create_user_rejects_legacy_mixed_case_email(tmp_path):
+    """Registration must not duplicate a mixed-case row created before normalization."""
+    import asyncio
+    from uuid import uuid4
+
+    from app.gateway.auth.repositories.sqlite import SQLiteUserRepository
+
+    async def _run() -> None:
+        from deerflow.persistence.engine import close_engine, get_session_factory, init_engine
+        from deerflow.persistence.user.model import UserRow
+
+        url = f"sqlite+aiosqlite:///{tmp_path}/scratch.db"
+        await init_engine("sqlite", url=url, sqlite_dir=str(tmp_path))
+        try:
+            sf = get_session_factory()
+            repo = SQLiteUserRepository(sf)
+            async with sf() as session:
+                session.add(
+                    UserRow(
+                        id=str(uuid4()),
+                        email="Victim@x.com",
+                        password_hash="h",
+                        system_role="user",
+                        needs_setup=False,
+                        token_version=0,
+                    )
+                )
+                await session.commit()
+
+            with pytest.raises(ValueError, match="Email already registered"):
+                await repo.create_user(User(email="victim@x.com", password_hash="h", system_role="user"))
+            assert await repo.count_users() == 1
+        finally:
+            await close_engine()
+
+    asyncio.run(_run())
+
+
+def test_update_user_normalizes_email(tmp_path):
+    """Changing an email through update_user stores the canonical lowercase form."""
+    import asyncio
+
+    from app.gateway.auth.repositories.sqlite import SQLiteUserRepository
+
+    async def _run() -> None:
+        from deerflow.persistence.engine import close_engine, get_session_factory, init_engine
+
+        url = f"sqlite+aiosqlite:///{tmp_path}/scratch.db"
+        await init_engine("sqlite", url=url, sqlite_dir=str(tmp_path))
+        try:
+            repo = SQLiteUserRepository(get_session_factory())
+            user = await repo.create_user(User(email="user@x.com", password_hash="h", system_role="user"))
+            user.email = "New@Mixed.COM"
+            await repo.update_user(user)
+            refetched = await repo.get_user_by_id(str(user.id))
+            assert refetched is not None
+            assert refetched.email == "new@mixed.com"
+            assert await repo.get_user_by_email("NEW@MIXED.com") is not None
+        finally:
+            await close_engine()
+
+    asyncio.run(_run())
+
+
+def test_distinct_emails_remain_distinct(tmp_path):
+    """Case-folding must not collapse genuinely different addresses."""
+    import asyncio
+
+    from app.gateway.auth.repositories.sqlite import SQLiteUserRepository
+
+    async def _run() -> None:
+        from deerflow.persistence.engine import close_engine, get_session_factory, init_engine
+
+        url = f"sqlite+aiosqlite:///{tmp_path}/scratch.db"
+        await init_engine("sqlite", url=url, sqlite_dir=str(tmp_path))
+        try:
+            repo = SQLiteUserRepository(get_session_factory())
+            alice = await repo.create_user(User(email="alice@x.com", password_hash="h", system_role="user"))
+            bob = await repo.create_user(User(email="bob@x.com", password_hash="h", system_role="user"))
+            assert await repo.count_users() == 2
+            fa = await repo.get_user_by_email("Alice@x.com")
+            fb = await repo.get_user_by_email("BOB@x.com")
+            assert fa is not None and str(fa.id) == str(alice.id)
+            assert fb is not None and str(fb.id) == str(bob.id)
+            assert str(fa.id) != str(fb.id)
+        finally:
+            await close_engine()
+
+    asyncio.run(_run())
+
+
+def test_legacy_mixed_case_duplicate_rows_resolve_without_error(tmp_path):
+    """A pre-fix DB with two case-variant rows resolves to the oldest, never 500s.
+
+    Migration-safety: existing installations may already hold ``Victim@x.com``
+    and ``victim@x.com`` as separate rows. The case-insensitive lookup must not
+    raise ``MultipleResultsFound``; it deterministically returns the oldest
+    (most-established) account.
+    """
+    import asyncio
+    from datetime import UTC, datetime, timedelta
+    from uuid import uuid4
+
+    from app.gateway.auth.repositories.sqlite import SQLiteUserRepository
+
+    async def _run() -> None:
+        from deerflow.persistence.engine import close_engine, get_session_factory, init_engine
+        from deerflow.persistence.user.model import UserRow
+
+        url = f"sqlite+aiosqlite:///{tmp_path}/scratch.db"
+        await init_engine("sqlite", url=url, sqlite_dir=str(tmp_path))
+        try:
+            sf = get_session_factory()
+            repo = SQLiteUserRepository(sf)
+            older = datetime.now(UTC) - timedelta(days=5)
+            newer = datetime.now(UTC)
+            # Insert raw rows (bypassing create_user's normalisation) to mimic
+            # data written before this fix.
+            async with sf() as session:
+                session.add(UserRow(id=str(uuid4()), email="Victim@x.com", password_hash="h", system_role="user", created_at=older, needs_setup=False, token_version=0))
+                session.add(UserRow(id=str(uuid4()), email="victim@x.com", password_hash="h", system_role="user", created_at=newer, needs_setup=False, token_version=0))
+                await session.commit()
+
+            found = await repo.get_user_by_email("VICTIM@X.COM")
+            assert found is not None
+            assert found.email == "Victim@x.com"  # oldest wins, deterministically
+        finally:
+            await close_engine()
+
+    asyncio.run(_run())
+
+
+def test_update_user_on_legacy_mixed_case_row_does_not_collide(tmp_path):
+    """A password-only update on a legacy mixed-case row must not 500.
+
+    Migration-safety, write side. A pre-fix DB may already hold two rows
+    differing only in case (``Victim@x.com`` + ``victim@x.com``). A password
+    change or admin reset reloads the row and calls ``update_user`` with the
+    email unchanged. ``update_user`` must not opportunistically re-lowercase the
+    mixed-case email, because that collides with the already-canonical row's
+    unique email and raises ``IntegrityError`` — which surfaces as a 500 on the
+    change-password / reset-admin paths that don't catch it. The read path was
+    hardened for this legacy state; the write path must match, so only a genuine
+    email change (differing case-insensitively from the stored value) rewrites
+    the column.
+    """
+    import asyncio
+    from datetime import UTC, datetime, timedelta
+    from uuid import uuid4
+
+    from app.gateway.auth.repositories.sqlite import SQLiteUserRepository
+
+    async def _run() -> None:
+        from deerflow.persistence.engine import close_engine, get_session_factory, init_engine
+        from deerflow.persistence.user.model import UserRow
+
+        url = f"sqlite+aiosqlite:///{tmp_path}/scratch.db"
+        await init_engine("sqlite", url=url, sqlite_dir=str(tmp_path))
+        try:
+            sf = get_session_factory()
+            repo = SQLiteUserRepository(sf)
+            mixed_id = str(uuid4())
+            canonical_id = str(uuid4())
+            older = datetime.now(UTC) - timedelta(days=5)
+            newer = datetime.now(UTC)
+            # Raw rows written before the fix (the mixed-case row is the oldest).
+            async with sf() as session:
+                session.add(UserRow(id=mixed_id, email="Victim@x.com", password_hash="old-hash", system_role="user", created_at=older, needs_setup=False, token_version=0))
+                session.add(UserRow(id=canonical_id, email="victim@x.com", password_hash="canonical-hash", system_role="user", created_at=newer, needs_setup=False, token_version=0))
+                await session.commit()
+
+            # Simulate a password change on the mixed-case row: reload it, keep
+            # the email as-stored, set a new hash + bump the token version.
+            mixed = await repo.get_user_by_id(mixed_id)
+            assert mixed is not None
+            assert mixed.email == "Victim@x.com"
+            mixed.password_hash = "new-hash-after-change"
+            mixed.token_version += 1
+
+            # Must not raise IntegrityError even though lowercasing the email
+            # would collide with the canonical row's unique email.
+            await repo.update_user(mixed)
+
+            # The mixed-case row kept its stored casing and took the new password.
+            refetched = await repo.get_user_by_id(mixed_id)
+            assert refetched is not None
+            assert refetched.email == "Victim@x.com"
+            assert refetched.password_hash == "new-hash-after-change"
+            assert refetched.token_version == 1
+
+            # The canonical row is untouched, and no row was lost or merged.
+            canonical = await repo.get_user_by_id(canonical_id)
+            assert canonical is not None
+            assert canonical.email == "victim@x.com"
+            assert canonical.password_hash == "canonical-hash"
+            assert await repo.count_users() == 2
+
+            # Case-insensitive lookup still resolves the mixed-case row (oldest wins).
+            found = await repo.get_user_by_email("VICTIM@X.COM")
+            assert found is not None
+            assert str(found.id) == mixed_id
+        finally:
+            await close_engine()
+
+    asyncio.run(_run())
+
+
+def test_oidc_login_blocked_by_existing_local_account_across_case(tmp_path):
+    """End-to-end invariant: an SSO login cannot create a duplicate of a local
+    account whose email differs only in case.
+
+    Uses the real repository + provider + provisioning (no mocks), so it covers
+    the cross-path gap the mock-based OIDC tests could not: local registration
+    keeps the local-part case (``Victim@x.com``) while OIDC lowercases the whole
+    address (``victim@x.com``).
+    """
+    import asyncio
+
+    from app.gateway.auth.local_provider import LocalAuthProvider
+    from app.gateway.auth.repositories.sqlite import SQLiteUserRepository
+    from app.gateway.auth.user_provisioning import get_or_provision_oidc_user
+    from deerflow.config.auth_config import OIDCProviderConfig
+
+    async def _run() -> None:
+        from fastapi import HTTPException
+
+        from app.gateway.auth.oidc import OIDCIdentity
+        from deerflow.persistence.engine import close_engine, get_session_factory, init_engine
+
+        url = f"sqlite+aiosqlite:///{tmp_path}/scratch.db"
+        await init_engine("sqlite", url=url, sqlite_dir=str(tmp_path))
+        try:
+            provider = LocalAuthProvider(SQLiteUserRepository(get_session_factory()))
+            await provider.create_user(email="Victim@x.com", password="pw-abc-123!", system_role="user")
+
+            cfg = OIDCProviderConfig(display_name="Test SSO", issuer="https://issuer.example.com", client_id="deer-flow", auto_create_users=True)
+            identity = OIDCIdentity(provider="keycloak", subject="sub-1", email="Victim@x.com", email_verified=True, name="Victim", claims={})
+
+            with pytest.raises(HTTPException) as exc_info:
+                await get_or_provision_oidc_user(provider_id="keycloak", provider_config=cfg, identity=identity, local_provider=provider)
+
+            assert exc_info.value.status_code == 409
+            # No duplicate auto-created — the local account still owns the email.
+            assert await provider.count_users() == 1
+        finally:
+            await close_engine()
+
+    asyncio.run(_run())
+
+
 # ── Token Versioning ───────────────────────────────────────────────────────
 
 

backend/tests/test_boxlite_provider.py

@@ -7,6 +7,7 @@ which need a live box.
 from __future__ import annotations
 
 import asyncio
+import hashlib
 import logging
 import sys
 import threading
@@ -18,6 +19,11 @@ import pytest
 from deerflow.community.boxlite.box import BoxliteBox
 from deerflow.community.boxlite.provider import BoxliteProvider, _import_simplebox
 
+_LEGACY_COLLIDING_IDENTITIES = (
+    ("user-9721", "thread-9721"),
+    ("user-94361", "thread-94361"),
+)
+
 # ── Fake BoxLite SDK ──────────────────────────────────────────────────
 
 
@@ -262,7 +268,7 @@ def test_sandbox_id_deterministic(monkeypatch):
     id1 = provider._sandbox_id("thread-1", "user-a")
     id2 = provider._sandbox_id("thread-1", "user-a")
     assert id1 == id2
-    assert len(id1) == 8
+    assert len(id1) == 16
 
 
 def test_sandbox_id_different_users(monkeypatch):
@@ -460,7 +466,7 @@ def test_explicit_recent_reclaim_skip_avoids_health_check(monkeypatch):
 
     monkeypatch.setattr(box, "execute_command", _fail_if_called)
 
-    reclaimed = provider._reclaim_warm_pool(sid)
+    reclaimed = provider._reclaim_warm_pool(sid, ("u1", "thread-1"))
     assert reclaimed == sid
     assert sid in provider._boxes
     assert sid not in provider._warm_pool
@@ -494,7 +500,7 @@ def test_recent_reclaim_validates_by_default(monkeypatch):
 
     monkeypatch.setattr(box, "execute_command", _record_health_check)
 
-    reclaimed = provider._reclaim_warm_pool(sid)
+    reclaimed = provider._reclaim_warm_pool(sid, ("u1", "thread-1"))
     assert reclaimed == sid
     assert calls == 1
     assert sid in provider._boxes
@@ -526,7 +532,7 @@ def test_default_recent_reclaim_drops_dead_warm_box(monkeypatch):
 
     monkeypatch.setattr(box, "execute_command", _dead_health_check)
 
-    reclaimed = provider._reclaim_warm_pool(sid)
+    reclaimed = provider._reclaim_warm_pool(sid, ("u1", "thread-1"))
     assert reclaimed is None
     assert sid not in provider._boxes
     assert sid not in provider._warm_pool
@@ -592,7 +598,10 @@ def test_adopted_warm_pool_box_still_health_checks(monkeypatch):
 
     monkeypatch.setattr(adopted, "execute_command", _record_health_check)
 
-    reclaimed = provider._reclaim_warm_pool("adopted")
+    reclaimed = provider._reclaim_warm_pool(
+        "adopted",
+        ("some-user", "some-thread"),
+    )
     assert reclaimed == "adopted"
     assert calls == 1
     assert "adopted" in provider._boxes
@@ -1044,3 +1053,143 @@ def test_shutdown_stops_idle_reaper_and_destroys_all_boxes(monkeypatch):
     assert len(provider._boxes) == 0
     assert len(provider._warm_pool) == 0
     assert len(provider._thread_boxes) == 0
+
+
+def test_wider_id_separates_known_legacy_collision():
+    identity_a, identity_b = _LEGACY_COLLIDING_IDENTITIES
+    user_a, thread_a = identity_a
+    user_b, thread_b = identity_b
+
+    old_a = hashlib.sha256(f"{user_a}:{thread_a}".encode()).hexdigest()[:8]
+    old_b = hashlib.sha256(f"{user_b}:{thread_b}".encode()).hexdigest()[:8]
+
+    assert old_a == old_b
+    assert BoxliteProvider._sandbox_id(thread_a, user_a) != BoxliteProvider._sandbox_id(
+        thread_b,
+        user_b,
+    )
+
+
+def test_forced_collision_never_overwrites_active_tenant(monkeypatch):
+    monkeypatch.setattr(
+        "deerflow.community.boxlite.provider.get_app_config",
+        lambda: _stub_config(),
+    )
+    monkeypatch.setattr(
+        "deerflow.community.boxlite.provider._import_simplebox",
+        lambda: _FakeBox,
+    )
+    monkeypatch.setattr(
+        BoxliteProvider,
+        "_sandbox_id",
+        staticmethod(lambda thread_id, user_id: "deadbeefdeadbeef"),
+    )
+
+    provider = BoxliteProvider()
+    provider._loop.run = _fake_run
+
+    sandbox_id = provider.acquire("thread-a", user_id="user-a")
+    box_a = provider.get(sandbox_id)
+    provider.release(sandbox_id)
+
+    assert box_a is not None
+    assert sandbox_id in provider._warm_pool
+
+    with pytest.raises(RuntimeError, match="Sandbox ID collision"):
+        provider.acquire("thread-b", user_id="user-b")
+
+    assert provider._warm_pool[sandbox_id][0] is box_a
+    assert not box_a.is_closed
+    assert provider.acquire("thread-a", user_id="user-a") == sandbox_id
+    assert provider.get(sandbox_id) is box_a
+    provider.shutdown()
+
+
+def test_late_same_tenant_collision_reuses_active_box(monkeypatch):
+    monkeypatch.setattr(
+        "deerflow.community.boxlite.provider.get_app_config",
+        lambda: _stub_config(),
+    )
+
+    provider = BoxliteProvider()
+    sandbox_id = "deadbeefdeadbeef"
+    key = ("user-a", "thread-a")
+    active = BoxliteBox(
+        sandbox_id,
+        _FakeBox(name=f"deer-flow-boxlite-{sandbox_id}"),
+        _fake_run,
+        default_env={},
+    )
+    duplicate = BoxliteBox(
+        sandbox_id,
+        _FakeBox(name=f"deer-flow-boxlite-{sandbox_id}"),
+        _fake_run,
+        default_env={},
+    )
+
+    monkeypatch.setattr(
+        BoxliteProvider,
+        "_sandbox_id",
+        staticmethod(lambda thread_id, user_id: sandbox_id),
+    )
+
+    def _create_box_with_late_registration(_sandbox_id):
+        with provider._lock:
+            provider._boxes[sandbox_id] = active
+            provider._active_box_identity[sandbox_id] = key
+        return duplicate
+
+    monkeypatch.setattr(provider, "_create_box", _create_box_with_late_registration)
+
+    assert provider.acquire("thread-a", user_id="user-a") == sandbox_id
+    assert duplicate.is_closed
+    assert provider.get(sandbox_id) is active
+    assert provider._thread_boxes[key] == sandbox_id
+    provider.shutdown()
+
+
+def test_failed_health_check_does_not_remove_swapped_warm_entry(monkeypatch):
+    monkeypatch.setattr(
+        "deerflow.community.boxlite.provider.get_app_config",
+        lambda: _stub_config(),
+    )
+
+    provider = BoxliteProvider()
+    stale = BoxliteBox(
+        "shared-id",
+        _FakeBox(name="deer-flow-boxlite-shared-id"),
+        _fake_run,
+        default_env={},
+    )
+    replacement = BoxliteBox(
+        "shared-id",
+        _FakeBox(name="deer-flow-boxlite-shared-id"),
+        _fake_run,
+        default_env={},
+    )
+    provider._warm_pool["shared-id"] = (stale, time.time())
+    provider._warm_pool_identity["shared-id"] = ("user-a", "thread-a")
+
+    def _swap_during_health_check(*args, **kwargs):
+        with provider._lock:
+            provider._warm_pool["shared-id"] = (replacement, time.time())
+            provider._warm_pool_identity["shared-id"] = (
+                "user-b",
+                "thread-b",
+            )
+        return "not healthy"
+
+    monkeypatch.setattr(stale, "execute_command", _swap_during_health_check)
+
+    with pytest.raises(RuntimeError, match="Sandbox ID collision"):
+        provider._reclaim_warm_pool(
+            "shared-id",
+            ("user-a", "thread-a"),
+        )
+    assert provider._warm_pool["shared-id"][0] is replacement
+    assert provider._warm_pool_identity["shared-id"] == (
+        "user-b",
+        "thread-b",
+    )
+    assert not replacement.is_closed
+    provider.shutdown()

backend/tests/test_branch_history_seed.py

@@ -19,7 +19,7 @@ def _seed(messages):
     return build_branch_history_seed_events(
         messages,
         thread_id="branch-thread",
-        run_id="branch-seed-branch-thread",
+        run_id_prefix="branch-seed-branch-thread",
         parent_thread_id="parent-thread",
     )
 
@@ -36,7 +36,7 @@ def test_seed_serializes_visible_history_in_order() -> None:
     assert [event["event_type"] for event in events] == ["llm.human.input", "llm.ai.response", "llm.tool.result"]
     assert all(event["category"] == "message" for event in events)
     assert all(event["thread_id"] == "branch-thread" for event in events)
-    assert all(event["run_id"] == "branch-seed-branch-thread" for event in events)
+    assert all(event["run_id"] == "branch-seed-branch-thread-1" for event in events)
     assert [event["content"]["id"] for event in events] == ["h1", "a1", "t1"]
     # Human/AI rows carry the journal's caller tag; every row carries the
     # seed provenance marker.
@@ -159,6 +159,60 @@ def test_seed_restores_original_user_content() -> None:
     assert "original_user_content" not in events[0]["content"]["additional_kwargs"]
 
 
+def test_seed_scopes_one_synthetic_run_per_inherited_turn() -> None:
+    """``run_id`` is a turn identity to the feed, not a provenance tag: the
+    regenerate path supersedes a whole ``run_id``, so packing every inherited
+    turn into one id deleted the complete history on the first regenerate
+    (#4458). Each human message opens the next synthetic run."""
+    events = _seed(
+        [
+            HumanMessage(id="h1", content="turn one"),
+            AIMessage(id="a1", content="answer one"),
+            HumanMessage(id="h2", content="turn two"),
+            AIMessage(id="a2", content="calling a tool", tool_calls=[{"name": "search", "args": {}, "id": "call-1", "type": "tool_call"}]),
+            ToolMessage(id="t2", content="tool output", tool_call_id="call-1"),
+            AIMessage(id="a2b", content="answer two"),
+        ]
+    )
+
+    assert [(event["content"]["id"], event["run_id"]) for event in events] == [
+        ("h1", "branch-seed-branch-thread-1"),
+        ("a1", "branch-seed-branch-thread-1"),
+        ("h2", "branch-seed-branch-thread-2"),
+        ("a2", "branch-seed-branch-thread-2"),
+        ("t2", "branch-seed-branch-thread-2"),
+        ("a2b", "branch-seed-branch-thread-2"),
+    ]
+
+
+def test_seed_opens_a_new_turn_at_a_hidden_clarification_reply() -> None:
+    """An answered clarification resumes as its own run, so the reply is a turn
+    boundary exactly like a visible user message."""
+    response = {
+        "version": 1,
+        "kind": "human_input_response",
+        "source": "ask_clarification",
+        "request_id": "req-1",
+        "value": "yes",
+        "response_kind": "text",
+    }
+    events = _seed(
+        [
+            HumanMessage(id="h1", content="question"),
+            AIMessage(id="a1", content="which one?"),
+            HumanMessage(id="h-response", content="yes", additional_kwargs={"hide_from_ui": True, "human_input_response": response}),
+            AIMessage(id="a2", content="answer"),
+        ]
+    )
+
+    assert [event["run_id"] for event in events] == [
+        "branch-seed-branch-thread-1",
+        "branch-seed-branch-thread-1",
+        "branch-seed-branch-thread-2",
+        "branch-seed-branch-thread-2",
+    ]
+
+
 def test_seed_roundtrips_through_memory_store_feed() -> None:
     """put_batch preserves order and list_messages returns the seeded feed."""
     store = MemoryRunEventStore()

backend/tests/test_channel_file_attachments.py

@@ -255,6 +255,33 @@ class TestResolveAttachments:
 
 
 class TestInboundFileIngestion:
+    def test_consumes_inline_channel_bytes_without_exposing_them_downstream(self, tmp_path):
+        from app.channels import manager
+
+        uploads_dir = tmp_path / "uploads"
+        uploads_dir.mkdir()
+        msg = InboundMessage(
+            channel_name="telegram",
+            chat_id="chat-1",
+            user_id="user-1",
+            text="see attachment",
+            files=[{"type": "file", "filename": "report.pdf", "_content": b"pdf bytes"}],
+        )
+
+        with patch("deerflow.uploads.manager.ensure_uploads_dir", return_value=uploads_dir):
+            result = _run(manager._ingest_inbound_files("thread-1", msg))
+
+        assert result == [
+            {
+                "filename": "report.pdf",
+                "size": len(b"pdf bytes"),
+                "path": "/mnt/user-data/uploads/report.pdf",
+                "is_image": False,
+            }
+        ]
+        assert (uploads_dir / "report.pdf").read_bytes() == b"pdf bytes"
+        assert "_content" not in msg.files[0]
+
     def test_rejects_preexisting_symlink_destination(self, tmp_path):
         from app.channels import manager
 

backend/tests/test_channels.py

@@ -6,6 +6,7 @@ import asyncio
 import json
 import logging
 import tempfile
+import threading
 from concurrent.futures import Future
 from pathlib import Path
 from types import SimpleNamespace
@@ -7805,6 +7806,9 @@ class TestTelegramSendRetry:
             def __and__(self, other):
                 return FakeFilter(f"{self.expr}&{other.expr}")
 
+            def __or__(self, other):
+                return FakeFilter(f"{self.expr}|{other.expr}")
+
             def __invert__(self):
                 return FakeFilter(f"~{self.expr}")
 
@@ -7836,7 +7840,12 @@ class TestTelegramSendRetry:
         telegram_ext_mod.ApplicationBuilder = FakeApplicationBuilder
         telegram_ext_mod.CommandHandler = fake_command_handler
         telegram_ext_mod.MessageHandler = fake_message_handler
-        telegram_ext_mod.filters = SimpleNamespace(TEXT=FakeFilter("TEXT"), COMMAND=FakeFilter("COMMAND"))
+        telegram_ext_mod.filters = SimpleNamespace(
+            TEXT=FakeFilter("TEXT"),
+            COMMAND=FakeFilter("COMMAND"),
+            PHOTO=FakeFilter("PHOTO"),
+            Document=SimpleNamespace(ALL=FakeFilter("DOCUMENT")),
+        )
         telegram_mod.ext = telegram_ext_mod
         monkeypatch.setitem(sys.modules, "telegram", telegram_mod)
         monkeypatch.setitem(sys.modules, "telegram.ext", telegram_ext_mod)
@@ -7852,6 +7861,9 @@ class TestTelegramSendRetry:
             def join(self, timeout=None):
                 return None
 
+            def is_alive(self):
+                return False
+
         monkeypatch.setattr("app.channels.telegram.threading.Thread", FakeThread)
 
         async def go():
@@ -7866,6 +7878,7 @@ class TestTelegramSendRetry:
                 assert "start" in registered_commands
                 message_filters = {handler.filter_expr.expr for handler in fake_app.handlers if handler.kind == "message"}
                 assert {"TEXT&COMMAND", "TEXT&~COMMAND"} <= message_filters
+                assert "PHOTO|DOCUMENT" in message_filters
             finally:
                 await ch.stop()
 
@@ -7958,13 +7971,25 @@ class TestFeishuSendRetry:
 # ---------------------------------------------------------------------------
 
 
-def _make_telegram_update(chat_type: str, message_id: int, *, reply_to_message_id: int | None = None, text: str = "hello"):
+def _make_telegram_update(
+    chat_type: str,
+    message_id: int,
+    *,
+    reply_to_message_id: int | None = None,
+    text: str | None = "hello",
+    caption: str | None = None,
+    photo: list[SimpleNamespace] | None = None,
+    document: SimpleNamespace | None = None,
+):
     """Build a minimal mock telegram Update for testing _on_text / _cmd_generic."""
     update = MagicMock()
     update.effective_chat.type = chat_type
     update.effective_chat.id = 100
     update.effective_user.id = 42
     update.message.text = text
+    update.message.caption = caption
+    update.message.photo = photo or []
+    update.message.document = document
     update.message.message_id = message_id
     if reply_to_message_id is not None:
         reply_msg = MagicMock()
@@ -7975,8 +8000,8 @@ def _make_telegram_update(chat_type: str, message_id: int, *, reply_to_message_i
     return update
 
 
-class TestTelegramPrivateChatThread:
-    """Verify that private chats use topic_id=None (single thread per chat)."""
+class TestTelegramInboundMessages:
+    """Verify Telegram inbound normalization and conversation thread context."""
 
     def test_private_chat_no_reply_uses_none_topic(self):
         from app.channels.telegram import TelegramChannel
@@ -7984,7 +8009,7 @@ class TestTelegramPrivateChatThread:
         async def go():
             bus = MessageBus()
             ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
-            ch._main_loop = asyncio.get_event_loop()
+            ch._main_loop = asyncio.get_running_loop()
 
             update = _make_telegram_update("private", message_id=10)
             await ch._on_text(update, None)
@@ -7994,13 +8019,519 @@ class TestTelegramPrivateChatThread:
 
         _run(go())
 
+    def test_photo_caption_uses_largest_size_and_preserves_thread_context(self):
+        from app.channels.telegram import TelegramChannel
+
+        async def go():
+            bus = MessageBus()
+            ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
+            ch._main_loop = asyncio.get_running_loop()
+            small = SimpleNamespace(file_id="photo-small", file_unique_id="unique-small", file_size=10, width=90, height=90)
+            large = SimpleNamespace(file_id="photo-large", file_unique_id="unique-large", file_size=200, width=800, height=600)
+            update = _make_telegram_update(
+                "group",
+                message_id=40,
+                reply_to_message_id=15,
+                text=None,
+                caption="  Describe this image  ",
+                # Do not rely on Telegram returning PhotoSize objects in order.
+                photo=[large, small],
+            )
+
+            await ch._on_text(update, None)
+
+            msg = await asyncio.wait_for(bus.get_inbound(), timeout=2)
+            assert msg.text == "Describe this image"
+            assert msg.topic_id == "15"
+            assert msg.thread_ts == "40"
+            assert len(msg.files) == 1
+            assert msg.files[0] == {
+                "type": "image",
+                "file_id": "photo-large",
+                "file_unique_id": "unique-large",
+                "filename": "telegram-photo-40.jpg",
+                "mime_type": "image/jpeg",
+                "size": 200,
+            }
+
+        _run(go())
+
+    def test_document_without_caption_still_publishes_an_inbound_turn(self):
+        from app.channels.telegram import TelegramChannel
+
+        async def go():
+            bus = MessageBus()
+            ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
+            ch._main_loop = asyncio.get_running_loop()
+            document = SimpleNamespace(
+                file_id="document-id",
+                file_unique_id="document-unique",
+                file_name="../report.pdf",
+                mime_type="application/pdf",
+                file_size=1234,
+            )
+            update = _make_telegram_update("private", message_id=41, text=None, document=document)
+
+            await ch._on_text(update, None)
+
+            msg = await asyncio.wait_for(bus.get_inbound(), timeout=2)
+            assert msg.text == ""
+            assert msg.topic_id is None
+            assert msg.files == [
+                {
+                    "type": "file",
+                    "file_id": "document-id",
+                    "file_unique_id": "document-unique",
+                    "filename": "report.pdf",
+                    "mime_type": "application/pdf",
+                    "size": 1234,
+                }
+            ]
+
+        _run(go())
+
+    def test_photo_without_caption_still_publishes_an_inbound_turn(self):
+        from app.channels.telegram import TelegramChannel
+
+        async def go():
+            bus = MessageBus()
+            ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
+            ch._main_loop = asyncio.get_running_loop()
+            photo = SimpleNamespace(file_id="photo-id", file_unique_id="photo-unique", file_size=25)
+            update = _make_telegram_update("private", message_id=42, text=None, photo=[photo])
+
+            await ch._on_text(update, None)
+
+            msg = await asyncio.wait_for(bus.get_inbound(), timeout=2)
+            assert msg.text == ""
+            assert msg.files[0]["filename"] == "telegram-photo-42.jpg"
+            assert msg.files[0]["file_id"] == "photo-id"
+
+        _run(go())
+
+    def test_document_caption_is_preserved_and_missing_filename_gets_safe_fallback(self):
+        from app.channels.telegram import TelegramChannel
+
+        async def go():
+            bus = MessageBus()
+            ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
+            ch._main_loop = asyncio.get_running_loop()
+            document = SimpleNamespace(
+                file_id="document-id",
+                file_unique_id="document-unique",
+                file_name=None,
+                mime_type="text/plain",
+                file_size=12,
+            )
+            update = _make_telegram_update("private", message_id=43, text=None, caption="  Review this  ", document=document)
+
+            await ch._on_text(update, None)
+
+            msg = await asyncio.wait_for(bus.get_inbound(), timeout=2)
+            assert msg.text == "Review this"
+            assert msg.files[0]["filename"] == "telegram-document-43.bin"
+
+        _run(go())
+
+    def test_document_staging_filename_gets_visible_safe_fallback(self):
+        from app.channels.telegram import TelegramChannel
+
+        document = SimpleNamespace(
+            file_id="document-id",
+            file_unique_id="document-unique",
+            file_name=".upload-hidden.part",
+            mime_type="application/pdf",
+            file_size=12,
+        )
+        update = _make_telegram_update("private", message_id=44, text=None, document=document)
+
+        files = TelegramChannel._extract_inbound_files(update.message)
+
+        assert files[0]["filename"] == "telegram-document-44.bin"
+
+    def test_receive_file_downloads_bytes_without_retaining_telegram_file_id(self):
+        from app.channels.telegram import TelegramChannel
+
+        async def go():
+            bus = MessageBus()
+            ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
+            downloaded = bytearray(b"data")
+            telegram_file = SimpleNamespace(file_size=4, download_as_bytearray=AsyncMock(return_value=downloaded))
+            bot = SimpleNamespace(get_file=AsyncMock(return_value=telegram_file))
+            ch._application = SimpleNamespace(bot=bot)
+            msg = InboundMessage(
+                channel_name="telegram",
+                chat_id="100",
+                user_id="42",
+                text="caption",
+                files=[
+                    {
+                        "type": "file",
+                        "file_id": "document-id",
+                        "file_unique_id": "document-unique",
+                        "filename": "report.pdf",
+                        "mime_type": "application/pdf",
+                        "size": 4,
+                    }
+                ],
+            )
+
+            result = await ch.receive_file(msg, "thread-1")
+
+            bot.get_file.assert_awaited_once_with("document-id")
+            telegram_file.download_as_bytearray.assert_awaited_once_with()
+            assert result.text == "caption"
+            assert result.files[0]["_content"] is downloaded
+            assert result.files == [
+                {
+                    "type": "file",
+                    "filename": "report.pdf",
+                    "mime_type": "application/pdf",
+                    "size": 4,
+                    "_content": b"data",
+                }
+            ]
+
+        _run(go())
+
+    def test_receive_file_marshals_ptb_download_to_telegram_event_loop(self):
+        from app.channels.telegram import TelegramChannel
+
+        async def go():
+            bus = MessageBus()
+            ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
+            telegram_loop = asyncio.new_event_loop()
+            loop_started: Future[None] = Future()
+
+            def run_telegram_loop():
+                asyncio.set_event_loop(telegram_loop)
+                telegram_loop.call_soon(loop_started.set_result, None)
+                telegram_loop.run_forever()
+
+            loop_thread = threading.Thread(target=run_telegram_loop, daemon=True)
+            loop_thread.start()
+            try:
+                loop_started.result(timeout=2)
+
+                class LoopBoundFile:
+                    file_size = 4
+
+                    async def download_as_bytearray(self):
+                        assert asyncio.get_running_loop() is telegram_loop
+                        return bytearray(b"data")
+
+                class LoopBoundBot:
+                    async def get_file(self, file_id):
+                        assert asyncio.get_running_loop() is telegram_loop
+                        assert file_id == "document-id"
+                        return LoopBoundFile()
+
+                ch._tg_loop = telegram_loop
+                ch._application = SimpleNamespace(bot=LoopBoundBot())
+                msg = InboundMessage(
+                    channel_name="telegram",
+                    chat_id="100",
+                    user_id="42",
+                    text="caption",
+                    files=[{"type": "file", "file_id": "document-id", "filename": "report.pdf", "size": 4}],
+                )
+                result = await ch.receive_file(msg, "thread-1")
+            finally:
+                telegram_loop.call_soon_threadsafe(telegram_loop.stop)
+                await asyncio.to_thread(loop_thread.join, 2)
+                if loop_thread.is_alive():
+                    pytest.fail("Telegram test event loop did not stop")
+                telegram_loop.close()
+
+            assert result.files[0]["_content"] == b"data"
+
+        _run(go())
+
+    def test_stop_cancels_and_drains_an_inflight_receive_download(self):
+        from app.channels.telegram import TelegramChannel
+
+        async def go():
+            bus = MessageBus()
+            ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
+            telegram_loop = asyncio.new_event_loop()
+            loop_started: Future[None] = Future()
+            download_started = threading.Event()
+            download_cancelled = threading.Event()
+
+            def run_telegram_loop():
+                asyncio.set_event_loop(telegram_loop)
+                telegram_loop.call_soon(loop_started.set_result, None)
+                telegram_loop.run_forever()
+
+            class SlowTelegramFile:
+                file_size = 4
+
+                async def download_as_bytearray(self):
+                    download_started.set()
+                    try:
+                        await asyncio.Event().wait()
+                    finally:
+                        download_cancelled.set()
+
+            class LoopBoundBot:
+                async def get_file(self, file_id):
+                    assert asyncio.get_running_loop() is telegram_loop
+                    assert file_id == "document-id"
+                    return SlowTelegramFile()
+
+            loop_thread = threading.Thread(target=run_telegram_loop, daemon=True)
+            loop_thread.start()
+            try:
+                loop_started.result(timeout=2)
+                ch._tg_loop = telegram_loop
+                ch._thread = loop_thread
+                ch._running = True
+                ch._application = SimpleNamespace(bot=LoopBoundBot())
+                msg = InboundMessage(
+                    channel_name="telegram",
+                    chat_id="100",
+                    user_id="42",
+                    text="caption",
+                    files=[{"type": "file", "file_id": "document-id", "filename": "report.pdf", "size": 4}],
+                )
+                receive_task = asyncio.create_task(ch.receive_file(msg, "thread-1"))
+                assert await asyncio.to_thread(download_started.wait, 2)
+
+                await ch.stop()
+
+                with pytest.raises(asyncio.CancelledError):
+                    await receive_task
+                assert download_cancelled.is_set()
+                assert not ch._tg_bridge_tasks
+            finally:
+                if telegram_loop.is_running():
+                    telegram_loop.call_soon_threadsafe(telegram_loop.stop)
+                await asyncio.to_thread(loop_thread.join, 2)
+                if loop_thread.is_alive():
+                    pytest.fail("Telegram test event loop did not stop")
+                telegram_loop.close()
+
+        _run(go())
+
+    def test_cancelled_stop_still_stops_thread_and_clears_state(self):
+        from app.channels.telegram import TelegramChannel
+
+        async def go():
+            bus = MessageBus()
+            ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
+            telegram_loop = asyncio.new_event_loop()
+            loop_started: Future[None] = Future()
+            drain_started = threading.Event()
+
+            def run_telegram_loop():
+                asyncio.set_event_loop(telegram_loop)
+                telegram_loop.call_soon(loop_started.set_result, None)
+                telegram_loop.run_forever()
+
+            async def slow_drain():
+                drain_started.set()
+                await asyncio.Event().wait()
+
+            loop_thread = threading.Thread(target=run_telegram_loop, daemon=True)
+            loop_thread.start()
+            try:
+                loop_started.result(timeout=2)
+                ch._tg_loop = telegram_loop
+                ch._thread = loop_thread
+                ch._running = True
+                ch._application = SimpleNamespace(bot=SimpleNamespace())
+                ch._cancel_telegram_bridge_tasks = slow_drain
+
+                stop_task = asyncio.create_task(ch.stop())
+                assert await asyncio.to_thread(drain_started.wait, 2)
+                stop_task.cancel()
+
+                with pytest.raises(asyncio.CancelledError):
+                    await stop_task
+
+                assert not loop_thread.is_alive()
+                assert ch._thread is None
+                assert ch._application is None
+            finally:
+                if telegram_loop.is_running():
+                    telegram_loop.call_soon_threadsafe(telegram_loop.stop)
+                await asyncio.to_thread(loop_thread.join, 2)
+                if loop_thread.is_alive():
+                    pytest.fail("Telegram test event loop did not stop")
+                telegram_loop.close()
+
+        _run(go())
+
+    def test_receive_file_does_not_reuse_ptb_client_after_telegram_loop_stops(self):
+        from app.channels.telegram import TelegramChannel
+
+        async def go():
+            bus = MessageBus()
+            ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
+            stopped_loop = asyncio.new_event_loop()
+            bot = SimpleNamespace(get_file=AsyncMock())
+            ch._tg_loop = stopped_loop
+            ch._application = SimpleNamespace(bot=bot)
+            msg = InboundMessage(
+                channel_name="telegram",
+                chat_id="100",
+                user_id="42",
+                text="caption",
+                files=[{"type": "file", "file_id": "document-id", "filename": "report.pdf", "size": 4}],
+            )
+
+            try:
+                result = await ch.receive_file(msg, "thread-1")
+            finally:
+                stopped_loop.close()
+
+            bot.get_file.assert_not_awaited()
+            assert result.files == []
+            assert result.text.startswith("caption")
+            assert "report.pdf" in result.text
+
+        _run(go())
+
+    def test_receive_file_rejects_declared_oversize_before_download(self):
+        from app.channels.telegram import TELEGRAM_MAX_INBOUND_FILE_BYTES, TelegramChannel
+
+        async def go():
+            bus = MessageBus()
+            ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
+            bot = SimpleNamespace(get_file=AsyncMock())
+            ch._application = SimpleNamespace(bot=bot)
+            msg = InboundMessage(
+                channel_name="telegram",
+                chat_id="100",
+                user_id="42",
+                text="",
+                files=[
+                    {
+                        "type": "file",
+                        "file_id": "too-large",
+                        "filename": "archive.zip",
+                        "mime_type": "application/zip",
+                        "size": TELEGRAM_MAX_INBOUND_FILE_BYTES + 1,
+                    }
+                ],
+            )
+
+            result = await ch.receive_file(msg, "thread-1")
+
+            bot.get_file.assert_not_called()
+            assert result.files == []
+            assert "archive.zip" in result.text
+            assert "20 MB" in result.text
+
+        _run(go())
+
+    def test_receive_file_rejects_download_larger_than_reported(self, monkeypatch):
+        from app.channels import telegram
+
+        async def go():
+            monkeypatch.setattr(telegram, "TELEGRAM_MAX_INBOUND_FILE_BYTES", 3)
+            bus = MessageBus()
+            ch = telegram.TelegramChannel(bus=bus, config={"bot_token": "test-token"})
+            telegram_file = SimpleNamespace(file_size=2, download_as_bytearray=AsyncMock(return_value=bytearray(b"four")))
+            ch._application = SimpleNamespace(bot=SimpleNamespace(get_file=AsyncMock(return_value=telegram_file)))
+            msg = InboundMessage(
+                channel_name="telegram",
+                chat_id="100",
+                user_id="42",
+                text="caption",
+                files=[{"type": "image", "file_id": "photo-id", "filename": "photo.jpg", "mime_type": "image/jpeg", "size": 2}],
+            )
+
+            result = await ch.receive_file(msg, "thread-1")
+
+            assert result.files == []
+            assert result.text.startswith("caption")
+            assert "photo.jpg" in result.text
+
+        _run(go())
+
+    def test_receive_file_rejects_resolved_oversize_before_downloading(self, monkeypatch):
+        from app.channels import telegram
+
+        async def go():
+            monkeypatch.setattr(telegram, "TELEGRAM_MAX_INBOUND_FILE_BYTES", 3)
+            bus = MessageBus()
+            ch = telegram.TelegramChannel(bus=bus, config={"bot_token": "test-token"})
+            download = AsyncMock(return_value=bytearray(b"four"))
+            telegram_file = SimpleNamespace(file_size=4, download_as_bytearray=download)
+            ch._application = SimpleNamespace(bot=SimpleNamespace(get_file=AsyncMock(return_value=telegram_file)))
+            msg = InboundMessage(
+                channel_name="telegram",
+                chat_id="100",
+                user_id="42",
+                text="",
+                files=[{"type": "file", "file_id": "file-id", "filename": "large.bin", "size": 2}],
+            )
+
+            result = await ch.receive_file(msg, "thread-1")
+
+            download.assert_not_awaited()
+            assert result.files == []
+            assert "large.bin" in result.text
+
+        _run(go())
+
+    def test_receive_file_accepts_exact_download_limit(self, monkeypatch):
+        from app.channels import telegram
+
+        async def go():
+            monkeypatch.setattr(telegram, "TELEGRAM_MAX_INBOUND_FILE_BYTES", 4)
+            bus = MessageBus()
+            ch = telegram.TelegramChannel(bus=bus, config={"bot_token": "test-token"})
+            telegram_file = SimpleNamespace(file_size=4, download_as_bytearray=AsyncMock(return_value=bytearray(b"four")))
+            ch._application = SimpleNamespace(bot=SimpleNamespace(get_file=AsyncMock(return_value=telegram_file)))
+            msg = InboundMessage(
+                channel_name="telegram",
+                chat_id="100",
+                user_id="42",
+                text="",
+                files=[{"type": "file", "file_id": "file-id", "filename": "exact.bin", "size": 4}],
+            )
+
+            result = await ch.receive_file(msg, "thread-1")
+
+            assert result.files[0]["_content"] == b"four"
+            assert result.files[0]["size"] == 4
+
+        _run(go())
+
+    def test_receive_file_download_failure_keeps_caption_and_drops_attachment(self, caplog):
+        from app.channels.telegram import TelegramChannel
+
+        async def go():
+            bus = MessageBus()
+            ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
+            ch._application = SimpleNamespace(bot=SimpleNamespace(get_file=AsyncMock(side_effect=RuntimeError("GET https://api.telegram.org/bottest-token/getFile failed"))))
+            msg = InboundMessage(
+                channel_name="telegram",
+                chat_id="100",
+                user_id="42",
+                text="Summarize this",
+                files=[{"type": "file", "file_id": "document-id", "filename": "report.pdf", "mime_type": "application/pdf", "size": 10}],
+            )
+
+            with caplog.at_level(logging.ERROR):
+                result = await ch.receive_file(msg, "thread-1")
+
+            assert result.files == []
+            assert result.text.startswith("Summarize this")
+            assert "report.pdf" in result.text
+            assert "test-token" not in caplog.text
+
+        _run(go())
+
     def test_private_chat_slash_skill_text_routes_as_chat(self):
         from app.channels.telegram import TelegramChannel
 
         async def go():
             bus = MessageBus()
             ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
-            ch._main_loop = asyncio.get_event_loop()
+            ch._main_loop = asyncio.get_running_loop()
 
             update = _make_telegram_update("private", message_id=12, text="/data-analysis analyze uploads/foo.csv")
             await ch._on_text(update, None)
@@ -8018,7 +8549,7 @@ class TestTelegramPrivateChatThread:
         async def go():
             bus = MessageBus()
             ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
-            ch._main_loop = asyncio.get_event_loop()
+            ch._main_loop = asyncio.get_running_loop()
 
             update = _make_telegram_update(
                 "group",
@@ -8041,7 +8572,7 @@ class TestTelegramPrivateChatThread:
         async def go():
             bus = MessageBus()
             ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
-            ch._main_loop = asyncio.get_event_loop()
+            ch._main_loop = asyncio.get_running_loop()
 
             update = _make_telegram_update("private", message_id=11, reply_to_message_id=5)
             await ch._on_text(update, None)
@@ -8057,7 +8588,7 @@ class TestTelegramPrivateChatThread:
         async def go():
             bus = MessageBus()
             ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
-            ch._main_loop = asyncio.get_event_loop()
+            ch._main_loop = asyncio.get_running_loop()
 
             update = _make_telegram_update("group", message_id=20)
             await ch._on_text(update, None)
@@ -8073,7 +8604,7 @@ class TestTelegramPrivateChatThread:
         async def go():
             bus = MessageBus()
             ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
-            ch._main_loop = asyncio.get_event_loop()
+            ch._main_loop = asyncio.get_running_loop()
 
             update = _make_telegram_update("group", message_id=21, reply_to_message_id=15)
             await ch._on_text(update, None)
@@ -8089,7 +8620,7 @@ class TestTelegramPrivateChatThread:
         async def go():
             bus = MessageBus()
             ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
-            ch._main_loop = asyncio.get_event_loop()
+            ch._main_loop = asyncio.get_running_loop()
 
             update = _make_telegram_update("supergroup", message_id=25)
             await ch._on_text(update, None)
@@ -8105,7 +8636,7 @@ class TestTelegramPrivateChatThread:
         async def go():
             bus = MessageBus()
             ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
-            ch._main_loop = asyncio.get_event_loop()
+            ch._main_loop = asyncio.get_running_loop()
 
             update = _make_telegram_update("private", message_id=30, text="/new")
             await ch._cmd_generic(update, None)
@@ -8122,7 +8653,7 @@ class TestTelegramPrivateChatThread:
         async def go():
             bus = MessageBus()
             ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
-            ch._main_loop = asyncio.get_event_loop()
+            ch._main_loop = asyncio.get_running_loop()
 
             update = _make_telegram_update("group", message_id=31, text="/status")
             await ch._cmd_generic(update, None)
@@ -8139,7 +8670,7 @@ class TestTelegramPrivateChatThread:
         async def go():
             bus = MessageBus()
             ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
-            ch._main_loop = asyncio.get_event_loop()
+            ch._main_loop = asyncio.get_running_loop()
 
             update = _make_telegram_update("group", message_id=32, reply_to_message_id=20, text="/status")
             await ch._cmd_generic(update, None)
@@ -8156,7 +8687,7 @@ class TestTelegramPrivateChatThread:
         async def go():
             bus = MessageBus()
             ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
-            ch._main_loop = asyncio.get_event_loop()
+            ch._main_loop = asyncio.get_running_loop()
 
             update = _make_telegram_update("group", message_id=33, text="/status@DeerFlowBot")
             context = SimpleNamespace(bot=SimpleNamespace(username="DeerFlowBot"))
@@ -8180,7 +8711,7 @@ class TestTelegramProcessingOrder:
             bus = MessageBus()
             ch = TelegramChannel(bus=bus, config={"bot_token": "test-token"})
 
-            ch._main_loop = asyncio.get_event_loop()
+            ch._main_loop = asyncio.get_running_loop()
 
             order = []
 

backend/tests/test_checkpoint_patches.py

@@ -0,0 +1,91 @@
+"""Guards for the BinaryOperatorAggregate first-write Overwrite patch (#4380).
+
+Upstream ``BinaryOperatorAggregate.update`` has a fast path for empty channels
+(``self.value is MISSING``) that stores the first incoming value without
+unwrapping ``Overwrite``. Reducer channels whose type is a Union
+(``SandboxState | None``, ``GoalState | None``, ...) have no constructible
+default, so they start MISSING — a replace-style write into a fresh thread
+(thread branching) or a never-written channel (state update) persists the
+wrapper literally, and the next consumer crashes with ``TypeError:
+'Overwrite' object is not subscriptable``. ``DeltaChannel`` already unwraps in
+the same situation; the patch aligns ``BinaryOperatorAggregate`` with it.
+"""
+
+import pytest
+from langgraph.channels.binop import BinaryOperatorAggregate
+from langgraph.errors import InvalidUpdateError
+from langgraph.types import Overwrite
+
+import deerflow.agents.thread_state  # noqa: F401 - applies checkpoint patches at import
+from deerflow import checkpoint_patches
+
+
+def _replace(existing, new):
+    return new
+
+
+def _empty_union_channel() -> BinaryOperatorAggregate:
+    """A channel shaped like ``sandbox``/``goal``/``todos``: Union type, starts MISSING."""
+    channel = BinaryOperatorAggregate(dict | None, _replace)
+    channel.key = "probe"
+    return channel
+
+
+def test_first_write_overwrite_unwraps_into_empty_channel() -> None:
+    channel = _empty_union_channel()
+    channel.update([Overwrite({"sandbox_id": "local:thread-2"})])
+
+    stored = channel.get()
+    assert not isinstance(stored, Overwrite)
+    # The exact read that crashed in #4380 (SandboxMiddleware.aafter_agent).
+    assert stored["sandbox_id"] == "local:thread-2"
+
+
+def test_overwrite_into_populated_channel_still_replaces() -> None:
+    channel = _empty_union_channel()
+    channel.update([{"sandbox_id": "old"}])
+    channel.update([Overwrite({"sandbox_id": "new"})])
+
+    assert channel.get() == {"sandbox_id": "new"}
+
+
+def test_plain_first_write_still_seeds_then_reduces() -> None:
+    channel = _empty_union_channel()
+    channel.update([{"sandbox_id": "first"}])
+    assert channel.get() == {"sandbox_id": "first"}
+
+    channel.update([{"sandbox_id": "second"}])
+    assert channel.get() == {"sandbox_id": "second"}
+
+
+def test_values_after_first_write_overwrite_are_ignored() -> None:
+    """Upstream semantics: after an Overwrite, later plain values in the batch are skipped."""
+    channel = _empty_union_channel()
+    channel.update([Overwrite({"sandbox_id": "kept"}), {"sandbox_id": "ignored"}])
+
+    assert channel.get() == {"sandbox_id": "kept"}
+
+
+def test_double_overwrite_in_one_batch_raises_on_empty_channel() -> None:
+    channel = _empty_union_channel()
+    with pytest.raises(InvalidUpdateError):
+        channel.update([Overwrite({"sandbox_id": "a"}), Overwrite({"sandbox_id": "b"})])
+
+
+def test_binop_overwrite_patch_is_active() -> None:
+    """The compatibility patch must be applied in every test/app process."""
+    assert getattr(BinaryOperatorAggregate, checkpoint_patches._BINOP_PATCH_FLAG, False) is True
+    assert BinaryOperatorAggregate.update is checkpoint_patches._binop_update_unwrapping_empty_channel
+
+
+def test_binop_overwrite_patch_stands_down_when_upstream_fixed(monkeypatch) -> None:
+    """If upstream unwraps the first write itself, the patch must not reinstall."""
+    monkeypatch.setattr(checkpoint_patches, "_binop_first_write_stores_overwrite_wrapper", lambda: False)
+    monkeypatch.delattr(BinaryOperatorAggregate, checkpoint_patches._BINOP_PATCH_FLAG, raising=False)
+    sentinel = object()
+    monkeypatch.setattr(BinaryOperatorAggregate, "update", sentinel)
+
+    checkpoint_patches.ensure_binop_overwrite_first_write_patch()
+
+    assert getattr(BinaryOperatorAggregate, checkpoint_patches._BINOP_PATCH_FLAG, False) is False
+    assert BinaryOperatorAggregate.update is sentinel