github/deer-flow
1
0
Fork 0
mirror of https://github.com/bytedance/deer-flow.git synced 2026-07-25 15:37:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' into rayhpeng/hexagonal-feedback-slice

Browse Source 
Renumber the feedback tags migration to 0008 on top of main's
0007_scheduled_run_active_index so the alembic chain keeps a single
head; bootstrap head pins follow.
This commit is contained in:
rayhpeng 2026-07-25 18:28:34 +08:00
commit c9ee725c0b
79 changed files with 5934 additions and 441 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
CHANGELOG.md
View File

@ -12,6 +12,11 @@ This section accumulates work toward the **2.1.0** milestone
### ⚠ Breaking changes
- **sandbox:** E2B now enforces `sandbox.replicas` as a process-local capacity
limit. The default `wait` policy waits for `acquire_timeout`, then fails the
agent turn. DeerFlow does not retry the turn automatically. Use `burst` with
`burst_limit` to permit bounded extra VMs. The `reject` policy can remove one
warm VM before it returns a capacity error. ([#4391])
- **skills:** A directory containing `SKILL.md` is now a runtime package
boundary. Nested `SKILL.md` files inside that package are supporting data and
are no longer registered as independent skills; unusual custom layouts must

30
README.md
View File

@ -275,7 +275,7 @@ Browser login uses `HttpOnly` session cookies. The login page offers a "keep me
DeerFlow still uses `Forwarded` / `X-Forwarded-*` headers to recover the browser-facing scheme and origin behind a proxy. The bundled nginx sets `X-Forwarded-Proto`, but preserves an upstream HTTPS value and does not overwrite every forwarded header. Configure the outer trusted proxy to replace or strip client-supplied forwarding headers before traffic reaches DeerFlow.
> [!IMPORTANT]
> The Gateway still owns active run tasks in process, so production defaults to a single Gateway worker (`GATEWAY_WORKERS=1`). The Redis stream bridge (`stream_bridge.type: redis`) shares SSE delivery and `Last-Event-ID` replay across workers, with a rolling retained-buffer TTL (`stream_ttl_seconds`) as a cleanup safety net. Malformed reconnect IDs live-tail new events instead of replaying the retained buffer. It does not make run cancellation, request de-duplication, or IM channel state fully cross-worker by itself; use single-worker Gateway or explicit sticky routing/ownership before raising `GATEWAY_WORKERS`.
> The Gateway still owns active run tasks in process, so production defaults to a single Gateway worker (`GATEWAY_WORKERS=1`). Multi-worker deployments require Postgres, the Redis stream bridge (`stream_bridge.type: redis`), and `run_ownership.heartbeat_enabled: true`. The bridge shares SSE delivery and `Last-Event-ID` replay across workers, while lease reconciliation marks runs from dead workers as errors, publishes the terminal stream marker, schedules retained-stream cleanup, and updates the affected thread status. SSE and `/wait` consumers also refresh durable status on heartbeats as a fallback if terminal publication fails. The rolling retained-buffer TTL (`stream_ttl_seconds`) remains a cleanup safety net rather than a run timeout. IM channel state and other process-local services still need their own multi-worker coordination.
See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed Docker development guide.
@ -765,12 +765,32 @@ Use `/compact` in the Web UI composer to summarize older context for the current
Complex tasks rarely fit in a single pass. DeerFlow decomposes them.
The lead agent can spawn sub-agents on the fly — each with its own scoped context, tools, and termination conditions. Sub-agents run in parallel when possible, report back structured results, and the lead agent synthesizes everything into a coherent output. Their configured skills are resolved from the same user-scoped catalog as the lead agent, so user-owned custom skills remain available without exposing another user's version. Their internal AI and tool messages stay scoped to the delegated graph instead of entering the parent chat stream. Long-running sub-agents compact older history when summarization is enabled and re-inject the summary as guarded, hidden durable context before continuing, so recent assistant/tool activity remains grounded in the task. Provider/model request failures are reported as failed sub-agent tasks rather than successful results, so the lead agent and Web UI can react to them correctly. Collapsed sub-agent cards show the effective model and, when the provider returns usage metadata, a cumulative token total that updates after each completed sub-agent LLM call and persists after a reload. When token usage tracking is enabled, completed sub-agent usage is also attributed back to the dispatching step.
The lead agent can spawn sub-agents on the fly — each with its own scoped context, tools, and termination conditions. Sub-agents run in parallel when possible, report back structured results, and the lead agent synthesizes everything into a coherent output. Their configured skills are resolved from the same user-scoped catalog as the lead agent, so user-owned custom skills remain available without exposing another user's version. Their internal AI and tool messages stay scoped to the delegated graph instead of entering the parent chat stream. Reloaded thread history enforces the same boundary: callback-captured sub-agent AI responses remain available in run-event diagnostics but are excluded from the parent transcript, while the parent `task` result remains attached to its subtask card. Long-running sub-agents compact older history when summarization is enabled and re-inject the summary as guarded, hidden durable context before continuing, so recent assistant/tool activity remains grounded in the task. Provider/model request failures are reported as failed sub-agent tasks rather than successful results, so the lead agent and Web UI can react to them correctly. Collapsed sub-agent cards show the effective model and, when the provider returns usage metadata, a cumulative token total that updates after each completed sub-agent LLM call and persists after a reload. When token usage tracking is enabled, completed sub-agent usage is also attributed back to the dispatching step.
This is how DeerFlow handles tasks that take minutes to hours: a research task might fan out into a dozen sub-agents, each exploring a different angle, then converge into a single report — or a website — or a slide deck with generated visuals. One harness, many hands.
### Sandbox & File System
`E2BSandboxProvider` uses `wait` as its default overflow policy. It waits for
`acquire_timeout`, then fails the agent turn. DeerFlow does not retry the turn
automatically. Clients can use the structured error to schedule a retry.
Use `burst` with `burst_limit` to permit bounded extra VMs. The `wait` and
`reject` policies use only `replicas`. The `reject` policy can remove one warm
VM before it returns an error.
`replicas` limits one Gateway process. It does not limit all Gateway processes.
E2B acquisition uses a bounded executor. Waiting acquisitions do not use the
default asyncio executor.
An E2B VM keeps its slot until E2B confirms destruction. This rule covers
create and reclaim operations. Discovery can find a VM from another Gateway.
Shutdown closes an unowned discovery client without destroying its VM.
Release stops counting a transition when the VM enters the warm pool.
Shutdown races retry remote cleanup after a transient kill failure.
Reset destroys tracked active and warm E2B VMs. The old provider instance
cannot accept new acquisitions.
DeerFlow doesn't just *talk* about doing things. It has its own computer.
Each task gets its own execution environment with a full filesystem view — skills, workspace, uploads, outputs. The agent reads, writes, and edits files. It can view images and, when configured safely, execute shell commands.
@ -865,6 +885,10 @@ response = client.chat("Analyze this paper for me", thread_id="my-thread")
for event in client.stream("hello"):
if event.type == "messages-tuple" and event.data.get("type") == "ai":
print(event.data["content"])
elif event.type == "messages-tuple" and event.data.get("type") == "tool" and "artifact" in event.data:
# Structured tool artifacts (for example, ask_clarification cards)
# are preserved when the ToolMessage provides one.
print(event.data["artifact"])
# Configuration & management — returns Gateway-aligned dicts
models = client.list_models() # {"models": [...]}
@ -876,6 +900,8 @@ client.get_goal("thread-1") # {"goal": {...}} or {"goal": None}
client.clear_goal("thread-1")
```
The HTTP Gateway accepts `values`, `messages-tuple`, `updates`, `debug`, `tasks`, `checkpoints`, and `custom` stream modes. Unsupported modes such as `messages` and `events`, unsupported non-default run options such as webhooks, delayed execution, or `multitask_strategy="enqueue"`, and undeclared SDK options such as checkpoint durability overrides return `422` before execution instead of being silently ignored or downgraded.
All dict-returning methods are validated against Gateway Pydantic response models in CI (`TestGatewayConformance`), ensuring the embedded client stays in sync with the HTTP API schemas. See `backend/packages/harness/deerflow/client.py` for full API documentation.
## Scheduled Tasks

39
backend/AGENTS.md
View File

@ -17,6 +17,7 @@ DeerFlow is a LangGraph-based AI super agent system with a full-stack architectu
- Gateway streams `write_file` and `str_replace` argument deltas in bounded batches when clients also subscribe to `values`; messages-only consumers retain the original per-chunk contract, while `values` preserves the complete tool call.
- With `stream_subgraphs`, subgraph frames keep their namespace in the SSE event name (`values|<ns>`, LangGraph Platform style) instead of impersonating root frames — a delegated subagent inherits the parent checkpoint namespace, so publishing its `values` snapshot as bare `values` replaces the whole thread view in SDK clients (#4399). Root-only consumers (file-tool chunk batcher, subagent event persistence, LLM error-fallback detection) ignore namespaced frames. The web frontend does not request subgraph streaming; subtask progress rides root-namespace `task_*` custom events.
- Scheduled-task executions must reuse that same Gateway run lifecycle. The scheduler may decide *when* work runs, but it must dispatch through the existing run path rather than introducing a parallel execution stack.
- Scheduled-task dispatch enforces "at most one active run per task when `overlap_policy=skip`" at the DB layer via the partial unique index `uq_scheduled_task_run_active` (`scheduled_task_runs.task_id WHERE status IN ('queued','running')`). `ScheduledTaskService.dispatch_task`'s `has_active_runs` check is a non-atomic fast path (its own session, separated from the `create()` insert by `await` points), so two concurrent dispatches — a manual `POST /scheduled-tasks/{id}/trigger` racing the poller, a double-click, or a client retry — can both pass it; the index is the atomic arbiter, and the losing `create` surfaces as `ActiveScheduledRunConflict` (translated from `IntegrityError` in the repository) and collapses to the same outcome as the fast path (manual → 409 conflict, scheduled → a `"skipped"` tombstone). The scheduled-skip tombstone is created directly as terminal `"skipped"` (not a transient `"queued"`) so it never occupies the active slot the pre-existing run still holds. Sibling of the `runs` table's `uq_runs_thread_active` (PR #4003), which keys on `thread_id` and so does not cover the default `fresh_thread_per_run` context where every dispatch gets a new thread. Index is status-only, not `overlap_policy`-conditional (the policy is fixed to `"skip"` in the MVP).
**Project Structure**:
```
@ -341,7 +342,7 @@ Before changing a later authorization phase, read the [authorization RFC](../doc
24. **McpRoutingMiddleware** - *(optional, if `tool_search.enabled` and PR1 MCP routing metadata produce a routing index)* Auto-promotes matching deferred MCP tool schemas before the model call by writing a minimal `promoted` state update. It matches only the latest real `HumanMessage`, uses the global `tool_search.auto_promote_top_k` limit (default 3, clamped to 1..5), never executes tools, and must be installed before `DeferredToolFilterMiddleware`
25. **DeferredToolFilterMiddleware** - *(optional, if `tool_search.enabled`)* Hides deferred (MCP) tool schemas from the bound model until `tool_search` or `McpRoutingMiddleware` promotes them (reads per-thread promotions from `ThreadState.promoted`, hash-scoped)
26. **SystemMessageCoalescingMiddleware** - Merges every SystemMessage into a single leading SystemMessage per request; provider-agnostic fix for strict backends (vLLM/SGLang/Qwen/Anthropic) that reject non-leading system messages. Touches the per-request payload only (checkpoint state unchanged); on midnight crossings only the latest `dynamic_context_reminder` SystemMessage survives
27. **SubagentLimitMiddleware** - *(optional, if `subagent_enabled`)* Truncates excess `task` tool calls to enforce both the per-response concurrency limit (`max_concurrent_subagents`, clamped to 2-4) and the per-run total delegation cap (`max_total_subagents` runtime override or `subagents.max_total_per_run`, default 6, clamped to 1-50). The total cap counts current-run entries in the durable delegation ledger (entries are tagged with `run_id` when captured), so repeated planning checkpoints in one run cannot keep launching legal-sized batches indefinitely, while later user turns in the same thread get a fresh run budget. If the cap is exhausted, the middleware strips remaining `task` calls, forces `finish_reason="stop"`, and appends a visible limit note so the run can synthesize existing results instead of ending with an empty tool-call response.
27. **SubagentLimitMiddleware** - *(optional, if `subagent_enabled`)* Truncates excess `task` tool calls to enforce both the per-response concurrency limit (`max_concurrent_subagents`, clamped to 1-4) and the per-run total delegation cap (`max_total_subagents` runtime override or `subagents.max_total_per_run`, default 6, clamped to 1-50). The total cap counts current-run entries in the durable delegation ledger (entries are tagged with `run_id` when captured), so repeated planning checkpoints in one run cannot keep launching legal-sized batches indefinitely, while later user turns in the same thread get a fresh run budget. If the cap is exhausted, the middleware strips remaining `task` calls, forces `finish_reason="stop"`, and appends a visible limit note so the run can synthesize existing results instead of ending with an empty tool-call response.
28. **LoopDetectionMiddleware** - *(optional, if `loop_detection.enabled`)* Detects repeated tool-call loops; hard-stop clears both structured `tool_calls` and raw provider tool-call metadata before forcing a final text answer; stamps `loop_capped` via `consume_stop_reason` (#3875 Phase 2), symmetric to `TokenBudgetMiddleware`
29. **TokenBudgetMiddleware** - *(optional, if `token_budget.enabled`)* Enforces per-run token limits
30. **Custom middlewares** - *(optional)* Any `custom_middlewares` passed to `build_middlewares` are injected here, before config-declared extensions and the terminal-response/safety/clarification tail
@ -421,7 +422,7 @@ Localhost persistence deliberately reads the direct request `Host` and ignores `
| **Artifacts** (`/api/threads/{id}/artifacts`) | `GET /{path}` - serve artifacts; active content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) are always forced as download attachments to reduce XSS risk; `?download=true` still forces download for other file types |
| **Suggestions** (`/api/suggestions`) | `GET /config` - returns global suggestions config boolean; `POST /threads/{id}/suggestions` - generate follow-up questions; rich list/block model content is normalized and inline reasoning (`<think>...</think>`, including unclosed/truncated blocks from reasoning models like MiniMax-M3) is stripped before JSON parsing |
| **Input Polish** (`/api/input-polish`) | `POST /` - rewrite a composer draft before it is sent. This is a short authenticated `runs:create` LLM request using `input_polish` config; it does not create a LangGraph run, persist a message, or modify thread state. Shares the non-graph one-shot LLM path (`deerflow.utils.oneshot_llm.run_oneshot_llm`) with the suggestions route so model build + Langfuse metadata + invoke stay in one place; validates the same stripped view of the draft it sends to the model, and preserves literal `<think>` substrings in the rewrite (`strip_think_blocks(truncate_unclosed=False)`) |
| **Thread Runs** (`/api/threads/{id}/runs`) | `POST /` - create background run; `POST /stream` - create + SSE stream; `POST /wait` - create + block; `POST /regenerate/prepare` - prepare clean input + checkpoint metadata for regenerating the latest assistant answer; `GET /` - list runs; `GET /{rid}` - run details; `POST /{rid}/cancel` - cancel; `GET /{rid}/join` - join SSE; `GET /{rid}/messages` - paginated per-run messages `{data, has_more}`; `GET /{rid}/events` - full event stream; `GET /{rid}/workspace-changes` - workspace/output file change summary and optional diffs; `GET /../messages` - legacy thread message array; `GET /../messages/page` - backward thread-global `seq` history page with middleware/successful-regenerate filtering and page-run-scoped feedback enrichment; `GET /../token-usage` - aggregate tokens |
| **Thread Runs** (`/api/threads/{id}/runs`) | `POST /` - create background run; `POST /stream` - create + SSE stream; `POST /wait` - create + block; `POST /regenerate/prepare` - prepare clean input + checkpoint metadata for regenerating the latest assistant answer; `GET /` - list runs; `GET /{rid}` - run details; `POST /{rid}/cancel` - cancel; `GET /{rid}/join` - join SSE; `GET /{rid}/messages` - paginated per-run messages `{data, has_more}`; `GET /{rid}/events` - full event stream; `GET /{rid}/workspace-changes` - workspace/output file change summary and optional diffs; `GET /../messages` - legacy thread message array; `GET /../messages/page` - backward thread-global `seq` history page with middleware/subagent-AI/successful-regenerate filtering and page-run-scoped feedback enrichment; subagent AI callbacks remain available through run events while parent `task` ToolMessages stay visible for card restoration; `GET /../token-usage` - aggregate tokens |
| **Feedback** (`/api/threads/{id}/runs/{rid}/feedback`) | `PUT /` - upsert feedback; `DELETE /` - delete user feedback; `POST /` - create feedback; `GET /` - list feedback; `GET /stats` - aggregate stats; `DELETE /{fid}` - delete specific |
| **Runs** (`/api/runs`) | `POST /stream` - stateless run + SSE; `POST /wait` - stateless run + block; `GET /{rid}/messages` - paginated messages by run_id `{data, has_more}` (cursor: `after_seq`/`before_seq`); `GET /{rid}/feedback` - list feedback by run_id |
| **GitHub Webhooks** (`/api/webhooks/github`) | `POST /` - receive GitHub App / repo webhook deliveries. Verifies `X-Hub-Signature-256` against `GITHUB_WEBHOOK_SECRET`; exempt from auth + CSRF because authenticity is enforced by HMAC. The route is fail-closed: mounted only when `GITHUB_WEBHOOK_SECRET` is set, or when explicit dev opt-in `DEER_FLOW_ALLOW_UNVERIFIED_GITHUB_WEBHOOKS=1` is set. Recognized events include `ping`, `issues`, `issue_comment`, `pull_request`, `pull_request_review`, and `pull_request_review_comment`; unknown events return 200 with `handled=false`. Fan-out runtime failures return 503, keeping the delivery recorded as failed for manual/API/scripted redelivery (GitHub does not automatically retry any failed delivery, 5xx included); permanent/non-retryable conditions such as `channels.github.enabled: false`, unknown events, malformed payloads, or unavailable channel service return 200 with a skipped/handled response. |
@ -436,6 +437,7 @@ are size-limited; binary, large, and sensitive-looking paths are persisted as
metadata only.
**RunManager / RunStore contract**:
- LangGraph-compatible run requests validate their supported subset before creating a run. `runtime/stream_modes.py` is the shared backend contract for public stream modes and the worker's `graph.astream` mapping; the public `messages-tuple` mode maps to LangGraph's internal `messages` mode, while public `messages`, `events`, and other unsupported modes are rejected instead of being dropped or replaced with `values`. `app/gateway/run_models.py::RunCreateRequest` is shared by HTTP and internal scheduled launch paths, retains only truthful compatibility defaults for unimplemented options (`if_not_exists="create"` plus `None` placeholders), returns 422 for unsupported values including `on_completion="complete"`, `on_completion="continue"`, and `multitask_strategy="enqueue"`, and forbids undeclared SDK options so fields such as `checkpoint_during` and `durability` cannot be silently discarded.
- `RunManager.get()` is async; direct callers must `await` it.
- The history batch helpers `list_successful_regenerate_sources()` and `get_many_by_thread()` default to `user_id=AUTO`: they resolve the request user and fail closed when no user context exists. Migration/admin callers that intentionally need an unscoped read must pass `user_id=None` explicitly.
- When a persistent `RunStore` is configured, `get()` and `list_by_thread()` hydrate historical runs from the store. In-memory records win for the same `run_id` so task, abort, and stream-control state stays attached to active local runs.
@ -443,7 +445,7 @@ metadata only.
- Store-only hydrated runs are readable history. In multi-worker mode with heartbeat enabled, cancel on a store-only run can take over (mark `error`) when the owner's lease has expired past the grace window; otherwise it fails with 409 + `Retry-After`. In single-worker mode (heartbeat off), store-only runs still return 409.
- Startup/orphan reconciliation must claim stale active rows with `RunStore.claim_for_takeover()`, not a plain `update_status()`. The final claim re-checks `status` and lease expiry atomically, so a heartbeat renewal between the candidate scan and the recovery write keeps the run active.
- `POST /wait` (both thread-scoped and `/api/runs/wait`) drains the stream bridge via `wait_for_run_completion()` instead of bare `await record.task`, so it honours the run's `on_disconnect` setting and cancels the background run on real client disconnect rather than returning a stale checkpoint (issue #3265).
- Redis `StreamBridge` keys use a rolling retained-buffer TTL (`stream_bridge.stream_ttl_seconds`, refreshed on `publish()` / `publish_end()`) as a leak safety net, not as a run timeout. Startup orphan recovery publishes `END_SENTINEL` and schedules stream cleanup for recovered runs; malformed `Last-Event-ID` reconnect values live-tail new Redis events rather than replaying the retained buffer. Do not broaden this into a shared-database multi-pod reaper without adding worker ownership/liveness first.
- Redis `StreamBridge` keys use a rolling retained-buffer TTL (`stream_bridge.stream_ttl_seconds`, refreshed on `publish()` / `publish_end()`) as a leak safety net, not as a run timeout. Startup and lease-driven periodic orphan recovery share one Gateway stream-terminalization path: after `RunManager` durably marks a run `error` with `stop_reason=orphan_recovered`, Gateway publishes `END_SENTINEL` and schedules stream cleanup. The periodic store scan, per-row status writes, and Gateway callback run as one supervised single-flight task, so a slow pass is skipped at the next interval instead of piling up or pausing the sole lease-renewal loop. Store retries have bounded attempts/backoff; an individual operation still relies on the database driver/pool timeout. `RunManager.shutdown()` gives active user runs priority within its shared deadline, then drains or cancels orphan recovery. Gateway tracks delayed recovered-stream cleanups and converts unfinished delays to immediate deletes before closing the bridge; the Redis TTL remains the outage safety net. Only startup recovery, before the runtime yields to requests, projects the latest affected thread to `error`; periodic recovery deliberately avoids that non-atomic projection because `ThreadMetaStore` has no `latest_run_id` conditional-update contract. Store-only SSE and `/wait` consumers wait for the bridge's real END marker after an ordinary durable terminal status, because status persistence can precede tail events. The explicit `orphan_recovered` signal is the only heartbeat fallback: its publisher is known to be gone, so it supplies the liveness boundary if END publication fails or the retained key expires. Malformed `Last-Event-ID` reconnect values live-tail new Redis events rather than replaying the retained buffer. Keep cross-component recovery orchestration in Gateway through the generic `RunManager.on_orphans_recovered` callback; do not introduce a harness-to-app dependency. Callback failure warnings include every recovered `run_id` so operators can identify rows whose Gateway-side terminalization needs inspection.
- Thread-scoped run creation accepts `checkpoint` / `checkpoint_id`; Gateway validates the checkpoint belongs to the request thread before writing `checkpoint_id` / `checkpoint_ns` into `config.configurable` for LangGraph branching.
- Thread-scoped Gateway runs evaluate an active `ThreadState.goal` after the visible turn completes. `runtime/goal.py` asks a non-thinking evaluator model to judge only visible conversation evidence and return a typed blocker; the evaluator model is created once per run and reused across hidden continuation checks. The evaluator runs after the graph root's tracing scope has already closed, so `create_goal_evaluator_model`/`evaluate_goal_completion` attach their own model-level tracing callbacks (`attach_tracing=True`) and inject Langfuse trace metadata (`thread_id`/`user_id`/`deerflow_trace_id`) directly onto the `ainvoke` call — the same standalone-caller pattern as `oneshot_llm.run_oneshot_llm` and `MemoryUpdater` (see Tracing System below). Satisfied goals are cleared; every non-satisfied evaluation — continuable or stand-down — is persisted with `last_evaluation` (the blocker, reason, and evidence summary; outcomes that stop the loop additionally record a `stand_down_reason` for observability), but only `goal_not_met_yet` evaluations are streamed as hidden `HumanMessage` continuations, and only when a durable assistant end-of-turn checkpoint exists, the run has not been aborted, the thread did not change during evaluation, and the no-progress breaker has not fired. The continuation cap is 8 — a hard maximum in the `0`–`8` range; callers requesting more are clamped (`set_goal`/TUI) or rejected with 422 (`PUT /goal`). The no-progress breaker keys on the latest visible assistant evidence (not the evaluator's free-text reason, which an LLM rewords every turn), so two consecutive continuations that add no new visible assistant output stop the loop after 2 attempts. Model-response cleanup helpers such as think-block stripping and code-fence stripping live in `deerflow.utils.llm_text` so `runtime/goal.py` and Gateway suggestion parsing share the same JSON-prep behavior.
- Run event stream changes must keep producer code, `deerflow/constants.py`, `runtime/events/catalog.py`, `contracts/run_event_stream_contract.json`, `backend/docs/RUN_EVENT_STREAM.md`, and `tests/test_run_event_stream_contract.py` in sync. The dependency-free constants module owns the persisted envelope limits (`event_type` 32 characters, `category` 16) and cross-layer workspace event identity; the catalog owns validated runtime definitions and categories. Dynamic middleware tags are limited to 21 characters after the `middleware:` prefix. The JSON contract owns payload schemas, backend-specific storage semantics, legacy aliases, and compatibility rules; conformance tests require both views and all producer groups to agree. `run.end.content` remains opaque and may retain nested Python values in memory while JSONL/database stores stringify non-JSON nested values, so consumers must not assume backend-identical nested output representations.
@ -468,7 +470,23 @@ copying a raw checkpoint because delta state is not self-contained in one tuple.
**Implementations**:
- `LocalSandboxProvider` - Local filesystem execution. `acquire(thread_id)` returns a per-thread `LocalSandbox` (id `local:{thread_id}`) whose `path_mappings` resolve `/mnt/user-data/{workspace,uploads,outputs}` and `/mnt/acp-workspace` to that thread's host directories, so the public `Sandbox` API honours the `/mnt/user-data` contract uniformly with AIO. `acquire()` / `acquire(None)` keeps the legacy generic singleton (id `local`) for callers without a thread context. Per-thread sandboxes are held in an LRU cache (default 256 entries) guarded by a `threading.Lock`. Legacy global-custom mounts are gated by the same user-scoped skill discovery rule used for prompt/list visibility; providers must not infer visibility from raw directory presence alone.
- `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker-based isolation. Active-cache and warm-pool entries are checked with the backend during acquire/reuse; definitively dead containers are dropped from all in-process maps so the thread can discover or create a fresh sandbox instead of reusing a stale client. Backend health-check failures are treated as unknown, not dead; local discovery likewise treats an unverifiable container as not adoptable and falls through to create rather than failing acquire. `get()` remains an in-memory lookup for event-loop-safe tool paths — it never touches the ownership store (that would be blocking IO on the event loop); ownership is published on acquire/reclaim and refreshed off the event loop by the dedicated renewal thread (`_renew_owned_leases`). Legacy global-custom mounts follow the same shared visibility helper as local and remote providers.
- `E2BSandboxProvider` (`packages/harness/deerflow/community/e2b_sandbox/`) - E2B-backed remote isolation. Acquire and the complete release transition (output sync, timeout refresh, client close, and warm-pool publication) share a per-user/thread lock, so a same-process acquire cannot discover or create a replacement while release is between active and warm states. The provider-wide registry lock is not held across remote IO.
- `E2BSandboxProvider` (`packages/harness/deerflow/community/e2b_sandbox/`) provides E2B remote isolation.
Acquire and release share a per-user and thread lock. The provider lock does
not cover remote IO. `burst_limit` adds capacity only for the `burst` policy.
The `wait` policy fails the turn after `acquire_timeout`. The runtime does not
retry the turn automatically. E2B acquisition uses a bounded executor.
Waiting calls do not consume the default asyncio executor. The `reject`
policy can evict one warm VM before it returns an error. `replicas` limits
one Gateway process. It does not provide multi-process capacity control.
Uncertain cleanup keeps a tombstone slot. Shutdown tracks owned remote
operation IDs. Discovery can find a VM from another Gateway. Shutdown closes
an unowned discovery client without destroying its VM. Release ends its
transition count when the VM enters the warm pool. Local client cleanup does
not consume a second slot. A create that returns after shutdown retries one
failed kill through a new client. An unconfirmed remote ID stays tracked.
`reset()` uses full shutdown semantics. It destroys tracked active and warm
VMs. It wakes capacity waiters. Callers cannot reuse the old provider
instance.
- **Cross-instance ownership store** (`aio_sandbox/ownership/`, #4206): gateway instances sharing a container backend coordinate container ownership through a pluggable lease store, selected by `sandbox.ownership.type` (`memory` | `redis`) and resolved like `stream_bridge` (`factory.py`, lazy per-branch import, `redis` optional extra, `DEER_FLOW_SANDBOX_OWNERSHIP_REDIS_URL` env escape hatch; a set `DEER_FLOW_STREAM_BRIDGE_REDIS_URL` implies a multi-instance deployment and infers `redis`). `memory` is single-instance only and declares `supports_cross_process = False`.
- **A lease answers "who reaps this container", not "who may use it".** That splits the interface in two: `take()` transfers ownership on the **acquire** path (a container is deterministic per user/thread, so consecutive turns legitimately land on different instances — a conditional claim there would strand the thread until the previous lease expired), while `claim()` succeeds only if the container is unowned or already ours and gates every **adopt/reap** path. `release()` never clears a peer's lease.
- **A lease carries a state, and that is what makes the destroy window safe.** `own:` = responsible for this container; `del:` = tearing it down (`claim(..., for_destroy=True)`). `take()` is refused against a `del:` lease, so a container cannot be re-acquired between a destroy path's claim and its container stop. Without the two states an unconditional `take()` would silently overwrite the destroyer's claim and the peer's stop would land on a container the new owner had already handed to an agent — i.e. #4206 again. That pairing is what replaced the previous same-host `flock` guard, which is gone; Redis makes the scope genuinely multi-instance instead of same-host. A destroyer that dies mid-stop leaves a `del:` marker that lapses with the TTL. On the acquire path a refused take raises `SandboxBeingDestroyedError`: the reuse/reclaim paths drop the container and cold-start, and the discover path propagates (falling through to create would collide with the not-yet-removed container name).
@ -516,7 +534,7 @@ copying a raw checkpoint because delta state is not self-contained in one tuple.
**Built-in Agents**: `general-purpose` (all tools except `task`) and `bash` (command specialist)
**User-scoped Skills**: Subagents resolve their configured skills through `get_or_new_user_skill_storage(user_id)` using the parent runtime identity, with `DEFAULT_USER_ID` only when no identity is available. This keeps custom-skill shadowing and visibility aligned with the lead agent instead of reading the global-only catalog.
**Execution**: Dual thread pool - `_scheduler_pool` (3 workers) + `_execution_pool` (3 workers)
**Concurrency and total delegation cap**: `MAX_CONCURRENT_SUBAGENTS = 3` is enforced by `SubagentLimitMiddleware` (truncates excess tool calls in `after_model`; runtime `max_concurrent_subagents` is clamped to 2-4). The same middleware also enforces `subagents.max_total_per_run` (default 6, config schema 1-50, runtime override `max_total_subagents` clamped to the same range) against current-run entries in the durable delegation ledger, so a long lead-agent run cannot bypass concurrency limits by launching repeated legal-sized batches at each planning checkpoint, but historical delegations from previous runs in the same thread do not consume the new run's budget. The lead-agent prompt uses the same clamped values, so model-visible limits match enforcement. Gateway `run_agent()` and embedded `DeerFlowClient.stream()` both provide a per-invocation `run_id` in runtime context; `DeerFlowClient.stream()` also tags its input `HumanMessage` with that same id so durable-context capture can identify the current request boundary. Gateway resume paths may not append a new `HumanMessage`, so the worker also exposes the pre-run checkpoint's message ids in runtime context; durable-context capture uses that as the current-run boundary and never re-tags older task calls as the resumed run. When no delegation slots remain, task calls are stripped, provider raw tool-call metadata is synced, `finish_reason` is forced to `stop`, and a visible "subagent delegation limit" note is appended so the agent can synthesize already-collected results. Default subagent timeout `subagents.timeout_seconds=1800` (30 min) and built-in `general-purpose` `max_turns=150` (raised from 100/15-min so deep-research subtasks stop hitting `GraphRecursionError` out of the box)
**Concurrency and total delegation cap**: `MAX_CONCURRENT_SUBAGENTS = 3` is enforced by `SubagentLimitMiddleware` (truncates excess tool calls in `after_model`; runtime `max_concurrent_subagents` is clamped to 1-4). The same middleware also enforces `subagents.max_total_per_run` (default 6, config schema 1-50, runtime override `max_total_subagents` clamped to the same range) against current-run entries in the durable delegation ledger, so a long lead-agent run cannot bypass concurrency limits by launching repeated legal-sized batches at each planning checkpoint, but historical delegations from previous runs in the same thread do not consume the new run's budget. The lead-agent prompt uses the same clamped values, so model-visible limits match enforcement. Gateway `run_agent()` and embedded `DeerFlowClient.stream()` both provide a per-invocation `run_id` in runtime context; `DeerFlowClient.stream()` also tags its input `HumanMessage` with that same id so durable-context capture can identify the current request boundary. Gateway resume paths may not append a new `HumanMessage`, so the worker also exposes the pre-run checkpoint's message ids in runtime context; durable-context capture uses that as the current-run boundary and never re-tags older task calls as the resumed run. When no delegation slots remain, task calls are stripped, provider raw tool-call metadata is synced, `finish_reason` is forced to `stop`, and a visible "subagent delegation limit" note is appended so the agent can synthesize already-collected results. Default subagent timeout `subagents.timeout_seconds=1800` (30 min) and built-in `general-purpose` `max_turns=150` (raised from 100/15-min so deep-research subtasks stop hitting `GraphRecursionError` out of the box)
**Flow**: `task()` tool → `SubagentExecutor` → background thread → poll 5s → SSE events → result. `task_started` carries the resolved effective model name. The per-subagent `SubagentTokenCollector` publishes a cumulative usage snapshot to the shared `SubagentResult` after every completed LLM response; the next `task_running` event carries that snapshot, so collapsed workspace cards can update without re-accounting parent-run totals. Terminal ToolMessage metadata (`subagent_model_name`, `subagent_token_usage`) and the persisted `subagent.end` event retain the model/usage after reload; absent provider usage stays absent rather than being estimated as zero.
**Events**: `task_started`, `task_running`, `task_completed`/`task_failed`/`task_timed_out`
**Handled LLM failures**: `LLMErrorHandlingMiddleware` deliberately converts provider/model exceptions into an `AIMessage` so the graph can end cleanly, stamping `additional_kwargs.deerflow_error_fallback=true` plus error metadata. Clean graph termination does not imply subagent success: `SubagentExecutor` inspects the last assistant message at terminalization and maps a marked fallback to `SubagentStatus.FAILED`, which then emits `task_failed` and the existing structured `subagent_error`. Only the marker is authoritative — error-looking assistant prose without it remains a normal completed result, so neither the executor nor frontend parses display text as a status protocol.
@ -652,7 +670,7 @@ Bridges external messaging platforms (Feishu, Slack, Telegram, Discord, DingTalk
4. For chat: look up/create thread through Gateway's LangGraph-compatible API
5. Feishu/Telegram chat: `runs.stream()` → accumulate AI text → publish multiple outbound updates (`is_final=False`) → publish final outbound (`is_final=True`)
6. Slack/Discord chat: `runs.wait()` → extract final response → publish outbound
6b. GitHub chat (`ChannelRunPolicy.fire_and_forget=True`): `runs.create()` returns once the run is `pending`; the manager does not wait for the final state and does not publish an outbound. The agent posts its own reply mid-run via `gh` from the sandbox. `ConflictError` on a busy thread still trips the standard `THREAD_BUSY_MESSAGE` path (log-only on GitHub).
6b. GitHub chat (`ChannelRunPolicy.fire_and_forget=True`): `runs.create()` returns once the run is `pending`; the manager does not wait for the final state and does not publish an outbound. The agent posts its own reply mid-run via `gh` from the sandbox. `ConflictError` on a busy thread still trips the standard `THREAD_BUSY_MESSAGE` path (log-only on GitHub); when the channel's policy also sets `buffer_followups_on_busy=True` (GitHub's default — see "Follow-up buffering while busy" below), the triggering message is additionally captured into a per-thread buffer instead of only logged, so a concurrent comment is not silently dropped.
7. Feishu channel sends one running reply card up front, then patches the same card for each outbound update (card JSON sets `config.update_multi=true` for Feishu's patch API requirement). Messages already sent inside an existing Feishu topic carry a compact source-message preview in that card, and queued same-thread follow-ups patch their own source message's card from queued → running → final without falling back to the generic busy reply.
8. Telegram streaming: the "Working on it..." placeholder message is registered as the stream target; non-final updates `editMessageText` it in place (channel-side throttle: 1s in private chats, 3s in groups due to Telegram's 20 msg/min group cap; 4096-char truncation; rate-limited updates dropped); the final update performs the last edit and splits >4096 texts into follow-up messages
9. DingTalk AI Card mode (when `card_template_id` configured): `runs.stream()` → create card with initial text → stream updates via `PUT /v1.0/card/streaming` → finalize on `is_final=True`. Falls back to `sampleMarkdown` if card creation or streaming fails
@ -689,6 +707,7 @@ The cached value is reused for both the blocking (`runs.wait`) and streaming (`_
**GitHub event-driven agents** (webhook-driven IM channel):
- Custom agents declare a `github:` block in their `config.yaml` to bind to repos and event triggers; the webhook route is fail-closed by default (mounted only when `GITHUB_WEBHOOK_SECRET` is set) and exempt from auth/CSRF because authenticity is enforced by HMAC.
- Outbound is **log-only** by design: each agent posts its own reply mid-run via the `gh` CLI from its sandbox, so the manager uses `fire_and_forget=True` and `runs.create()` returns once pending.
- **Follow-up buffering while busy** (issue #4121): because outbound is log-only, the pre-existing `THREAD_BUSY_MESSAGE` reply on a `ConflictError` was invisible to the commenter — a comment posted while a run was already active looked like it had been silently ignored. When `ChannelRunPolicy.buffer_followups_on_busy=True` (GitHub's default), a `ConflictError` on `runs.create()` now also appends the triggering message to a per-thread, in-memory buffer (`ChannelManager._followup_buffers`) — deduped by GitHub delivery id, capped at `FOLLOWUP_BUFFER_MAX_PER_THREAD` (20, oldest dropped with a WARNING log on overflow). The first successful `runs.create()` on a thread now captures its `run_id` and spawns a background watcher that subscribes to that run's `StreamBridge` stream; once it observes `END_SENTINEL`, the watcher drains up to `FOLLOWUP_DRAIN_BATCH_SIZE` (10) buffered entries into one `<followups-while-busy>`-wrapped input and fires a follow-up `runs.create()` — itself watched the same way, so a backlog deeper than one batch chains into further drain cycles instead of growing one unbounded input. If that follow-up `runs.create()` itself hits `ConflictError` (e.g. a manual Web UI turn or a scheduled run raced onto the same thread), the batch is requeued rather than lost, and is retried whenever this manager next successfully creates and watches a run on that thread. Reactions/acknowledgment (e.g. GitHub's `eyes`/`confused` reaction API) on buffered comments are intentionally **out of scope** for this mechanism and left to a follow-up — comments are coalesced silently. **Plumbing**: the watcher needs the Gateway's `StreamBridge` singleton, which `ChannelManager` did not previously have access to; it is threaded from `app.py`'s lifespan (where `app.state.stream_bridge` is already set by `langgraph_runtime`) through `start_channel_service(get_stream_bridge=...)` → `ChannelService.__init__` → `ChannelManager.__init__`, as a zero-arg closure mirroring the existing `launch_run=lambda **kwargs: launch_scheduled_thread_run(app=app, **kwargs)` pattern used for `ScheduledTaskService` in the same lifespan function. A `ChannelManager` constructed without it (e.g. directly in a test) still buffers safely — it just has no watcher to auto-drain. **Scope limitation**: the buffer and watcher state are per-process, in-memory. Under `GATEWAY_WORKERS>1` or multi-pod, a follow-up comment routed to a different worker process than the one running the busy thread's agent will not see that buffer. This is a known, deliberately deferred limitation with the same shape as the cross-pod gap described for issue #4120 (a shared buffer store or IM-leader election would be needed to close it) — single-process/single-pod deployments, the safe default, see no correctness issue from this, only the documented per-process scope.
- See [backend/docs/GITHUB_AGENTS.md](docs/GITHUB_AGENTS.md) for the architecture diagrams: webhook → fan-out → `InboundMessage` dispatch, `preferred_thread_id = UUID5(repo, number, agent_name)` thread determinism, mention-handle precedence chain, GH token lifecycle via `GH_TOKEN`/`GITHUB_TOKEN` per-call `extra_env`, and the narrow `ConflictError` (HTTP 409) thread-create race recovery.
@ -809,8 +828,10 @@ This invokes `alembic revision --autogenerate` against the live ORM models. Revi
- `migrations/_helpers.py` — `safe_add_column` / `safe_drop_column`
- `migrations/versions/0001_baseline.py` — chain root, matches the schema `create_all` produces from `Base.metadata`
- `migrations/versions/0002_runs_token_usage.py` — fixes issue #3682
- `migrations/versions/0004_run_ownership.py` — `runs` multi-worker ownership + the `uq_runs_thread_active` partial unique index, with a `_dedupe_active_runs_per_thread()` pre-step so `CREATE UNIQUE INDEX` cannot fail on a field DB that already has duplicate active rows per thread
- `migrations/versions/0007_scheduled_run_active_index.py` — the `uq_scheduled_task_run_active` partial unique index (at most one queued/running `scheduled_task_runs` row per `task_id`), with a `_dedupe_active_scheduled_runs_per_task()` pre-step (keeps the newest active row per task, supersedes the rest to `interrupted` with an explanatory `error` + `finished_at`) mirroring 0004; chains after `0006_agents`
- `persistence/bootstrap.py` — `bootstrap_schema(engine, backend=...)`, the three-branch decision + locking
- Tests: `tests/test_persistence_bootstrap.py` (branches), `tests/test_persistence_bootstrap_concurrency.py` (concurrency), `tests/test_persistence_bootstrap_regression.py` (issue #3682), `tests/test_persistence_migrations_env.py` (filter), `tests/blocking_io/test_persistence_bootstrap.py` (asyncio.to_thread anchor)
- Tests: `tests/test_persistence_bootstrap.py` (branches), `tests/test_persistence_bootstrap_concurrency.py` (concurrency), `tests/test_persistence_bootstrap_regression.py` (issue #3682), `tests/test_persistence_migrations_env.py` (filter), `tests/blocking_io/test_persistence_bootstrap.py` (asyncio.to_thread anchor), `tests/test_migration_0004_run_ownership_dedupe.py` + `tests/test_migration_0007_scheduled_run_active_dedupe.py` (dedupe-before-unique-index pre-steps)
### Checkpoint Channel Modes (`full` / `delta`)
@ -975,8 +996,8 @@ Gateway API endpoints and `DeerFlowClient` methods can modify MCP servers and sk
**Agent Conversation**:
- `chat(message, thread_id)` — synchronous, accumulates streaming deltas per message-id and returns the final AI text
- `stream(message, thread_id)` — subscribes to LangGraph `stream_mode=["values", "messages", "custom"]` and yields `StreamEvent`:
- `"values"` — full state snapshot (title, messages, artifacts); AI text already delivered via `messages` mode is **not** re-synthesized here to avoid duplicate deliveries
- `"messages-tuple"` — per-chunk update: for AI text this is a **delta** (concat per `id` to rebuild the full message); tool calls and tool results are emitted once each
- `"values"` — full state snapshot (title, messages, artifacts); AI text already delivered via `messages` mode is **not** re-synthesized here to avoid duplicate deliveries; serialized `ToolMessage` entries preserve a non-`None` native `artifact`
- `"messages-tuple"` — per-chunk update: for AI text this is a **delta** (concat per `id` to rebuild the full message); tool calls and tool results are emitted once each, and tool results preserve a non-`None` native `artifact`
- `"custom"` — forwarded from `StreamWriter`; DeerFlow-built-in custom events are dual-emitted through `deerflow.utils.custom_events`, so `astream_events(version="v2")` consumers also receive one `on_custom_event` with `name=payload["type"]` and the unchanged payload as `data`
- `"end"` — stream finished (carries cumulative `usage` counted once per message id)
- **Custom-event invariant** — production DeerFlow emitters must use `emit_custom_event` / `aemit_custom_event`, not call `StreamWriter` alone. Every built-in payload must carry a non-empty string `type`; typeless payloads remain writer-only and are intentionally absent from `astream_events`. The writer runs first and remains authoritative for Gateway, Web UI, and embedded-client compatibility; callback dispatch is best-effort and must not break that path. Async graph hooks must await the async helper rather than invoking synchronous dispatch on a running event loop.

415
backend/app/channels/manager.py
View File

@ -10,6 +10,7 @@ import time
from collections import OrderedDict
from collections.abc import Awaitable, Callable, Mapping
from dataclasses import dataclass
from html import escape
from pathlib import Path
from typing import Any
from urllib.parse import quote
@ -37,6 +38,7 @@ from app.gateway.github import run_policy as _github_run_policy # noqa: F401
from app.gateway.internal_auth import create_internal_auth_headers
from deerflow.config.agents_config import load_agent_config
from deerflow.config.paths import make_safe_user_id
from deerflow.runtime import END_SENTINEL, StreamBridge
from deerflow.runtime.goal import parse_goal_command
from deerflow.runtime.user_context import get_effective_user_id
from deerflow.skills.slash import parse_slash_skill_reference
@ -102,6 +104,15 @@ BOUND_IDENTITY_UNAVAILABLE_MESSAGE = "Channel connection verification is tempora
# mechanism, same TTL), not a channel-specific gap. True idempotency against
# a late/manual redelivery would require persisting the dedupe key in
# ``ChannelStore`` instead, which is not implemented here.
# Follow-up buffering for busy fire_and_forget threads (issue #4121 Slice 2).
# A ConflictError on a channel opted into ChannelRunPolicy.buffer_followups_on_busy
# buffers the triggering message per-thread instead of only logging it; a
# background watcher drains the buffer into a coalesced follow-up run once the
# busy run's StreamBridge stream reaches END_SENTINEL. See _buffer_followup,
# _drain_followups_for_thread, and _watch_run_and_drain_followups below.
FOLLOWUP_BUFFER_MAX_PER_THREAD = 20
FOLLOWUP_DRAIN_BATCH_SIZE = 10
FOLLOWUP_BLOCK_TAG = "followups-while-busy"
INBOUND_DEDUPE_TTL_SECONDS = 10 * 60
INBOUND_DEDUPE_MAX_ENTRIES = 4096
# Only server-stable provider message ids: client-generated ids (client_msg_id,
@ -225,6 +236,23 @@ class _SerializedThreadRunState:
waiters: int = 0
@dataclass(slots=True)
class _FollowupEntry:
"""One inbound message's text, buffered because its thread was busy.
Routing/policy identity (channel_name, metadata, owner headers) for the
eventual drained run comes from a separate ``carrier_msg`` — see
``ChannelManager._drain_followups_for_thread`` — not from a per-entry
message, since every buffered entry for one thread_id shares that
identity already (thread_id is itself derived deterministically from
(repo, number, agent_name) for GitHub). Only the text needs to survive
per entry.
"""
dedupe_key: str
text: str
def _is_thread_busy_error(exc: BaseException | None) -> bool:
if exc is None:
return False
@ -233,6 +261,59 @@ def _is_thread_busy_error(exc: BaseException | None) -> bool:
return "already running a task" in str(exc)
def _followup_dedupe_key(msg: InboundMessage) -> str:
"""Best-effort stable identifier for a buffered follow-up comment.
Mirrors ``_inbound_dedupe_key``'s provider-id preference order (a GitHub
webhook delivery id first, then the generic provider-message-id metadata
keys), but scoped to one thread's follow-up buffer rather than the
global cross-channel inbound dedupe map, and always returns a usable key
— falling back to an object-identity key — since the follow-up buffer
must still accept an entry even when a provider omits every known id
field (unlike ``_inbound_dedupe_key``, which returns ``None`` to skip
dedupe entirely in that case).
"""
metadata = msg.metadata or {}
gh = metadata.get("github")
if isinstance(gh, dict):
delivery_id = gh.get("delivery_id")
if delivery_id:
return f"github:delivery:{delivery_id}"
for key in INBOUND_DEDUPE_METADATA_KEYS:
value = metadata.get(key)
if value:
return f"{key}:{value}"
raw_message = metadata.get("raw_message")
if isinstance(raw_message, Mapping):
for key in INBOUND_DEDUPE_METADATA_KEYS:
value = raw_message.get(key)
if value:
return f"{key}:{value}"
# No stable provider id available: fall back to a per-message key so the
# entry is still buffered (just never deduped against a redelivery).
return f"__no_id__:{id(msg)}:{msg.created_at}"
def _format_followup_block(entries: list[_FollowupEntry]) -> str:
"""Coalesce buffered follow-up entries into one templated input block."""
lines = [
f"<{FOLLOWUP_BLOCK_TAG}>",
"The following messages arrived on this thread while a previous run was still in progress. They were queued and are now delivered together as one turn:",
"",
]
for idx, entry in enumerate(entries, start=1):
escaped_text = escape(entry.text, quote=False).replace(
"\n",
"\n ",
)
lines.append(f"{idx}. {escaped_text}")
lines.append(f"</{FOLLOWUP_BLOCK_TAG}>")
return "\n".join(lines)
def _as_dict(value: Any) -> dict[str, Any]:
return dict(value) if isinstance(value, Mapping) else {}
@ -829,6 +910,7 @@ class ChannelManager:
channel_sessions: dict[str, Any] | None = None,
connection_repo: Any | None = None,
require_bound_identity: bool = False,
get_stream_bridge: Callable[[], StreamBridge | None] | None = None,
) -> None:
self.bus = bus
self.store = store
@ -840,6 +922,14 @@ class ChannelManager:
self._channel_sessions = dict(channel_sessions or {})
self._connection_repo = connection_repo
self._require_bound_identity = require_bound_identity
# Zero-arg accessor for the FastAPI app's StreamBridge singleton,
# threaded in from app.py's lifespan via start_channel_service() ->
# ChannelService.__init__ (mirrors how ScheduledTaskService gets a
# launch_run closure over `app` in the same lifespan function). None
# when not wired (e.g. a ChannelManager constructed directly in
# tests) — follow-up buffering still works, but no watcher is
# spawned to auto-drain it (see _maybe_spawn_followup_watcher).
self._get_stream_bridge = get_stream_bridge
self._client = None # lazy init — langgraph_sdk async client
self._channel_metadata_synced: set[str] = set()
# Per-conversation locks so concurrent inbound messages for the same
@ -852,11 +942,30 @@ class ChannelManager:
self._csrf_token = generate_csrf_token()
self._semaphore: asyncio.Semaphore | None = None
self._running = False
# Distinct from self._running: that flag is also False before the
# very first start() (so tests that call internal drain/handler
# methods directly without going through start()/stop() keep
# working unchanged). self._stopped tracks specifically whether
# stop() has run, for the follow-up drain guard below.
self._stopped = False
self._task: asyncio.Task | None = None
# Insertion order == chronological (keys are never re-inserted), so an
# OrderedDict lets us evict expired/overflow entries from the front in
# O(k) instead of scanning all entries on every inbound message.
self._recent_inbound_events: OrderedDict[tuple[str, str, str, str], float] = OrderedDict()
# Per-thread follow-up buffers for busy fire_and_forget channels that
# opted into ChannelRunPolicy.buffer_followups_on_busy (issue #4121
# Slice 2). Keyed by thread_id -> OrderedDict[dedupe_key -> entry],
# oldest-first, mirroring _recent_inbound_events's shape but scoped
# per-thread with a hard cap instead of a global TTL (see
# _buffer_followup / _enforce_followup_cap).
self._followup_buffers: dict[str, OrderedDict[str, _FollowupEntry]] = {}
# Background watcher tasks spawned by _maybe_spawn_followup_watcher,
# tracked so stop() can cancel+await them instead of leaving them as
# orphaned fire-and-forget tasks that could still fire a follow-up
# run after this manager has been shut down. Discarded via the same
# task's done-callback (see _maybe_spawn_followup_watcher).
self._followup_watcher_tasks: set[asyncio.Task] = set()
@staticmethod
def _channel_supports_streaming(channel_name: str) -> bool:
@ -911,6 +1020,271 @@ class ChannelManager:
if state.waiters == 0 and not state.lock.locked():
self._serialized_thread_runs.pop((channel_name, thread_id), None)
# -- follow-up buffering for busy fire_and_forget threads (issue #4121) --
def _resolve_stream_bridge(self) -> StreamBridge | None:
"""Resolve the current StreamBridge via the injected accessor, if any."""
if self._get_stream_bridge is None:
return None
try:
return self._get_stream_bridge()
except Exception:
logger.exception("[Manager] get_stream_bridge callable raised; follow-up watch disabled for this run")
return None
def _enforce_followup_cap(self, thread_id: str, buffer: OrderedDict[str, _FollowupEntry]) -> None:
"""Drop the OLDEST buffered entries once *buffer* exceeds the per-thread cap.
Dropping the oldest (rather than the newest, incoming) entry means a
thread that is deep enough in the backlog to hit the cap still keeps
the most recent activity — a better signal for the eventual coalesced
turn than the stalest queued comment. No reaction/acknowledgment is
sent on drop (out of scope for this slice); a WARNING is logged so
operators can see it in gateway.log.
"""
while len(buffer) > FOLLOWUP_BUFFER_MAX_PER_THREAD:
dropped_key, _ = buffer.popitem(last=False)
logger.warning(
"[Manager] follow-up buffer overflow for thread_id=%s (cap=%d); dropped oldest buffered comment (dedupe_key=%s)",
thread_id,
FOLLOWUP_BUFFER_MAX_PER_THREAD,
dropped_key,
)
def _buffer_followup(self, thread_id: str, msg: InboundMessage) -> None:
"""Append *msg* to thread_id's follow-up buffer (ConflictError path).
Dedupe mirrors ``_is_duplicate_inbound``'s OrderedDict idiom, scoped
per-thread instead of global: a redelivered webhook for a comment
already buffered (same dedupe key) is a no-op instead of a second
entry.
"""
key = _followup_dedupe_key(msg)
buffer = self._followup_buffers.setdefault(thread_id, OrderedDict())
if key in buffer:
logger.info(
"[Manager] duplicate follow-up ignored for thread_id=%s (dedupe_key=%s)",
thread_id,
key,
)
return
buffer[key] = _FollowupEntry(dedupe_key=key, text=msg.text)
self._enforce_followup_cap(thread_id, buffer)
logger.info(
"[Manager] buffered follow-up for busy thread_id=%s (dedupe_key=%s, buffered=%d)",
thread_id,
key,
len(buffer),
)
def _pop_followup_batch(self, thread_id: str, *, limit: int) -> list[_FollowupEntry]:
"""Pop up to *limit* buffered entries FIFO (oldest first)."""
buffer = self._followup_buffers.get(thread_id)
if not buffer:
return []
batch: list[_FollowupEntry] = []
for _ in range(min(limit, len(buffer))):
_, entry = buffer.popitem(last=False)
batch.append(entry)
if not buffer:
self._followup_buffers.pop(thread_id, None)
return batch
def _requeue_followups(self, thread_id: str, entries: list[_FollowupEntry]) -> None:
"""Put a popped batch back at the front of the buffer (oldest-first).
Used when the drain's own ``runs.create`` call itself fails —
including the ``ConflictError`` edge case where something this
manager did not create (a manual Web UI turn, a scheduled run) is
occupying the thread. The entries are not lost: the next time *any*
run this manager creates on this thread completes, its watcher will
attempt another drain and find them still buffered.
"""
if not entries:
return
existing = self._followup_buffers.get(thread_id, OrderedDict())
merged: OrderedDict[str, _FollowupEntry] = OrderedDict()
for entry in entries:
merged[entry.dedupe_key] = entry
for key, entry in existing.items():
merged.setdefault(key, entry)
self._enforce_followup_cap(thread_id, merged)
self._followup_buffers[thread_id] = merged
def _maybe_spawn_followup_watcher(
self,
thread_id: str,
run_result: Any,
carrier_msg: InboundMessage,
) -> None:
"""Spawn a background watcher for a just-created run, if wired up.
No-ops (spawns nothing) when no ``get_stream_bridge`` accessor was
threaded in — e.g. a ``ChannelManager`` constructed directly without
going through ``start_channel_service()`` — so tests and any
not-yet-wired deployment never see a dangling background task for
this. When wired, mirrors the existing ``_dispatch_loop`` pattern:
``asyncio.create_task`` + ``add_done_callback(self._log_task_error)``
so an unexpected watcher failure is surfaced in the logs instead of
silently vanishing. The task is also tracked in
``self._followup_watcher_tasks`` (discarded via its own done-callback)
so ``stop()`` can cancel+await any watcher still in flight instead of
leaving it to fire a follow-up run after shutdown.
"""
if self._get_stream_bridge is None:
return
run_id = run_result.get("run_id") if isinstance(run_result, dict) else None
if not run_id:
logger.warning(
"[Manager] runs.create returned no run_id for thread_id=%s; cannot watch for follow-up drain",
thread_id,
)
return
task = asyncio.create_task(self._watch_run_and_drain_followups(thread_id, run_id, carrier_msg))
self._followup_watcher_tasks.add(task)
task.add_done_callback(self._followup_watcher_tasks.discard)
task.add_done_callback(self._log_task_error)
async def _watch_run_and_drain_followups(
self,
thread_id: str,
run_id: str,
carrier_msg: InboundMessage,
) -> None:
"""Watch *run_id* until it ends, then attempt to drain thread_id's buffer.
Subscribes to the StreamBridge the same way existing consumers do
(``entry is END_SENTINEL``, see ``app/gateway/services.py``). Runs
for as long as the underlying run does — GitHub coding runs
routinely take several minutes, so this deliberately does not apply
an artificial timeout, mirroring why the dispatch path itself uses
``runs.create`` instead of ``runs.wait`` in the first place. Draining
is a no-op when the buffer is empty, which is the common case (most
runs never hit a busy-thread conflict).
"""
stream_bridge = self._resolve_stream_bridge()
if stream_bridge is None:
logger.warning(
"[Manager] no stream bridge available; cannot watch run_id=%s for thread_id=%s follow-up drain (any buffered follow-ups will be drained by a later watched run on this thread)",
run_id,
thread_id,
)
return
try:
async for entry in stream_bridge.subscribe(run_id):
if entry is END_SENTINEL:
break
except Exception:
logger.exception(
"[Manager] error watching run_id=%s for thread_id=%s follow-up drain",
run_id,
thread_id,
)
return
client = self._get_client()
await self._drain_followups_for_thread(client, thread_id, carrier_msg)
async def _drain_followups_for_thread(
self,
client,
thread_id: str,
carrier_msg: InboundMessage,
) -> None:
"""Coalesce up to one batch of buffered follow-ups into a fresh run.
``carrier_msg`` supplies routing/policy identity (channel_name,
metadata, owner headers) for the drained run — it is safe to reuse
across an entire drain chain because every buffered entry for one
thread_id shares that identity (thread_id itself is derived
deterministically from (repo, number, agent_name) for GitHub).
A batch larger than ``FOLLOWUP_DRAIN_BATCH_SIZE`` is intentionally
NOT drained in one shot: only the oldest batch is popped here, and
the run created for it is itself watched (via
``_maybe_spawn_followup_watcher``), so a deeper backlog chains into
another drain cycle once this run ends, rather than growing one
unbounded coalesced input block.
If anything from here through ``runs.create`` fails — resolving run
params, applying channel policy, or ``runs.create`` itself
(including ``ConflictError`` from something this manager did not
create racing onto the same thread) — the popped batch is requeued
(not lost) and this coroutine returns without raising or looping:
the next run this manager successfully creates and watches on this
thread will attempt the drain again.
No-ops if the manager has already been ``stop()``-ped: a watcher
task that slips past its own cancellation and reaches this point
after shutdown must not fire a brand new run into a stopped manager.
(Deliberately keyed on ``self._stopped``, not ``self._running`` —
the latter is also ``False`` before the very first ``start()``,
which would otherwise make this guard fire for callers that invoke
the drain directly without going through the dispatch lifecycle.)
"""
if self._stopped:
logger.info(
"[Manager] skipping follow-up drain for thread_id=%s; manager is stopped",
thread_id,
)
return
entries = self._pop_followup_batch(thread_id, limit=FOLLOWUP_DRAIN_BATCH_SIZE)
if not entries:
return
logger.info(
"[Manager] draining %d buffered follow-up(s) for thread_id=%s",
len(entries),
thread_id,
)
try:
# Everything from here through runs.create is covered by the
# same except below: a pre-create failure (e.g. the target agent
# config was removed mid-run, or channel-policy/credential
# resolution raises) must requeue the popped batch exactly like
# a runs.create failure does — none of these steps get to
# silently drop entries that were already popped off the buffer.
assistant_id, run_config, run_context = self._resolve_run_params(carrier_msg, thread_id)
await self._apply_channel_policy(carrier_msg, run_context)
human_message = _human_input_message(_format_followup_block(entries))
run_kwargs: dict[str, Any] = {
"input": {"messages": [human_message]},
"config": run_config,
"context": run_context,
"multitask_strategy": "reject",
}
if owner_headers := _owner_headers(carrier_msg):
run_kwargs["headers"] = owner_headers
result = await client.runs.create(thread_id, assistant_id, **run_kwargs)
except Exception as exc:
if _is_thread_busy_error(exc):
logger.warning(
"[Manager] follow-up drain hit a busy thread_id=%s (a run this manager did not create is active); re-buffering %d entries",
thread_id,
len(entries),
)
else:
logger.exception(
"[Manager] follow-up drain failed for thread_id=%s; re-buffering %d entries",
thread_id,
len(entries),
)
self._requeue_followups(thread_id, entries)
return
self._maybe_spawn_followup_watcher(thread_id, result, carrier_msg)
async def _publish_progress_update(self, msg: InboundMessage, thread_id: str, text: str) -> None:
await self.bus.publish_outbound(
OutboundMessage(
@ -1113,6 +1487,7 @@ class ChannelManager:
if self._running:
return
self._running = True
self._stopped = False
self._semaphore = asyncio.Semaphore(self._max_concurrency)
self._task = asyncio.create_task(self._dispatch_loop())
logger.info("ChannelManager started (max_concurrency=%d)", self._max_concurrency)
@ -1120,6 +1495,7 @@ class ChannelManager:
async def stop(self) -> None:
"""Stop the dispatch loop."""
self._running = False
self._stopped = True
if self._task:
self._task.cancel()
try:
@ -1127,6 +1503,25 @@ class ChannelManager:
except asyncio.CancelledError:
pass
self._task = None
# Follow-up watchers are long-lived background tasks (they await a
# run's full stream, which can take minutes) started outside the
# dispatch loop, so cancelling self._task above does not touch them.
# Left unmanaged, one still subscribed to a run that ends AFTER this
# point would drain its buffer and fire a brand new runs.create()
# into a manager that has already been stopped.
watcher_tasks = list(self._followup_watcher_tasks)
for task in watcher_tasks:
task.cancel()
for task in watcher_tasks:
try:
await task
except asyncio.CancelledError:
pass
except Exception:
logger.exception("[Manager] follow-up watcher task raised during stop()")
self._followup_watcher_tasks.clear()
logger.info("ChannelManager stopped")
# -- dispatch loop -----------------------------------------------------
@ -1647,17 +2042,27 @@ class ChannelManager:
len(msg.text or ""),
)
try:
await client.runs.create(thread_id, assistant_id, **run_kwargs)
# Capturing the return value is new (issue #4121 Slice 2):
# it carries ``run_id``, which the follow-up watcher below
# needs to subscribe to this run's StreamBridge stream. When
# ``buffer_followups_on_busy`` is off this is otherwise
# behaviorally identical to the previous bare ``await``.
result = await client.runs.create(thread_id, assistant_id, **run_kwargs)
except Exception as exc:
if _is_thread_busy_error(exc):
logger.warning("[Manager] thread busy (concurrent run rejected): thread_id=%s", thread_id)
# Swallowed like the generic handler would not be: release the
# key so the provider's redelivery can retry once the thread
# frees, instead of being dropped for the dedupe TTL.
self._release_inbound_dedupe_key(msg)
if policy.buffer_followups_on_busy:
self._buffer_followup(thread_id, msg)
else:
# Swallowed like the generic handler would not be: release the
# key so the provider's redelivery can retry once the thread
# frees, instead of being dropped for the dedupe TTL.
self._release_inbound_dedupe_key(msg)
await self._send_error(msg, THREAD_BUSY_MESSAGE)
return
raise
if policy.buffer_followups_on_busy:
self._maybe_spawn_followup_watcher(thread_id, result, msg)
return
logger.info("[Manager] invoking runs.wait(thread_id=%s, text_len=%d)", thread_id, len(msg.text or ""))

16
backend/app/channels/run_policy.py
View File

@ -84,6 +84,21 @@ class ChannelRunPolicy:
unrelated DeerFlow threads continue concurrently. Defaults
to False so existing channels keep the runtime's native
multitask behavior unless they opt in explicitly.
buffer_followups_on_busy: When True, a ``ConflictError`` on the
``fire_and_forget`` dispatch path (see
:meth:`ChannelManager._handle_chat_on_thread`) does more than
log + reply with the generic busy message: the triggering
message is appended to a per-thread follow-up buffer, and a
background watcher subscribes to the active run's
``StreamBridge`` stream so it can coalesce the buffer into a
follow-up run as soon as that run ends. This targets
``fire_and_forget`` channels whose ``send`` is otherwise the
only feedback a busy sender gets (e.g. GitHub, where ``send``
is log-only) — without it, a concurrent comment is silently
dropped from the sender's point of view. Defaults to False so
channels that have not opted in keep the exact old
silent-drop-with-log behavior; see
``app.gateway.github.run_policy`` for GitHub's opt-in.
"""
is_interactive: bool = True
@ -92,6 +107,7 @@ class ChannelRunPolicy:
requires_bound_identity: bool = True
fire_and_forget: bool = False
serialize_thread_runs: bool = False
buffer_followups_on_busy: bool = False
# Channel name → policy. Channels absent from this map fall through to

42
backend/app/channels/service.py
View File

@ -5,6 +5,7 @@ from __future__ import annotations
import asyncio
import logging
import os
from collections.abc import Callable
from typing import TYPE_CHECKING, Any
from app.channels.base import Channel
@ -18,6 +19,7 @@ logger = logging.getLogger(__name__)
if TYPE_CHECKING:
from deerflow.config.app_config import AppConfig
from deerflow.config.channel_connections_config import ChannelConnectionsConfig
from deerflow.runtime import StreamBridge
# Channel name → import path for lazy loading
_CHANNEL_REGISTRY: dict[str, str] = {
@ -97,10 +99,12 @@ class ChannelService:
*,
connection_repo: Any | None = None,
require_bound_identity: bool = False,
get_stream_bridge: Callable[[], StreamBridge | None] | None = None,
) -> None:
self.bus = MessageBus()
self.store = ChannelStore()
self._connection_repo = connection_repo
self._get_stream_bridge = get_stream_bridge
config = dict(channels_config or {})
langgraph_url = _resolve_service_url(config, "langgraph_url", _CHANNELS_LANGGRAPH_URL_ENV, DEFAULT_LANGGRAPH_URL)
gateway_url = _resolve_service_url(config, "gateway_url", _CHANNELS_GATEWAY_URL_ENV, DEFAULT_GATEWAY_URL)
@ -115,6 +119,7 @@ class ChannelService:
channel_sessions=channel_sessions,
connection_repo=connection_repo,
require_bound_identity=require_bound_identity,
get_stream_bridge=get_stream_bridge,
)
self._channels: dict[str, Any] = {} # name -> Channel instance
self._config = config
@ -122,8 +127,19 @@ class ChannelService:
self._readiness_locks: dict[str, asyncio.Lock] = {}
@classmethod
def from_app_config(cls, app_config: AppConfig | None = None) -> ChannelService:
"""Create a ChannelService from the application config."""
def from_app_config(
cls,
app_config: AppConfig | None = None,
*,
get_stream_bridge: Callable[[], StreamBridge | None] | None = None,
) -> ChannelService:
"""Create a ChannelService from the application config.
``get_stream_bridge`` is threaded straight through to the
``ChannelManager`` (see its docstring); it is optional so direct
callers (including most tests) that don't need follow-up-buffer
auto-draining can omit it.
"""
if app_config is None:
from deerflow.config.app_config import get_app_config
@ -141,6 +157,7 @@ class ChannelService:
channels_config=channels_config,
connection_repo=_make_connection_repo(connection_config),
require_bound_identity=require_bound_identity,
get_stream_bridge=get_stream_bridge,
)
async def start(self) -> None:
@ -407,14 +424,27 @@ def get_channel_service() -> ChannelService | None:
return _channel_service
async def start_channel_service(app_config: AppConfig | None = None) -> ChannelService:
"""Create and start the global ChannelService from app config."""
async def start_channel_service(
app_config: AppConfig | None = None,
*,
get_stream_bridge: Callable[[], StreamBridge | None] | None = None,
) -> ChannelService:
"""Create and start the global ChannelService from app config.
``get_stream_bridge`` is threaded through to ``ChannelService.from_app_config``
-> ``ChannelManager`` so fire_and_forget channels that opt into
``ChannelRunPolicy.buffer_followups_on_busy`` (currently GitHub) can watch
a run's completion and auto-drain buffered follow-ups. ``app.py``'s
lifespan passes a closure over ``app.state.stream_bridge`` here, the same
pattern it already uses for ``ScheduledTaskService``'s ``launch_run``.
"""
global _channel_service
if _channel_service is not None:
return _channel_service
# from_app_config reads the JSON channel store and runtime config files;
# keep that disk IO off the event loop.
_channel_service = await asyncio.to_thread(ChannelService.from_app_config, app_config)
# keep that disk IO off the event loop. asyncio.to_thread forwards both
# args and kwargs to the target callable.
_channel_service = await asyncio.to_thread(ChannelService.from_app_config, app_config, get_stream_bridge=get_stream_bridge)
await _channel_service.start()
return _channel_service

14
backend/app/gateway/app.py
View File

@ -250,7 +250,19 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
try:
from app.channels.service import start_channel_service
channel_service = await start_channel_service(startup_config)
# Closure over `app` (mirrors ScheduledTaskService's `launch_run`
# below) rather than resolving `app.state.stream_bridge` here
# directly: `stream_bridge` is a STARTUP_ONLY_FIELDS singleton set
# once, above, by `langgraph_runtime(app, startup_config)`, so
# either shape is safe by construction — the closure is just the
# more defensive/consistent-with-precedent form, and it is what
# ChannelManager's follow-up-drain watcher (issue #4121 Slice 2)
# uses to reach the same StreamBridge every other run consumer
# goes through `get_stream_bridge(request)` for.
channel_service = await start_channel_service(
startup_config,
get_stream_bridge=lambda: getattr(app.state, "stream_bridge", None),
)
logger.info("Channel service started: %s", channel_service.get_status())
except Exception:
logger.exception("No IM channels configured or channel service failed to start")

132
backend/app/gateway/deps.py
View File

@ -30,7 +30,7 @@ from langgraph.types import Checkpointer
from deerflow.community.browser_automation.session import browser_multi_worker_error
from deerflow.config.app_config import AppConfig, get_app_config
from deerflow.domain.feedback import FeedbackService
from deerflow.runtime import RunContext, RunManager, StreamBridge
from deerflow.runtime import ORPHAN_RECOVERY_STOP_REASON, STARTUP_ORPHAN_RECOVERY_ERROR, RunContext, RunManager, StreamBridge
from deerflow.runtime.events.store.base import RunEventStore
from deerflow.runtime.runs.store.base import RunStore
@ -163,8 +163,10 @@ async def _publish_recovered_run_stream_end(
recovered_runs: list[RunRecord],
*,
cleanup_delay: float = 60.0,
) -> None:
"""Terminate retained streams for runs recovered as orphaned at startup."""
on_cleanup_scheduled: Callable[[str, asyncio.Task[None]], None] | None = None,
) -> list[tuple[str, asyncio.Task[None]]]:
"""Terminate retained streams for runs recovered as orphaned."""
cleanup_tasks: list[tuple[str, asyncio.Task[None]]] = []
for record in recovered_runs:
stream_exists = getattr(bridge, "stream_exists", None)
if stream_exists is not None:
@ -185,6 +187,10 @@ async def _publish_recovered_run_stream_end(
continue
task = asyncio.create_task(bridge.cleanup(record.run_id, delay=cleanup_delay))
task.add_done_callback(lambda task, run_id=record.run_id: _log_recovered_stream_cleanup_result(task, run_id))
cleanup_tasks.append((record.run_id, task))
if on_cleanup_scheduled is not None:
on_cleanup_scheduled(record.run_id, task)
return cleanup_tasks
def _log_recovered_stream_cleanup_result(task: asyncio.Task[None], run_id: str) -> None:
@ -196,6 +202,45 @@ def _log_recovered_stream_cleanup_result(task: asyncio.Task[None], run_id: str)
logger.warning("Failed to clean up recovered run stream for %s", run_id, exc_info=True)
async def _flush_recovered_stream_cleanups(
bridge: StreamBridge,
cleanup_tasks: dict[asyncio.Task[None], str],
*,
timeout: float = 1.0,
) -> None:
"""Cancel delayed cleanups and delete their streams before bridge shutdown."""
pending = [(task, run_id) for task, run_id in cleanup_tasks.items() if not task.done()]
if not pending:
return
for task, _run_id in pending:
task.cancel()
await asyncio.gather(*(task for task, _run_id in pending), return_exceptions=True)
run_ids = list(dict.fromkeys(run_id for _task, run_id in pending))
try:
results = await asyncio.wait_for(
asyncio.gather(
*(bridge.cleanup(run_id, delay=0) for run_id in run_ids),
return_exceptions=True,
),
timeout=max(0.0, timeout),
)
except TimeoutError:
logger.warning(
"Immediate recovered stream cleanup exceeded %.1fs for run_ids=%s; bridge TTL remains the final safety net",
timeout,
run_ids,
)
else:
for run_id, result in zip(run_ids, results):
if isinstance(result, BaseException):
logger.warning(
"Failed to immediately clean up recovered run stream for %s: %r",
run_id,
result,
)
if TYPE_CHECKING:
from app.gateway.auth.local_provider import LocalAuthProvider
from app.gateway.auth.repositories.sqlite import SQLiteUserRepository
@ -206,12 +251,19 @@ if TYPE_CHECKING:
T = TypeVar("T")
async def _mark_latest_recovered_threads_error(
async def _mark_latest_startup_recovered_threads_error(
run_manager: RunManager,
thread_store: ThreadMetaStore,
recovered_runs: list[RunRecord],
) -> None:
"""Mark thread status as error only when its newest run was recovered."""
"""Project startup recovery before request-serving concurrency begins.
This helper must remain on the pre-``yield`` startup path. ``ThreadMetaStore``
has no ``latest_run_id`` column, so it cannot express an atomic conditional
update keyed by the recovered run. Periodic recovery deliberately skips this
projection; moving this helper after request serving starts would reintroduce
a read/update race with newer runs.
"""
recovered_by_thread: dict[str, set[str]] = {}
for record in recovered_runs:
recovered_by_thread.setdefault(record.thread_id, set()).add(record.run_id)
@ -230,6 +282,22 @@ async def _mark_latest_recovered_threads_error(
logger.warning("Failed to mark thread %s as error during run reconciliation", thread_id, exc_info=True)
async def _terminalize_recovered_runs(
bridge: StreamBridge,
recovered_runs: list[RunRecord],
*,
cleanup_delay: float,
on_cleanup_scheduled: Callable[[str, asyncio.Task[None]], None] | None = None,
) -> list[tuple[str, asyncio.Task[None]]]:
"""Publish terminal markers and schedule retained-stream cleanup."""
return await _publish_recovered_run_stream_end(
bridge,
recovered_runs,
cleanup_delay=cleanup_delay,
on_cleanup_scheduled=on_cleanup_scheduled,
)
def get_config() -> AppConfig:
"""Return the freshest ``AppConfig`` for the current request.
@ -367,9 +435,29 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen
# RunManager with store backing for persistence
run_ownership_config = getattr(config, "run_ownership", None)
sb_config = getattr(config, "stream_bridge", None)
cleanup_delay = getattr(sb_config, "recovered_stream_cleanup_delay_seconds", 60.0) if sb_config else 60.0
recovered_stream_cleanup_tasks: dict[asyncio.Task[None], str] = {}
def track_recovered_stream_cleanup(
run_id: str,
task: asyncio.Task[None],
) -> None:
recovered_stream_cleanup_tasks[task] = run_id
task.add_done_callback(lambda completed: recovered_stream_cleanup_tasks.pop(completed, None))
async def terminalize_recovered_runs(recovered_runs: list[RunRecord]) -> None:
await _terminalize_recovered_runs(
app.state.stream_bridge,
recovered_runs,
cleanup_delay=cleanup_delay,
on_cleanup_scheduled=track_recovered_stream_cleanup,
)
app.state.run_manager = RunManager(
store=app.state.run_store,
run_ownership_config=run_ownership_config,
on_orphans_recovered=terminalize_recovered_runs,
)
# Startup recovery: mark inflight runs whose lease has expired as error.
# In single-worker mode (SQLite / backend=memory), no run has a lease, so
@ -379,13 +467,21 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen
from deerflow.utils.time import now_iso
recovered_runs = await app.state.run_manager.reconcile_orphaned_inflight_runs(
error="Gateway restarted before this run reached a durable final state.",
error=STARTUP_ORPHAN_RECOVERY_ERROR,
before=now_iso(),
stop_reason=ORPHAN_RECOVERY_STOP_REASON,
)
await _terminalize_recovered_runs(
app.state.stream_bridge,
recovered_runs,
cleanup_delay=cleanup_delay,
on_cleanup_scheduled=track_recovered_stream_cleanup,
)
await _mark_latest_startup_recovered_threads_error(
app.state.run_manager,
app.state.thread_store,
recovered_runs,
)
sb_config = getattr(config, "stream_bridge", None)
cleanup_delay = getattr(sb_config, "recovered_stream_cleanup_delay_seconds", 60.0) if sb_config else 60.0
await _publish_recovered_run_stream_end(app.state.stream_bridge, recovered_runs, cleanup_delay=cleanup_delay)
await _mark_latest_recovered_threads_error(app.state.run_manager, app.state.thread_store, recovered_runs)
# Start the lease heartbeat if enabled (multi-worker deployments).
await app.state.run_manager.start_heartbeat()
@ -400,7 +496,21 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen
# raises PoolClosed (issue #3373).
run_manager = getattr(app.state, "run_manager", None)
if run_manager is not None:
await _drain_inflight_runs(run_manager)
shutdown_deadline = asyncio.get_running_loop().time() + _RUN_DRAIN_TIMEOUT_SECONDS
try:
await _drain_inflight_runs(run_manager)
finally:
await _flush_recovered_stream_cleanups(
app.state.stream_bridge,
recovered_stream_cleanup_tasks,
timeout=min(
1.0,
max(
0.0,
shutdown_deadline - asyncio.get_running_loop().time(),
),
),
)
await close_engine()

9
backend/app/gateway/github/run_policy.py
View File

@ -129,6 +129,15 @@ def register_policy() -> None:
# still raised synchronously by ``start_run`` before the run is
# accepted, so the busy-thread path is preserved.
fire_and_forget=True,
# GitHub's ``send`` is log-only (agents post via ``gh`` themselves),
# so a busy-thread ``THREAD_BUSY_MESSAGE`` is otherwise invisible to
# the commenter — a concurrent comment is silently dropped from
# their point of view (issue #4121). Buffering + draining follow-ups
# once the busy run ends directly fixes that for the one channel
# where it is a real, routinely-triggered problem; other
# fire_and_forget channels keep the dataclass default (False) until
# they have the same log-only-send shape and opt in explicitly.
buffer_followups_on_busy=True,
)

2
backend/app/gateway/routers/runs.py
View File

@ -16,7 +16,7 @@ from fastapi.responses import StreamingResponse
from app.gateway.authz import require_permission
from app.gateway.deps import get_run_event_store, get_run_manager, get_run_store, get_stream_bridge
from app.gateway.pagination import trim_run_message_page
from app.gateway.routers.thread_runs import RunCreateRequest
from app.gateway.run_models import RunCreateRequest
from app.gateway.services import build_checkpoint_state_accessor, sse_consumer, start_run, wait_for_run_completion
from deerflow.runtime import serialize_channel_values_for_api

16
backend/app/gateway/routers/skills.py
View File

@ -327,10 +327,20 @@ async def get_custom_skill_history(skill_name: str, request: Request, config: Ap
await require_admin_user(request, detail=_ADMIN_REQUIRED_DETAIL)
try:
skill_name = skill_name.replace("\r\n", "").replace("\n", "")
storage = _get_user_skill_storage(config)
if not storage.custom_skill_exists(skill_name) and not storage.get_skill_history_file(skill_name).exists():
def _read_history() -> list[dict] | None:
# Worker thread: storage construction, the existence probes, and the
# history-file read are blocking filesystem IO that must stay off the
# event loop. None signals 404 to the caller.
storage = _get_user_skill_storage(config)
if not storage.custom_skill_exists(skill_name) and not storage.get_skill_history_file(skill_name).exists():
return None
return storage.read_history(skill_name)
history = await asyncio.to_thread(_read_history)
if history is None:
raise HTTPException(status_code=404, detail=f"Custom skill '{skill_name}' not found")
return CustomSkillHistoryResponse(history=storage.read_history(skill_name))
return CustomSkillHistoryResponse(history=history)
except HTTPException:
raise
except Exception as e:

31
backend/app/gateway/routers/thread_runs.py
View File

@ -34,6 +34,7 @@ from app.gateway.checkpoint_lineage import (
)
from app.gateway.deps import get_current_user, get_feedback_service, get_run_event_store, get_run_manager, get_run_store, get_stream_bridge
from app.gateway.pagination import trim_run_message_page
from app.gateway.run_models import RunCreateRequest
from app.gateway.services import build_checkpoint_state_accessor, build_thread_checkpoint_state_accessor, sse_consumer, start_run, wait_for_run_completion
from app.gateway.utils import sanitize_log_param
from deerflow.runtime import CancelOutcome, RunRecord, RunStatus, serialize_channel_values_for_api
@ -78,29 +79,6 @@ def compute_run_durations(runs) -> dict[str, int]:
# ---------------------------------------------------------------------------
class RunCreateRequest(BaseModel):
assistant_id: str | None = Field(default=None, description="Agent / assistant to use")
input: dict[str, Any] | None = Field(default=None, description="Graph input (e.g. {messages: [...]})")
command: dict[str, Any] | None = Field(default=None, description="LangGraph Command")
metadata: dict[str, Any] | None = Field(default=None, description="Run metadata")
config: dict[str, Any] | None = Field(default=None, description="RunnableConfig overrides")
context: dict[str, Any] | None = Field(default=None, description="DeerFlow context overrides (model_name, thinking_enabled, etc.)")
webhook: str | None = Field(default=None, description="Completion callback URL")
checkpoint_id: str | None = Field(default=None, description="Resume from checkpoint")
checkpoint: dict[str, Any] | None = Field(default=None, description="Full checkpoint object")
interrupt_before: list[str] | Literal["*"] | None = Field(default=None, description="Nodes to interrupt before")
interrupt_after: list[str] | Literal["*"] | None = Field(default=None, description="Nodes to interrupt after")
stream_mode: list[str] | str | None = Field(default=None, description="Stream mode(s)")
stream_subgraphs: bool = Field(default=False, description="Include subgraph events")
stream_resumable: bool | None = Field(default=None, description="SSE resumable mode")
on_disconnect: Literal["cancel", "continue"] = Field(default="cancel", description="Behaviour on SSE disconnect")
on_completion: Literal["delete", "keep"] = Field(default="keep", description="Delete temp thread on completion")
multitask_strategy: Literal["reject", "rollback", "interrupt", "enqueue"] = Field(default="reject", description="Concurrency strategy")
after_seconds: float | None = Field(default=None, description="Delayed execution")
if_not_exists: Literal["reject", "create"] = Field(default="create", description="Thread creation policy")
feedback_keys: list[str] | None = Field(default=None, description="LangSmith feedback keys")
class RegeneratePrepareRequest(BaseModel):
message_id: str = Field(..., min_length=1, description="Assistant message id to regenerate")
@ -299,8 +277,9 @@ def _is_visible_ai_message(message: Any) -> bool:
return _message_type(message) == "ai" and not _is_hidden_or_control_message(message)
def _is_middleware_message_row(row: dict[str, Any]) -> bool:
return str((row.get("metadata") or {}).get("caller", "")).startswith("middleware:")
def _is_thread_history_hidden_message_row(row: dict[str, Any]) -> bool:
caller = str((row.get("metadata") or {}).get("caller", ""))
return caller.startswith("middleware:") or (caller.startswith("subagent:") and _message_type(row.get("content")) == "ai")
def _checkpoint_messages(snapshot: Any) -> list[Any]:
@ -860,7 +839,7 @@ async def _scan_thread_message_page(
raise RuntimeError("Run event message rows are missing sequence values")
for row in reversed(raw):
if _is_middleware_message_row(row) or row.get("run_id") in superseded_run_ids:
if _is_thread_history_hidden_message_row(row) or row.get("run_id") in superseded_run_ids:
continue
visible_desc.append(row)
if len(visible_desc) == limit + 1:

92
backend/app/gateway/run_models.py Normal file
View File

@ -0,0 +1,92 @@
"""Shared request models for the LangGraph-compatible run boundary."""
from __future__ import annotations
from typing import Any, Literal
from pydantic import BaseModel, ConfigDict, Field, ValidationInfo, field_validator
from pydantic_core import PydanticCustomError
from deerflow.runtime.stream_modes import RunStreamMode, UnsupportedStreamModeError, normalize_stream_modes
class RunCreateRequest(BaseModel):
"""Validated run request used by both HTTP and internal launch paths."""
model_config = ConfigDict(extra="forbid")
assistant_id: str | None = Field(default=None, description="Agent / assistant to use")
input: dict[str, Any] | None = Field(default=None, description="Graph input (e.g. {messages: [...]})")
command: dict[str, Any] | None = Field(default=None, description="LangGraph Command")
metadata: dict[str, Any] | None = Field(default=None, description="Run metadata")
config: dict[str, Any] | None = Field(default=None, description="RunnableConfig overrides")
context: dict[str, Any] | None = Field(default=None, description="DeerFlow context overrides (model_name, thinking_enabled, etc.)")
webhook: None = Field(default=None, description="Compatibility placeholder; completion callbacks are not supported")
checkpoint_id: str | None = Field(default=None, description="Resume from checkpoint")
checkpoint: dict[str, Any] | None = Field(default=None, description="Full checkpoint object")
interrupt_before: list[str] | Literal["*"] | None = Field(default=None, description="Nodes to interrupt before")
interrupt_after: list[str] | Literal["*"] | None = Field(default=None, description="Nodes to interrupt after")
stream_mode: list[RunStreamMode] | RunStreamMode | None = Field(default=None, description="Supported stream mode(s)")
stream_subgraphs: bool = Field(default=False, description="Include subgraph events")
stream_resumable: None = Field(default=None, description="Compatibility placeholder; resumable SSE is not supported")
on_disconnect: Literal["cancel", "continue"] = Field(default="cancel", description="Behaviour on SSE disconnect")
on_completion: None = Field(default=None, description="Compatibility placeholder; completion behavior is not supported")
multitask_strategy: Literal["reject", "rollback", "interrupt"] = Field(default="reject", description="Concurrency strategy")
after_seconds: None = Field(default=None, description="Compatibility placeholder; delayed execution is not supported")
if_not_exists: Literal["create"] = Field(default="create", description="Compatibility default; missing threads are created")
feedback_keys: None = Field(default=None, description="Compatibility placeholder; feedback key collection is not supported")
@field_validator(
"webhook",
"stream_resumable",
"on_completion",
"multitask_strategy",
"after_seconds",
"if_not_exists",
"feedback_keys",
mode="before",
)
@classmethod
def reject_unsupported_run_options(cls, value: Any, info: ValidationInfo) -> Any:
if info.field_name in {"multitask_strategy", "if_not_exists"} and not isinstance(value, str):
return value
supported_defaults = {
"webhook": None,
"stream_resumable": None,
"on_completion": None,
"multitask_strategy": {"reject", "rollback", "interrupt"},
"after_seconds": None,
"if_not_exists": "create",
"feedback_keys": None,
}
supported = supported_defaults[info.field_name]
if isinstance(supported, set):
is_supported = isinstance(value, str) and value in supported
else:
is_supported = value == supported
if not is_supported:
raise PydanticCustomError(
"unsupported_run_option",
"Run option '{option}' is not supported by DeerFlow",
{"option": info.field_name},
)
return value
@field_validator("stream_mode", mode="before")
@classmethod
def reject_unsupported_stream_modes(cls, value: Any) -> Any:
if value is None:
return value
if not isinstance(value, str) and (not isinstance(value, list) or not all(isinstance(mode, str) for mode in value)):
return value
try:
normalize_stream_modes(value)
except UnsupportedStreamModeError as exc:
modes = ", ".join(exc.modes)
raise PydanticCustomError(
"unsupported_stream_mode",
"Unsupported stream mode(s): {modes}",
{"modes": modes},
) from exc
return value

54
backend/app/gateway/services.py
View File

@ -28,6 +28,7 @@ from app.gateway.internal_auth import (
get_internal_user,
get_trusted_internal_owner_user_id,
)
from app.gateway.run_models import RunCreateRequest
from app.gateway.utils import sanitize_log_param
from deerflow.agents.middlewares.dynamic_context_middleware import _DYNAMIC_CONTEXT_REMINDER_KEY, _REMINDER_DATE_KEY
from deerflow.agents.middlewares.view_image_middleware import _IMAGE_CONTEXT_MESSAGE_MARKER_KEY
@ -35,6 +36,7 @@ from deerflow.config.app_config import get_app_config
from deerflow.runtime import (
END_SENTINEL,
HEARTBEAT_SENTINEL,
ORPHAN_RECOVERY_STOP_REASON,
CheckpointStateAccessor,
ConflictError,
DisconnectMode,
@ -56,6 +58,7 @@ from deerflow.runtime.checkpoint_state import graph_state_schema
from deerflow.runtime.goal import goal_thread_lock
from deerflow.runtime.runs.naming import resolve_root_run_name
from deerflow.runtime.secret_context import redact_config_secrets
from deerflow.runtime.stream_modes import normalize_stream_modes
from deerflow.runtime.user_context import reset_current_user, set_current_user
from deerflow.utils.messages import ORIGINAL_USER_CONTENT_KEY
@ -120,23 +123,29 @@ async def _terminal_record_stream_missing(bridge: StreamBridge, record: RunRecor
return False
async def _orphan_recovery_observed_after_heartbeat(
record: RunRecord,
run_mgr: RunManager,
) -> bool:
"""Return whether durable orphan recovery is the consumer's liveness edge.
A normal terminal status is not sufficient: the producer persists status
before publishing its final error/data frames and END. Orphan recovery is
different because the producer is known to be gone and the durable
``stop_reason`` is written atomically with the terminal status. Only that
explicit signal may synthesize END after a heartbeat.
"""
if not record.store_only:
return False
refreshed = await run_mgr.get(record.run_id, user_id=record.user_id)
return refreshed is not None and _run_is_terminal(refreshed) and refreshed.stop_reason == ORPHAN_RECOVERY_STOP_REASON
# ---------------------------------------------------------------------------
# Input / config helpers
# ---------------------------------------------------------------------------
def normalize_stream_modes(raw: list[str] | str | None) -> list[str]:
"""Normalize the stream_mode parameter to a list.
Default matches what ``useStream`` expects: values + messages-tuple.
"""
if raw is None:
return ["values"]
if isinstance(raw, str):
return [raw]
return raw if raw else ["values"]
def _strip_external_message_metadata(message: Any) -> Any:
"""Remove server-owned metadata from an untrusted input message."""
if not isinstance(message, BaseMessage):
@ -884,7 +893,7 @@ async def apply_checkpoint_to_run_config(
async def start_run(
body: Any,
body: RunCreateRequest,
thread_id: str,
request: Request,
) -> RunRecord:
@ -893,13 +902,13 @@ async def start_run(
Parameters
----------
body : RunCreateRequest
The validated request body (typed as Any to avoid circular import
with the router module that defines the Pydantic model).
The validated request body shared by HTTP and internal launch paths.
thread_id : str
Target thread.
request : Request
FastAPI request — used to retrieve singletons from ``app.state``.
"""
stream_modes = normalize_stream_modes(body.stream_mode)
bridge = get_stream_bridge(request)
run_mgr = get_run_manager(request)
run_ctx = get_run_context(request)
@ -1019,8 +1028,6 @@ async def start_run(
request_context=getattr(body, "context", None),
)
stream_modes = normalize_stream_modes(body.stream_mode)
task = asyncio.create_task(
run_agent(
bridge,
@ -1070,10 +1077,7 @@ async def launch_scheduled_thread_run(
),
cookies={},
)
# SimpleNamespace stands in for the Pydantic run-request body that the
# HTTP path parses. If start_run gains a new body.* attribute that it reads
# directly, add the matching field here so the scheduler path stays in sync.
body = SimpleNamespace(
body = RunCreateRequest(
assistant_id=assistant_id,
input={"messages": [{"role": "user", "content": prompt}]},
command=None,
@ -1093,10 +1097,10 @@ async def launch_scheduled_thread_run(
stream_subgraphs=False,
stream_resumable=None,
on_disconnect="continue",
on_completion="keep",
on_completion=None,
multitask_strategy="reject",
after_seconds=None,
if_not_exists="reject",
if_not_exists="create",
feedback_keys=None,
)
record = await start_run(body, thread_id, request)
@ -1126,7 +1130,7 @@ async def sse_consumer(
break
if entry is HEARTBEAT_SENTINEL:
if await _terminal_record_stream_missing(bridge, record):
if await _orphan_recovery_observed_after_heartbeat(record, run_mgr):
yield format_sse("end", None)
return
yield ": heartbeat\n\n"
@ -1189,7 +1193,7 @@ async def wait_for_run_completion(
if entry is END_SENTINEL:
completed = True
return True
if entry is HEARTBEAT_SENTINEL and await _terminal_record_stream_missing(bridge, record):
if entry is HEARTBEAT_SENTINEL and await _orphan_recovery_observed_after_heartbeat(record, run_mgr):
completed = True
return True
if await request.is_disconnected():

96
backend/app/scheduler/service.py
View File

@ -9,11 +9,17 @@ from typing import Any, Literal
from fastapi import HTTPException
from deerflow.persistence.scheduled_task_runs import ActiveScheduledRunConflict
from deerflow.runtime import ConflictError, RunRecord
from deerflow.scheduler.schedules import next_run_at
logger = logging.getLogger(__name__)
# Shared so the has_active_runs fast path and the unique-index race path return
# byte-identical outcomes for the same "task already has an active run" condition.
_ACTIVE_RUN_CONFLICT_ERROR = "task already has an active run"
_SKIP_ACTIVE_RUN_ERROR = "skipped: a previous run of this task is still active"
class ScheduledTaskService:
def __init__(
@ -94,28 +100,37 @@ class ScheduledTaskService:
# does not count itself as the active run. A manual trigger against an
# active run is rejected outright (409 at the router) instead of being
# recorded as a skipped occurrence — nothing was scheduled to happen.
skip_error: str | None = None
if task.get("overlap_policy", "skip") == "skip" and await self._task_run_repo.has_active_runs(task["id"]):
#
# This has_active_runs check is a non-atomic fast path: it runs in its
# own session and is separated from the create() below by await points,
# so two concurrent dispatches (double-click / client retry / a manual
# trigger racing the poller) can both observe no active run. The DB is
# the atomic arbiter — the partial unique index uq_scheduled_task_run_active
# rejects the second active insert, surfaced as ActiveScheduledRunConflict
# and collapsed to the SAME outcome as this fast path just below.
overlap_skip = task.get("overlap_policy", "skip") == "skip"
if overlap_skip and await self._task_run_repo.has_active_runs(task["id"]):
if trigger == "manual":
return {
"outcome": "conflict",
"task_run_id": None,
"run_id": None,
"thread_id": execution_thread_id,
"error": "task already has an active run",
}
skip_error = "skipped: a previous run of this task is still active"
return self._active_run_conflict_result(execution_thread_id)
return await self._record_scheduled_skip(task, thread_id=execution_thread_id, now=now, trigger=trigger)
task_run_id = f"task-run-{uuid.uuid4().hex}"
await self._task_run_repo.create(
run_record_id=task_run_id,
task_id=task["id"],
thread_id=execution_thread_id,
scheduled_for=now,
trigger=trigger,
status="queued",
)
if skip_error is not None:
return await self._finalize_skip(task, task_run_id=task_run_id, thread_id=execution_thread_id, now=now, error=skip_error)
try:
await self._task_run_repo.create(
run_record_id=task_run_id,
task_id=task["id"],
thread_id=execution_thread_id,
scheduled_for=now,
trigger=trigger,
status="queued",
)
except ActiveScheduledRunConflict:
# Lost the create race for the task's single active slot: a
# concurrent dispatch passed the same fast-path check and inserted
# its active row first. Identical outcome to the fast path above.
if trigger == "manual":
return self._active_run_conflict_result(execution_thread_id)
return await self._record_scheduled_skip(task, thread_id=execution_thread_id, now=now, trigger=trigger)
try:
result = await self._launch_run(
thread_id=execution_thread_id,
@ -209,6 +224,47 @@ class ScheduledTaskService:
"error": str(exc),
}
def _active_run_conflict_result(self, thread_id: str) -> dict[str, Any]:
"""Manual-trigger response when the task already has an active run.
Nothing was scheduled to happen, so no run-history row is recorded; the
router maps this to a 409.
"""
return {
"outcome": "conflict",
"task_run_id": None,
"run_id": None,
"thread_id": thread_id,
"error": _ACTIVE_RUN_CONFLICT_ERROR,
}
async def _record_scheduled_skip(
self,
task: dict[str, Any],
*,
thread_id: str,
now: datetime,
trigger: str,
) -> dict[str, Any]:
"""Record a skipped occurrence for a scheduled dispatch that overlapped an active run.
The tombstone is created directly as terminal ``"skipped"`` rather than
the transient ``"queued"`` the launch path uses: a queued row is active
and would itself trip ``uq_scheduled_task_run_active`` against the
pre-existing run that is still holding the task's single active slot.
``"skipped"`` is outside the index predicate, so it never conflicts.
"""
task_run_id = f"task-run-{uuid.uuid4().hex}"
await self._task_run_repo.create(
run_record_id=task_run_id,
task_id=task["id"],
thread_id=thread_id,
scheduled_for=now,
trigger=trigger,
status="skipped",
)
return await self._finalize_skip(task, task_run_id=task_run_id, thread_id=thread_id, now=now, error=_SKIP_ACTIVE_RUN_ERROR)
async def _finalize_skip(
self,
task: dict[str, Any],

10
backend/docs/API.md
View File

@ -103,8 +103,14 @@ Content-Type: application/json
```
**Stream Mode Compatibility:**
- Use: `values`, `messages-tuple`, `custom`, `updates`, `events`, `debug`, `tasks`, `checkpoints`
- Do not use: `tools` (deprecated/invalid in current `langgraph-api` and will trigger schema validation errors)
- Use: `values`, `messages-tuple`, `custom`, `updates`, `debug`, `tasks`, `checkpoints`
- Unsupported modes, including `messages`, `events`, and `tools`, return `422` before a run is created. DeerFlow never substitutes `values` for an unsupported mode.
**Run Option Compatibility:**
- Supported concurrency strategies: `reject`, `rollback`, and `interrupt`
- Compatibility default: `if_not_exists="create"`; this matches DeerFlow's current behavior
- Unsupported options return `422`: `webhook`, `stream_resumable`, `after_seconds`, `feedback_keys`, any non-null `on_completion` value (including the SDK values `"complete"` and `"continue"`), `if_not_exists="reject"`, and `multitask_strategy="enqueue"`
- Undeclared SDK options, including `checkpoint_during` and `durability`, also return `422` instead of being silently discarded
**Recursion Limit:**

47
backend/docs/GITHUB_AGENTS.md
View File

@ -11,6 +11,7 @@ This document covers the **architecture** of that pipeline:
- GH token lifecycle (`GITHUB_APP_ID` + `PRIVATE_KEY` → `run_context["github_token"]` → sandbox `GH_TOKEN`/`GITHUB_TOKEN`)
- `ConflictError` (HTTP 409) thread-create race recovery
- Why **outbound is log-only** (agents post via `gh` from their sandbox)
- Follow-up buffering while busy (issue #4121): buffer → watch → drain, and the single-process scope limit
## Overview
@ -218,6 +219,49 @@ The recovery is **narrow**: only `langgraph_sdk.errors.ConflictError` (HTTP 409)
The follow-up `threads.get(preferred_thread_id)` is itself verified before caching — if it also rejects, the store underneath is in an inconsistent state and the failure surfaces.
## Follow-up Buffering While Busy (#4121)
A **different** conflict from the one above: not two deliveries racing to *create* a thread, but a new comment arriving while a *run* is already active on an existing thread. `runs.create()` raises `ConflictError` for that too, and until this fix the manager only logged it and replied with `THREAD_BUSY_MESSAGE` — invisible to the commenter, since `GitHubChannel.send()` is log-only (see below). The comment looked silently ignored.
```mermaid
sequenceDiagram
autonumber
participant C1 as Comment 1
participant C2 as Comment 2 (while busy)
participant Mgr as ChannelManager
participant Client as langgraph_sdk client
participant SB as StreamBridge
C1->>Mgr: InboundMessage
Mgr->>Client: runs.create() [fire_and_forget]
Client-->>Mgr: {run_id: run-1, status: pending}
Mgr->>SB: subscribe(run-1) (background watcher)
C2->>Mgr: InboundMessage (same thread_id)
Mgr->>Client: runs.create()
Client-->>Mgr: 409 ConflictError
Mgr->>Mgr: _buffer_followup(thread_id, msg)<br/>(deduped by delivery_id, capped at 20)
Mgr-->>C2: THREAD_BUSY_MESSAGE (log-only on GitHub)
Note over SB: run-1 completes
SB-->>Mgr: END_SENTINEL
Mgr->>Mgr: _drain_followups_for_thread()<br/>(pop up to 10, oldest first)
Mgr->>Client: runs.create() with <followups-while-busy> input
Client-->>Mgr: {run_id: run-2, status: pending}
Mgr->>SB: subscribe(run-2) (watch again — chains if >10 queued)
```
Key properties:
- **Dedupe**: buffering keys on the GitHub webhook delivery id (falling back to the generic provider-message-id metadata keys, mirroring `_inbound_dedupe_key`), so a redelivered webhook for a comment already buffered is a no-op rather than a duplicate entry.
- **Cap**: 20 entries per thread. Overflow drops the *oldest* buffered entry (not the newest) with a WARNING log — recent activity is a more useful signal than the stalest queued comment once a thread is deep enough in the backlog to hit the cap. No reaction/acknowledgment is sent on drop; see the reactions note below.
- **Batching**: a drain coalesces at most 10 entries into one `<followups-while-busy>` input block. A backlog deeper than 10 is not force-fit into a single turn — the drained run is itself watched, so its own completion triggers another drain cycle for the remainder.
- **Drain-conflict edge case**: if the drain's own `runs.create()` also hits `ConflictError` — e.g. a manual Web UI turn or a scheduled run raced onto the same thread — the popped batch is requeued (not lost, not retried in a tight loop). It is picked up again the next time this manager successfully creates *and watches* a run on that thread, which is guaranteed to eventually happen once whatever is occupying the thread finishes and any subsequent trigger succeeds.
- **Reactions are out of scope for this slice.** GitHub's reaction API (`eyes`/`confused` acknowledgment) has no existing integration in this codebase and needs its own design pass; buffered comments are coalesced silently, with no per-comment acknowledgment.
- **Config**: gated behind `ChannelRunPolicy.buffer_followups_on_busy` (default `False`); GitHub's own registration in `app/gateway/github/run_policy.py` opts in, since it is exactly the fire_and_forget + log-only-send channel this was designed for. Any other channel that adopts `fire_and_forget=True` in the future keeps the old silent-drop-with-log behavior unless it opts in too.
- **Plumbing**: the watcher subscribes to the Gateway's `StreamBridge` singleton (`app.state.stream_bridge`), which `ChannelManager` previously had no way to reach — every other consumer gets it via `get_stream_bridge(request)`, which needs an HTTP `Request` the bus-consumer loop doesn't have. It is threaded as a zero-arg closure from `app.py`'s lifespan (`get_stream_bridge=lambda: getattr(app.state, "stream_bridge", None)`) through `start_channel_service()` → `ChannelService.__init__` → `ChannelManager.__init__`, mirroring the existing `launch_run=lambda **kwargs: launch_scheduled_thread_run(app=app, **kwargs)` closure already used for `ScheduledTaskService` in the same lifespan function.
- **Single-process scope (known limitation)**: the buffer and watcher tasks live in one `ChannelManager` instance's process memory. Under `GATEWAY_WORKERS>1` or a multi-pod deployment, a follow-up comment that a load balancer or webhook fan-out routes to a *different* worker than the one that created the busy run will not see that worker's buffer. This mirrors the cross-pod gap already documented for issue #4120 (IM leader election or a shared buffer store would close it) and is deliberately deferred — single-process/single-pod deployments, the safe default, are unaffected.
## Outbound is Log-only
```mermaid
@ -248,9 +292,10 @@ This is also why the GitHub channel registers `ChannelRunPolicy.fire_and_forget=
## Cross-references
- [AGENTS.md](../AGENTS.md) → "GitHub event-driven agents" — the index view in `backend/AGENTS.md` (binding shape, per-event triggers, mention precedence, token env summary)
- [AGENTS.md](../AGENTS.md) → "GitHub event-driven agents" — the index view in `backend/AGENTS.md` (binding shape, per-event triggers, mention precedence, token env summary, follow-up buffering summary)
- [IM_CHANNEL_CONNECTIONS.md](IM_CHANNEL_CONNECTIONS.md) — interactive IM channels (Telegram/Slack/etc.) for the full `_handle_chat` and owner-scoped file storage flow
- `app/gateway/github/dispatcher.py` — `fanout_event`, `_is_self_event`, mention precedence chain
- `app/channels/manager.py` — `_buffer_followup`, `_drain_followups_for_thread`, `_watch_run_and_drain_followups` (follow-up buffering while busy, issue #4121)
- `app/gateway/github/identity.py` — `resolve_thread_id` (UUID5), `extract_target`
- `app/gateway/github/triggers.py` — `event_should_fire`, `DEFAULT_TRIGGERS`
- `app/gateway/github/run_policy.py` — `inject_github_credentials`, `register_policy`

9
backend/docs/RUN_EVENT_STREAM.md
View File

@ -41,8 +41,11 @@ database store.
`category="message"` means an event is eligible for a message projection; it
does not guarantee that the row is visible in the UI. Thread-history APIs also
filter middleware model calls and superseded regenerate runs, and the frontend
honors message-level visibility markers such as `hide_from_ui`.
filter middleware model calls, subagent AI responses, and superseded regenerate
runs, and the frontend honors message-level visibility markers such as
`hide_from_ui`. Subagent events remain available through the run-events
endpoint; parent `task` ToolMessages remain in thread history so subtask cards
can restore their terminal status after reload.
All other categories are excluded from message projections and are available
through run-event or specialized APIs:
@ -122,7 +125,7 @@ Schema. It is the authoritative field-level reference.
| Consumer | Read path and behavior |
| --- | --- |
| Frontend thread history | `GET /api/threads/{thread_id}/messages/page` scans `list_messages()`, removes middleware rows and superseded regenerate runs, then applies frontend message visibility rules. |
| Frontend thread history | `GET /api/threads/{thread_id}/messages/page` scans `list_messages()`, removes middleware rows, subagent AI responses, and superseded regenerate runs, then applies frontend message visibility rules. |
| Per-run message clients | Thread-scoped and stateless run message endpoints call `list_messages_by_run()`. |
| Run debug/audit | `GET /api/threads/{thread_id}/runs/{run_id}/events` calls `list_events()` and supports `event_types`, `task_id`, `limit`, and `after_seq`. |
| Historical subtask cards | Fetch `subagent.step` through the run-events endpoint, filtered and paginated by `task_id`. |

5
backend/docs/STREAMING.md
View File

@ -143,7 +143,7 @@ sequenceDiagram
关键组件：
- `runtime/runs/worker.py::run_agent` — 在 `asyncio.Task` 里跑 `agent.astream()`，把每个 chunk 通过 `serialize(chunk, mode=mode)` 转成 JSON，再 `bridge.publish()`。
- `runtime/stream_bridge` — 抽象 Queue。`publish/subscribe` 解耦生产者和消费者，支持 `Last-Event-ID` 重连、心跳、多订阅者 fan-out。Redis backend 会在每次 `publish()` / `publish_end()` 刷新 retained stream key TTL；启动恢复将 orphan run 标记为 error 后也会发布 `END_SENTINEL`，避免重连的 SSE 客户端只收到心跳。注意：TTL 是 Redis 内存安全网（防止 key 泄漏），不是 subscriber 终止机制——如果 worker 和 gateway 同时挂掉，已连接的 SSE 客户端在 TTL 过期后仍无法收到 END 信号，需要依赖客户端侧超时。完整的跨 pod subscriber 终止需要 worker 存活检测（liveness），当前版本不包含此功能。
- `runtime/stream_bridge` — 抽象 Queue。`publish/subscribe` 解耦生产者和消费者，支持 `Last-Event-ID` 重连、心跳、多订阅者 fan-out。Redis backend 会在每次 `publish()` / `publish_end()` 刷新 retained stream key TTL；启动恢复与基于 worker lease 的周期恢复共用 Gateway stream terminalization 路径：`RunManager` 先将 orphan run 持久化为 `error` 并写入显式的 `stop_reason=orphan_recovered`，随后 Gateway 发布 `END_SENTINEL` 并安排 stream cleanup。周期扫描、逐行状态写入和 Gateway callback 作为一个受监督的 single-flight 后台 task 执行；慢任务不会堆积，也不会阻塞唯一的 lease heartbeat。shutdown 优先收敛活跃 run，再处理恢复 task；尚未执行的延迟 stream cleanup 会改为立即删除。只有 runtime `yield` 前、无并发请求的启动恢复会把最新受影响 thread 标记为 error；周期恢复不做非原子的 thread 投影。store-only SSE 与 `/wait` consumer 不能把普通 durable terminal status 当成流已完成，否则可能跳过延迟发布的 error 等尾部事件；只有 `orphan_recovered` 信号能在 heartbeat 时触发 END fallback，因为此时 producer 已被确认失联。TTL 仍是 Redis 内存和故障安全网，不是正常的 subscriber 终止机制。
- `app/gateway/services.py::sse_consumer` — 从 bridge 订阅，格式化为 SSE wire 帧。
- `runtime/serialization.py::serialize` — mode-aware 序列化；`messages` mode 下 `serialize_messages_tuple` 把 `(chunk, metadata)` 转成 `[chunk.model_dump(), metadata]`。
@ -173,7 +173,7 @@ sequenceDiagram
- 没有 `RunManager` —— 一次 `stream()` 调用对应一次生命周期，只生成轻量 `run_id` 供 runtime context、tracing 和 per-run middleware 使用。
- 没有 `StreamBridge` —— 直接 `yield`，生产和消费在同一个 Python 调用栈，不需要跨 task 中介。
- 没有 JSON 序列化 —— `StreamEvent.data` 直接装原生 LangChain 对象（`AIMessage.content`、`usage_metadata` 的 `UsageMetadata` TypedDict）。Jupyter 用户拿到的是真正的类型，不是匿名 dict。
- 没有 JSON 序列化 —— `StreamEvent.data` 直接装原生 LangChain 值（`AIMessage.content`、`usage_metadata` 的 `UsageMetadata` TypedDict，以及非 `None` 的 `ToolMessage.artifact`）。`messages-tuple` 工具结果和 `values` 快照里的工具消息都会保留 artifact；没有 artifact 的工具消息维持原有字段形状。Jupyter 用户拿到的是真正的类型，不是经过网络序列化后的匿名值。
- 没有 asyncio —— 调用者可以直接 `for event in ...`，不必写 `async for`。
---
@ -348,6 +348,7 @@ assert "messages" in agent.stream.call_args.kwargs["stream_mode"]
| 关心什么 | 看这里 |
|---|---|
| DeerFlowClient 嵌入式流 | `packages/harness/deerflow/client.py::DeerFlowClient.stream` |
| Embedded ToolMessage artifact 序列化 | `packages/harness/deerflow/client.py::_tool_message_event` / `_serialize_message` |
| `chat()` 的 delta 累加器 | `packages/harness/deerflow/client.py::DeerFlowClient.chat` |
| Gateway async 流 | `packages/harness/deerflow/runtime/runs/worker.py::run_agent` |
| HTTP SSE 帧输出 | `app/gateway/services.py::sse_consumer` / `format_sse` |

4
backend/packages/harness/deerflow/agents/middlewares/subagent_limit_middleware.py
View File

@ -36,7 +36,7 @@ _TOTAL_LIMIT_STOP_MSG = (
def _clamp_subagent_limit(value: int) -> int:
"""Clamp subagent limit to valid range [2, 4]."""
"""Clamp subagent limit to valid range [1, 4]."""
return clamp_subagent_concurrency(value)
@ -104,7 +104,7 @@ class SubagentLimitMiddleware(AgentMiddleware[AgentState]):
Args:
max_concurrent: Maximum number of concurrent subagent calls allowed.
Defaults to MAX_CONCURRENT_SUBAGENTS (3). Clamped to [2, 4].
Defaults to MAX_CONCURRENT_SUBAGENTS (3). Clamped to [1, 4].
max_total: Maximum number of subagent calls allowed across the run.
Defaults to 6. Clamped to [1, 50].
"""

23
backend/packages/harness/deerflow/client.py
View File

@ -431,16 +431,16 @@ class DeerFlowClient:
@staticmethod
def _tool_message_event(msg: ToolMessage) -> "StreamEvent":
"""Build a ``messages-tuple`` tool-result event from a ToolMessage."""
return StreamEvent(
type="messages-tuple",
data={
"type": "tool",
"content": DeerFlowClient._extract_text(msg.content),
"name": msg.name,
"tool_call_id": msg.tool_call_id,
"id": msg.id,
},
)
data: dict[str, Any] = {
"type": "tool",
"content": DeerFlowClient._extract_text(msg.content),
"name": msg.name,
"tool_call_id": msg.tool_call_id,
"id": msg.id,
}
if (artifact := getattr(msg, "artifact", None)) is not None:
data["artifact"] = artifact
return StreamEvent(type="messages-tuple", data=data)
@staticmethod
def _serialize_message(msg) -> dict:
@ -464,6 +464,8 @@ class DeerFlowClient:
}
if additional_kwargs := DeerFlowClient._serialize_additional_kwargs(msg):
d["additional_kwargs"] = additional_kwargs
if (artifact := getattr(msg, "artifact", None)) is not None:
d["artifact"] = artifact
return d
if isinstance(msg, HumanMessage):
d = {"type": "human", "content": msg.content, "id": getattr(msg, "id", None)}
@ -803,6 +805,7 @@ class DeerFlowClient:
- type="messages-tuple" data={"type": "ai", "content": "", "id": str, "tool_calls": [...]}
- type="messages-tuple" data={"type": "ai", "content": "", "id": str, "additional_kwargs": {...}}
- type="messages-tuple" data={"type": "tool", "content": str, "name": str, "tool_call_id": str, "id": str}
Tool results also include ``"artifact"`` when the source ToolMessage has a non-None artifact.
- type="end" data={"usage": {"input_tokens": int, "output_tokens": int, "total_tokens": int}}
"""
if thread_id is None:

683
backend/packages/harness/deerflow/community/e2b_sandbox/e2b_sandbox_provider.py
View File

@ -1,8 +1,7 @@
"""``E2BSandboxProvider`` — DeerFlow :class:`SandboxProvider` for e2b cloud.
Configuration is read from :class:`SandboxConfig` (which has
``extra="allow"``), so any keys below can appear under ``sandbox:`` in
``config.yaml`` even though they are not declared on the model:
Configuration is read from :class:`SandboxConfig`. E2B reports unknown
provider fields during startup.
.. code-block:: yaml
@ -11,8 +10,11 @@ Configuration is read from :class:`SandboxConfig` (which has
api_key: $E2B_API_KEY # required (or via E2B_API_KEY env var)
template: code-interpreter-v1 # default: e2b code-interpreter template
domain: e2b.dev # optional; for self-hosted e2b
idle_timeout: 600 # forwarded to ``set_timeout``
replicas: 3 # max concurrent sandboxes
idle_timeout: 1800 # forwarded to ``set_timeout``
replicas: 3 # max capacity (active + warm)
overflow_policy: wait # wait | reject | burst (default: wait)
acquire_timeout: 30 # seconds for ``wait`` policy (default: 30)
burst_limit: 2 # extra slots for ``burst`` policy (default: 0)
mounts: # one-shot uploads on sandbox start
- host_path: /data/skills
container_path: /home/user/skills
@ -35,7 +37,9 @@ import threading
import time
import uuid
from collections import OrderedDict
from concurrent.futures import ThreadPoolExecutor
from decimal import Decimal, InvalidOperation
from functools import partial
from pathlib import Path
from typing import Any
@ -43,6 +47,7 @@ from e2b_code_interpreter import Sandbox as E2BClientSandbox
from deerflow.config import get_app_config
from deerflow.runtime.user_context import get_effective_user_id
from deerflow.sandbox.exceptions import SandboxCapacityExceededError
from deerflow.sandbox.sandbox import Sandbox
from deerflow.sandbox.sandbox_provider import SandboxProvider
@ -55,6 +60,8 @@ logger = logging.getLogger(__name__)
DEFAULT_TEMPLATE = "code-interpreter-v1" # the public e2b code-interpreter template
DEFAULT_IDLE_TIMEOUT = 1800 # 30 minutes; passed to ``Sandbox.set_timeout``.
DEFAULT_REPLICAS = 3
DEFAULT_OVERFLOW_POLICY = "wait" # wait | reject | burst
DEFAULT_ACQUIRE_TIMEOUT = 30 # seconds for wait policy
# Hard upper bound for ``set_timeout`` (e2b currently caps at 24h on the
# free plan; passing an excessive value is rejected by the control-plane).
MAX_E2B_TIMEOUT = 24 * 60 * 60
@ -65,6 +72,7 @@ META_KEY_USER = "deer_flow_user"
META_KEY_THREAD = "deer_flow_thread"
META_KEY_PROVIDER = "deer_flow_provider"
META_VAL_PROVIDER = "e2b_sandbox_provider"
E2B_EXTRA_CONFIG_KEYS = frozenset({"api_key", "domain", "home_dir", "template"})
class E2BSandboxProvider(SandboxProvider):
@ -90,9 +98,31 @@ class E2BSandboxProvider(SandboxProvider):
# Warm pool: released sandboxes whose remote micro-VM is still alive.
# ``OrderedDict`` maintains insertion / move_to_end order for LRU.
self._warm_pool: OrderedDict[str, tuple[str, float]] = OrderedDict()
# Evictions with unknown remote state. Each id keeps its transition
# slot until a later eviction attempt confirms destruction.
self._eviction_tombstones: set[str] = set()
# IDs currently reconnecting or stopping. This grants one retry owner
# to each tombstone and prevents a second call from freeing its slot.
self._evictions_in_progress: set[str] = set()
# Remote IDs that are between tracked lifecycle states. Shutdown uses
# this set to retain cleanup visibility during remote calls.
self._remote_ops_in_progress: set[str] = set()
# Discovery can find a VM that another Gateway still uses.
# Shutdown closes these clients but does not destroy their VMs.
self._unowned_remote_ops_in_progress: set[str] = set()
# In-flight creates that have reserved capacity but not yet committed
# to ``_sandboxes``. Guarded by ``_lock``.
self._reserved_slots = 0
self._transitioning_slots = 0
self._capacity_cond = threading.Condition(self._lock)
self._shutdown_called = False
self._config = self._load_config()
acquire_workers = max(4, min(32, self._capacity_limit() + 1))
self._acquire_executor = ThreadPoolExecutor(
max_workers=acquire_workers,
thread_name_prefix="e2b-sandbox-acquire",
)
atexit.register(self.shutdown)
self._register_signal_handlers()
@ -100,6 +130,12 @@ class E2BSandboxProvider(SandboxProvider):
def _load_config(self) -> dict[str, Any]:
"""Read e2b options off ``SandboxConfig`` (``extra="allow"``)."""
sandbox_config = get_app_config().sandbox
unknown_keys = sorted(set(getattr(sandbox_config, "model_extra", None) or {}) - E2B_EXTRA_CONFIG_KEYS)
if unknown_keys:
logger.warning(
"E2BSandboxProvider: unknown sandbox config fields: %s",
", ".join(unknown_keys),
)
def _opt(name: str, default: Any = None) -> Any:
return getattr(sandbox_config, name, default)
@ -114,7 +150,24 @@ class E2BSandboxProvider(SandboxProvider):
idle_timeout = max(0, min(int(idle_timeout), MAX_E2B_TIMEOUT))
replicas = _opt("replicas")
replicas = DEFAULT_REPLICAS if replicas is None else max(1, int(replicas))
replicas = DEFAULT_REPLICAS if replicas is None else int(replicas)
overflow_policy = _opt("overflow_policy") or DEFAULT_OVERFLOW_POLICY
if overflow_policy not in ("wait", "reject", "burst"):
logger.warning("E2BSandboxProvider: invalid overflow_policy %r; falling back to %r", overflow_policy, DEFAULT_OVERFLOW_POLICY)
overflow_policy = DEFAULT_OVERFLOW_POLICY
acquire_timeout = _opt("acquire_timeout")
if acquire_timeout is None:
acquire_timeout = DEFAULT_ACQUIRE_TIMEOUT
else:
acquire_timeout = max(1, int(acquire_timeout))
burst_limit_raw = _opt("burst_limit")
burst_limit = max(0, int(burst_limit_raw)) if burst_limit_raw is not None else 0
if overflow_policy == "burst" and burst_limit == 0:
logger.warning("E2BSandboxProvider: overflow_policy is 'burst' but burst_limit is 0; falling back to 'reject'")
overflow_policy = "reject"
return {
"api_key": api_key,
@ -123,6 +176,9 @@ class E2BSandboxProvider(SandboxProvider):
"home_dir": _opt("home_dir") or DEFAULT_E2B_HOME_DIR,
"idle_timeout": idle_timeout,
"replicas": replicas,
"overflow_policy": overflow_policy,
"acquire_timeout": acquire_timeout,
"burst_limit": burst_limit,
"mounts": _opt("mounts") or [],
"environment": self._resolve_env_vars(_opt("environment") or {}),
}
@ -209,7 +265,9 @@ class E2BSandboxProvider(SandboxProvider):
async def acquire_async(self, thread_id: str | None = None, *, user_id: str | None = None) -> str:
effective_user_id = self._effective_acquire_user_id(user_id)
return await asyncio.to_thread(self.acquire, thread_id, user_id=effective_user_id)
loop = asyncio.get_running_loop()
acquire = partial(self.acquire, thread_id, user_id=effective_user_id)
return await loop.run_in_executor(self._acquire_executor, acquire)
def _acquire_internal(self, thread_id: str | None, *, user_id: str) -> str:
if thread_id:
@ -274,6 +332,13 @@ class E2BSandboxProvider(SandboxProvider):
return sid
def _reclaim_warm_pool_sandbox(self, thread_id: str, *, user_id: str) -> str | None:
"""Reclaim a warm-pool sandbox, holding a transitioning slot throughout.
The warm-pool entry is popped and a transitioning slot is taken
immediately. The slot is committed to active when the sandbox is
registered in ``_sandboxes``, or freed if the reclaim fails.
If the provider shut down during the transition, the VM is killed.
"""
seed = self._stable_seed(thread_id, user_id)
with self._lock:
target_id = next(
@ -283,6 +348,8 @@ class E2BSandboxProvider(SandboxProvider):
if target_id is None:
return None
self._warm_pool.pop(target_id)
self._begin_transition_locked()
self._remote_ops_in_progress.add(target_id)
try:
client = self._reconnect_live_client(self._get_sandbox_cls(), target_id)
@ -292,6 +359,7 @@ class E2BSandboxProvider(SandboxProvider):
target_id,
e,
)
self._complete_transition_remote_op(target_id, remote_destroyed=False)
return None
if client is None:
@ -299,12 +367,36 @@ class E2BSandboxProvider(SandboxProvider):
"Warm-pool e2b sandbox %s is no longer alive (reaped by control plane); dropping and falling back to create",
target_id,
)
self._complete_transition_remote_op(target_id, remote_destroyed=True)
return None
self._refresh_remote_timeout(client)
if self._bootstrap_or_discard(client, target_id) is not None:
bootstrap_error, remote_destroyed = self._bootstrap_or_discard(client, target_id)
if bootstrap_error is not None:
if remote_destroyed:
self._complete_transition_remote_op(target_id, remote_destroyed=True)
else:
self._complete_transition_remote_op(target_id, remote_destroyed=False)
return None
self._register_connected_sandbox(target_id, client, thread_id=thread_id, user_id=user_id)
discard_after_shutdown = False
with self._lock:
if self._shutdown_called:
logger.info(
"Provider shut down during reclaim of sandbox %s; killing VM",
target_id,
)
discard_after_shutdown = True
else:
self._remote_ops_in_progress.discard(target_id)
self._register_connected_sandbox(target_id, client, thread_id=thread_id, user_id=user_id)
self._end_transition_locked()
if discard_after_shutdown:
self._kill_client(client)
self._safe_close_client(client)
return None
logger.info(
"Reclaimed warm-pool e2b sandbox %s for user/thread %s/%s",
target_id,
@ -415,10 +507,44 @@ class E2BSandboxProvider(SandboxProvider):
)
return None
try:
self._reserve_capacity(
thread_id,
user_id,
remote_id=target_id,
remote_owned=False,
)
except SandboxCapacityExceededError as error:
if error.reason == "shutdown":
logger.info(
"Discovered e2b sandbox %s while the provider is shutting down; not adopting it",
target_id,
)
else:
logger.info(
"Discovered e2b sandbox %s, but capacity is full; not adopting it",
target_id,
)
self._safe_close_client(client)
raise
self._refresh_remote_timeout(client)
if self._bootstrap_or_discard(client, target_id) is not None:
bootstrap_error, remote_destroyed = self._bootstrap_or_discard(client, target_id)
if bootstrap_error is not None:
self._complete_reserved_remote_op(target_id, remote_destroyed=remote_destroyed)
return None
discard_after_shutdown = False
with self._lock:
if self._shutdown_called:
discard_after_shutdown = True
else:
self._unowned_remote_ops_in_progress.discard(target_id)
self._register_connected_sandbox(target_id, client, thread_id=thread_id, user_id=user_id)
self._commit_capacity()
if discard_after_shutdown:
self._safe_close_client(client)
return None
self._register_connected_sandbox(target_id, client, thread_id=thread_id, user_id=user_id)
logger.info(
"Discovered remote e2b sandbox %s for user/thread %s/%s (seed=%s)",
target_id,
@ -428,20 +554,235 @@ class E2BSandboxProvider(SandboxProvider):
)
return target_id
def _create_sandbox(self, thread_id: str | None, *, user_id: str) -> str:
"""Allocate a fresh e2b sandbox and hydrate it with configured mounts."""
# ── Capacity reservation ──────────────────────────────────────────────
#
# Every sandbox holds one *slot*. A slot is in exactly one of four states:
#
# reserved _reserved_slots create in flight, not yet in _sandboxes
# active _sandboxes serving a thread
# warm _warm_pool released, parked for reuse
# transitioning _transitioning_slots being moved between states
#
# Total = _reserved_slots + len(_sandboxes) + len(_warm_pool) + _transitioning_slots
#
# The ``transitioning`` bucket closes the window where a sandbox has been
# removed from ``_sandboxes`` (or ``_warm_pool``) but not yet parked in its
# destination. Without it, ``_release_internal`` (active → warm),
# ``_reclaim_warm_pool_sandbox`` (warm → active), and ``_evict_oldest_warm``
# (warm → destroyed) all temporarily appear to have *zero* slots occupied,
# letting a concurrent acquire reserve a new slot and exceed the configured
# ``replicas``.
def _total_capacity_used_locked(self) -> int:
"""Return reserved + active + warm + transitioning (``_lock`` must be held)."""
return self._reserved_slots + len(self._sandboxes) + len(self._warm_pool) + self._transitioning_slots
def _capacity_limit(self) -> int:
"""Hard ceiling for reserved + active + warm + transitioning slots."""
replicas = int(self._config["replicas"])
if self._config["overflow_policy"] == "burst":
return replicas + int(self._config["burst_limit"])
return replicas
def _begin_transition_locked(self) -> None:
"""Increment the transitioning counter (``_lock`` must be held).
Call before removing a sandbox from ``_sandboxes`` or ``_warm_pool``
so the slot is still counted while the transition is in flight.
"""
self._transitioning_slots += 1
def _end_transition_locked(self) -> None:
"""Decrement the transitioning counter (``_lock`` must be held).
Call after the transition completes — either the slot has been parked
in its destination dict or the sandbox has been destroyed.
"""
if self._transitioning_slots > 0:
self._transitioning_slots -= 1
self._capacity_cond.notify_all()
def _free_transitioning_slot(self) -> None:
"""Release a transitioning slot after the sandbox was destroyed."""
with self._lock:
in_use = len(self._sandboxes) + len(self._warm_pool)
if in_use >= replicas:
self._end_transition_locked()
def _reserve_capacity(
self,
thread_id: str | None,
user_id: str,
*,
remote_id: str | None = None,
remote_owned: bool = True,
) -> None:
"""Acquire a capacity slot, blocking or raising as configured.
Must be called before ``Sandbox.create()``. The caller MUST call
``_commit_capacity()`` on success or ``_release_capacity()`` on
failure — otherwise the reserved slot is leaked until shutdown.
Raises:
SandboxCapacityExceededError: when the overflow policy is
``reject`` or the wait timeout expires.
"""
policy = self._config["overflow_policy"]
timeout = float(self._config["acquire_timeout"])
deadline = time.monotonic() + timeout
while True:
# Reject immediately if the provider is shutting down.
with self._lock:
if self._shutdown_called:
raise SandboxCapacityExceededError(
"Sandbox provider is shutting down; cannot acquire capacity",
replicas=int(self._config["replicas"]),
retry_after_seconds=30.0,
reason="shutdown",
)
# 1. Try immediate atomic reservation.
with self._lock:
if self._shutdown_called:
raise SandboxCapacityExceededError(
"Sandbox provider is shutting down; cannot acquire capacity",
replicas=int(self._config["replicas"]),
retry_after_seconds=30.0,
reason="shutdown",
)
cap = self._capacity_limit()
if self._total_capacity_used_locked() < cap:
self._reserved_slots += 1
if remote_id is not None:
remote_ops = self._remote_ops_in_progress if remote_owned else self._unowned_remote_ops_in_progress
remote_ops.add(remote_id)
return
# 2. Try evicting a warm entry to free a slot.
evicted = self._evict_oldest_warm()
if evicted is None:
logger.warning(
"All %d e2b replica slots are in active use; creating a new sandbox beyond the soft limit (active=%d, warm=%d)",
replicas,
len(self._sandboxes),
len(self._warm_pool),
)
if evicted is not None:
with self._lock:
if self._shutdown_called:
raise SandboxCapacityExceededError(
"Sandbox provider shut down while acquiring capacity",
replicas=int(self._config["replicas"]),
retry_after_seconds=30.0,
reason="shutdown",
)
if self._total_capacity_used_locked() < cap:
self._reserved_slots += 1
if remote_id is not None:
remote_ops = self._remote_ops_in_progress if remote_owned else self._unowned_remote_ops_in_progress
remote_ops.add(remote_id)
return
# Slot was stolen; fall through to policy / wait.
# 3. Apply overflow policy.
with self._lock:
if self._shutdown_called:
raise SandboxCapacityExceededError(
"Sandbox provider is shutting down; cannot acquire capacity",
replicas=int(self._config["replicas"]),
retry_after_seconds=30.0,
reason="shutdown",
)
used = self._total_capacity_used_locked()
cap = self._capacity_limit()
if used >= cap:
if policy == "reject":
raise SandboxCapacityExceededError(
f"All {cap} sandbox capacity slots are in use and overflow_policy is 'reject'",
active=len(self._sandboxes),
warm=len(self._warm_pool),
reserved=self._reserved_slots,
replicas=int(self._config["replicas"]),
)
if policy == "burst":
raise SandboxCapacityExceededError(
f"All {cap} sandbox capacity slots are in use (replicas={self._config['replicas']}, burst={self._config['burst_limit']})",
active=len(self._sandboxes),
warm=len(self._warm_pool),
reserved=self._reserved_slots,
replicas=int(self._config["replicas"]),
)
# policy == "wait": block until a slot frees or timeout.
remaining = deadline - time.monotonic()
if remaining <= 0:
raise SandboxCapacityExceededError(
f"Timed out after {timeout}s waiting for a sandbox capacity slot (replicas={self._config['replicas']}, active={len(self._sandboxes)}, warm={len(self._warm_pool)}, reserved={self._reserved_slots})",
active=len(self._sandboxes),
warm=len(self._warm_pool),
reserved=self._reserved_slots,
replicas=int(self._config["replicas"]),
)
self._capacity_cond.wait(timeout=min(remaining, 1.0))
def _release_capacity(self) -> None:
"""Release a reserved slot (call on create failure or destroy)."""
with self._lock:
if self._reserved_slots > 0:
self._reserved_slots -= 1
self._capacity_cond.notify_all()
def _complete_transition_remote_op(self, sandbox_id: str, *, remote_destroyed: bool) -> None:
"""Finish a remote operation that already owns a transition slot."""
with self._lock:
if sandbox_id not in self._remote_ops_in_progress:
return
self._remote_ops_in_progress.discard(sandbox_id)
if self._shutdown_called:
return
if remote_destroyed:
self._end_transition_locked()
else:
self._eviction_tombstones.add(sandbox_id)
def _track_reserved_remote_op(self, sandbox_id: str) -> bool:
"""Make a reserved remote ID visible to shutdown."""
with self._lock:
if self._shutdown_called:
return False
self._remote_ops_in_progress.add(sandbox_id)
return True
def _complete_reserved_remote_op(self, sandbox_id: str, *, remote_destroyed: bool) -> None:
"""Finish a remote operation that owns a reserved slot."""
with self._lock:
tracked = sandbox_id in self._remote_ops_in_progress
tracked_unowned = sandbox_id in self._unowned_remote_ops_in_progress
if not tracked and not tracked_unowned:
return
self._remote_ops_in_progress.discard(sandbox_id)
self._unowned_remote_ops_in_progress.discard(sandbox_id)
if self._shutdown_called:
return
if self._reserved_slots > 0:
self._reserved_slots -= 1
if remote_destroyed:
self._capacity_cond.notify_all()
else:
self._begin_transition_locked()
self._eviction_tombstones.add(sandbox_id)
self._capacity_cond.notify_all()
def _commit_capacity(self) -> None:
"""Convert a reserved slot to a committed active slot.
The reservation is dropped and the newly-created sandbox fills the
slot. Must be called inside the same critical section that inserts
into ``_sandboxes``.
"""
if self._reserved_slots > 0:
self._reserved_slots -= 1
def _create_sandbox(self, thread_id: str | None, *, user_id: str) -> str:
"""Allocate a fresh e2b sandbox and hydrate it with configured mounts.
Capacity is enforced atomically via :meth:`_reserve_capacity`.
"""
self._reserve_capacity(thread_id, user_id)
sandbox_cls = self._get_sandbox_cls()
metadata: dict[str, str] = {
@ -465,18 +806,58 @@ class E2BSandboxProvider(SandboxProvider):
client = sandbox_cls.create(**create_kwargs) # type: ignore[attr-defined]
except Exception as e:
logger.error("Failed to create e2b sandbox: %s", e)
self._release_capacity()
raise
sandbox_id: str = getattr(client, "sandbox_id", None) or str(uuid.uuid4())[:8]
if not self._track_reserved_remote_op(sandbox_id):
kill_error = self._kill_client(client)
cleanup_confirmed = kill_error is None
self._safe_close_client(client)
if kill_error is not None:
with self._lock:
self._remote_ops_in_progress.add(sandbox_id)
try:
retry_client = self._reconnect_client(sandbox_cls, sandbox_id)
except Exception as reconnect_error:
logger.warning(
"Failed to reconnect e2b sandbox %s after shutdown cleanup failed: %s",
sandbox_id,
reconnect_error,
)
else:
retry_error = self._kill_client(retry_client)
self._safe_close_client(retry_client)
if retry_error is None:
cleanup_confirmed = True
with self._lock:
self._remote_ops_in_progress.discard(sandbox_id)
else:
logger.warning(
"Failed to kill e2b sandbox %s after reconnecting during shutdown: %s",
sandbox_id,
retry_error,
)
if cleanup_confirmed:
message = f"Sandbox provider shut down during sandbox creation; cleaned up remote sandbox {sandbox_id}"
else:
message = f"Sandbox provider shut down during sandbox creation; could not confirm cleanup for remote sandbox {sandbox_id}"
raise SandboxCapacityExceededError(
message,
replicas=int(self._config["replicas"]),
retry_after_seconds=30.0,
reason="shutdown",
)
# Materialise DeerFlow's virtual path layout (/mnt/user-data/...) inside
# the e2b VM. Without this step shell commands the agent emits — which
# use the same /mnt/user-data prefix as LocalSandbox / AioSandbox — fail
# with PermissionError because /mnt is owned by root in the e2b
# template. See the path-mapping note in :class:`E2BSandbox`.
error = self._bootstrap_or_discard(client, sandbox_id)
if error is not None:
raise RuntimeError(f"Failed to bootstrap e2b sandbox {sandbox_id}") from error
bootstrap_error, remote_destroyed = self._bootstrap_or_discard(client, sandbox_id)
if bootstrap_error is not None:
self._complete_reserved_remote_op(sandbox_id, remote_destroyed=remote_destroyed)
raise RuntimeError(f"Failed to bootstrap e2b sandbox {sandbox_id}") from bootstrap_error
# One-shot mount uploads. e2b has no host bind-mount, so we copy
# files from ``host_path`` into ``container_path`` at sandbox start.
@ -486,13 +867,34 @@ class E2BSandboxProvider(SandboxProvider):
logger.warning("Failed to apply some mounts to e2b sandbox %s: %s", sandbox_id, e)
sandbox = E2BSandbox(id=sandbox_id, client=client, home_dir=self._config["home_dir"])
with self._lock:
self._sandboxes[sandbox_id] = sandbox
if thread_id:
self._thread_sandboxes[self._thread_key(thread_id, user_id)] = sandbox_id
# Commit atomically. If the provider shut down during bootstrap or
# mounts, kill the VM rather than parking it under ``_sandboxes``
# where the next shutdown won't see it.
should_kill = False
with self._lock:
if self._shutdown_called:
should_kill = True
else:
self._remote_ops_in_progress.discard(sandbox_id)
self._commit_capacity()
self._sandboxes[sandbox_id] = sandbox
if thread_id:
self._thread_sandboxes[self._thread_key(thread_id, user_id)] = sandbox_id
if should_kill:
self._kill_client(client)
self._safe_close_client(client)
raise SandboxCapacityExceededError(
f"Sandbox provider shut down during sandbox creation; killed remote sandbox {sandbox_id}",
replicas=int(self._config["replicas"]),
retry_after_seconds=30.0,
reason="shutdown",
)
replicas = self._config["replicas"]
logger.info(
"Created e2b sandbox %s for user/thread %s/%s (template=%s, replicas=%d)",
"Created e2b sandbox %s for user/thread %s/%s (template=%s, replicas=%s)",
sandbox_id,
user_id,
thread_id,
@ -539,11 +941,13 @@ class E2BSandboxProvider(SandboxProvider):
thread_id: str,
user_id: str,
) -> None:
"""Track a live reconnected sandbox under its thread ownership."""
"""Track a live reconnected sandbox under its thread ownership.
The caller must hold ``self._lock``.
"""
sandbox = E2BSandbox(id=sandbox_id, client=client, home_dir=self._config["home_dir"])
with self._lock:
self._sandboxes[sandbox_id] = sandbox
self._thread_sandboxes[self._thread_key(thread_id, user_id)] = sandbox_id
self._sandboxes[sandbox_id] = sandbox
self._thread_sandboxes[self._thread_key(thread_id, user_id)] = sandbox_id
def _refresh_remote_timeout(self, client: E2BClientSandbox) -> None:
"""Push the configured idle timeout to the e2b control plane."""
@ -606,17 +1010,18 @@ class E2BSandboxProvider(SandboxProvider):
logger.debug("e2b client close raised: %s", e)
return
def _bootstrap_or_discard(self, client: E2BClientSandbox, sandbox_id: str) -> Exception | None:
"""Bootstrap a sandbox or return its error after cleanup."""
def _bootstrap_or_discard(self, client: E2BClientSandbox, sandbox_id: str) -> tuple[Exception | None, bool]:
"""Bootstrap a sandbox and report whether cleanup destroyed the VM."""
try:
self._bootstrap_sandbox_paths(client)
except Exception as e:
logger.exception("Failed to bootstrap e2b sandbox %s. Discarding the unusable sandbox.", sandbox_id)
if error := self._kill_client(client):
logger.warning("Failed to kill e2b sandbox %s after bootstrap failure: %s", sandbox_id, error)
kill_error = self._kill_client(client)
if kill_error:
logger.warning("Failed to kill e2b sandbox %s after bootstrap failure: %s", sandbox_id, kill_error)
self._safe_close_client(client)
return e
return None
return e, kill_error is None
return None, True
def _bootstrap_sandbox_paths(self, client: E2BClientSandbox) -> None:
"""Materialise DeerFlow's virtual path layout inside the e2b VM.
@ -1036,29 +1441,64 @@ class E2BSandboxProvider(SandboxProvider):
pass
def _evict_oldest_warm(self) -> str | None:
"""Evict the oldest warm entry, holding a transitioning slot.
The warm entry is popped and a transitioning slot is taken. The slot
stays occupied until the control plane confirms the VM is gone.
"""
with self._lock:
if not self._warm_pool:
retryable = self._eviction_tombstones - self._evictions_in_progress
if retryable:
evict_id = next(iter(retryable))
self._evictions_in_progress.add(evict_id)
elif self._warm_pool:
evict_id, (_, _) = self._warm_pool.popitem(last=False)
self._eviction_tombstones.add(evict_id)
self._evictions_in_progress.add(evict_id)
self._begin_transition_locked()
else:
return None
evict_id, (_, _) = self._warm_pool.popitem(last=False)
try:
client = self._reconnect_client(self._get_sandbox_cls(), evict_id)
client = self._reconnect_live_client(self._get_sandbox_cls(), evict_id)
except Exception as e:
logger.warning(
"Evicted warm-pool e2b sandbox %s could not be reconnected for kill: %s",
evict_id,
e,
)
with self._lock:
if evict_id in self._evictions_in_progress:
self._evictions_in_progress.discard(evict_id)
if not self._shutdown_called:
self._eviction_tombstones.add(evict_id)
return None
if client is None:
with self._lock:
if evict_id in self._evictions_in_progress:
self._evictions_in_progress.discard(evict_id)
self._eviction_tombstones.discard(evict_id)
self._end_transition_locked()
logger.info("Evicted warm-pool e2b sandbox %s was already gone", evict_id)
return evict_id
if error := self._kill_client(client):
logger.warning("Failed to kill evicted e2b sandbox %s: %s", evict_id, error)
close = getattr(client, "close", None)
if callable(close):
try:
close()
except Exception:
pass
self._safe_close_client(client)
with self._lock:
if evict_id in self._evictions_in_progress:
self._evictions_in_progress.discard(evict_id)
if not self._shutdown_called:
self._eviction_tombstones.add(evict_id)
return None
self._safe_close_client(client)
with self._lock:
if evict_id in self._evictions_in_progress:
self._evictions_in_progress.discard(evict_id)
self._eviction_tombstones.discard(evict_id)
self._end_transition_locked()
logger.info("Evicted warm-pool e2b sandbox %s", evict_id)
return evict_id
@ -1088,13 +1528,25 @@ class E2BSandboxProvider(SandboxProvider):
self._release_internal(sandbox_id)
def _release_internal(self, sandbox_id: str) -> None:
"""Complete one release while the thread transition lock is held."""
"""Complete one release while the thread transition lock is held.
The active slot becomes a *transitioning* slot the moment the sandbox
is removed from ``_sandboxes``. It stays counted through output sync
and timeout refresh. The transition ends when the VM enters its
destination or destruction completes. Shutdown kills the VM instead
of parking it in ``_warm_pool``.
"""
sandbox: E2BSandbox | None = None
seed: str | None = None
removed_keys: list[tuple[str, str]] = []
transition_slot_held = False
with self._lock:
sandbox = self._sandboxes.pop(sandbox_id, None)
# Find the (user, thread) the sandbox was bound to.
if sandbox is None:
return
self._begin_transition_locked()
transition_slot_held = True
removed_keys = [key for key, sid in self._thread_sandboxes.items() if sid == sandbox_id]
for key in removed_keys:
self._thread_sandboxes.pop(key, None)
@ -1102,53 +1554,72 @@ class E2BSandboxProvider(SandboxProvider):
user_id, thread_id = removed_keys[0]
seed = self._stable_seed(thread_id, user_id)
if sandbox is None:
return
# E2BSandbox.close() clears its client reference. Keep this reference
# so a shutdown that races release can still kill the remote VM.
client = sandbox.client
if sandbox.is_dead:
logger.info(
"Releasing dead e2b sandbox %s; skipping output sync and warm pool, killing remote VM",
sandbox_id,
)
self._kill_and_close(sandbox)
return
sync_failed_due_to_dead_vm = False
if seed is not None and removed_keys:
user_id_sync, thread_id_sync = removed_keys[0]
try:
self._sync_outputs_to_host(sandbox, thread_id=thread_id_sync, user_id=user_id_sync)
except Exception as e: # pragma: no cover - defensive
logger.warning(
"Failed to mirror e2b sandbox %s outputs to host: %s",
sandbox_id,
e,
)
try:
if sandbox.is_dead:
sync_failed_due_to_dead_vm = True
logger.info(
"Releasing dead e2b sandbox %s; skipping output sync and warm pool, killing remote VM",
sandbox_id,
)
self._kill_and_close(sandbox)
return
if sync_failed_due_to_dead_vm:
logger.info(
"Sandbox %s was reaped during release; not parking in warm pool",
sandbox_id,
)
self._kill_and_close(sandbox)
return
sync_failed_due_to_dead_vm = False
if seed is not None and removed_keys:
user_id_sync, thread_id_sync = removed_keys[0]
try:
self._sync_outputs_to_host(sandbox, thread_id=thread_id_sync, user_id=user_id_sync)
except Exception as e: # pragma: no cover - defensive
logger.warning(
"Failed to mirror e2b sandbox %s outputs to host: %s",
sandbox_id,
e,
)
if sandbox.is_dead:
sync_failed_due_to_dead_vm = True
try:
self._refresh_remote_timeout(sandbox.client)
except Exception as e:
logger.debug("Failed to refresh timeout during release: %s", e)
if sync_failed_due_to_dead_vm:
logger.info(
"Sandbox %s was reaped during release; not parking in warm pool",
sandbox_id,
)
self._kill_and_close(sandbox)
return
try:
sandbox.close()
except Exception as e:
logger.warning("Error closing e2b sandbox %s during release: %s", sandbox_id, e)
try:
self._refresh_remote_timeout(client)
except Exception as e:
logger.debug("Failed to refresh timeout during release: %s", e)
with self._lock:
self._warm_pool[sandbox_id] = (seed or "", time.time())
self._warm_pool.move_to_end(sandbox_id)
logger.info("Released e2b sandbox %s to warm pool", sandbox_id)
with self._lock:
should_kill = self._shutdown_called
if not should_kill:
self._warm_pool[sandbox_id] = (seed or "", time.time())
self._warm_pool.move_to_end(sandbox_id)
self._end_transition_locked()
transition_slot_held = False
logger.info("Released e2b sandbox %s to warm pool", sandbox_id)
if should_kill:
logger.info(
"Provider shut down during release of sandbox %s; killing instead of parking in warm pool",
sandbox_id,
)
if error := self._kill_client(client):
logger.debug("Failed to kill e2b sandbox %s during release: %s", sandbox_id, error)
self._safe_close_client(client)
return
try:
sandbox.close()
except Exception as e:
logger.warning("Error closing e2b sandbox %s during release: %s", sandbox_id, e)
finally:
if transition_slot_held:
self._free_transitioning_slot()
def _kill_and_close(self, sandbox: E2BSandbox) -> None:
if error := self._kill_client(getattr(sandbox, "_client", None)):
@ -1168,21 +1639,19 @@ class E2BSandboxProvider(SandboxProvider):
) -> Exception | None:
"""Kill a remote VM and return an exception for the caller to log."""
if client is None:
return None
return RuntimeError("Cannot confirm remote VM destruction without a client")
try:
kill = getattr(client, "kill", None)
if callable(kill):
kill()
if not callable(kill):
return RuntimeError("Cannot confirm remote VM destruction without a callable kill method")
kill()
except Exception as e:
return e
return None
def reset(self) -> None:
with self._lock:
self._sandboxes.clear()
self._thread_sandboxes.clear()
self._thread_locks.clear()
self._warm_pool.clear()
"""Destroy tracked E2B VMs and make this detached provider unusable."""
self.shutdown()
def shutdown(self) -> None:
with self._lock:
@ -1190,10 +1659,22 @@ class E2BSandboxProvider(SandboxProvider):
return
self._shutdown_called = True
active = list(self._sandboxes.items())
warm_ids = list(self._warm_pool.keys())
warm_ids = list(self._warm_pool.keys() | self._eviction_tombstones | self._remote_ops_in_progress)
self._sandboxes.clear()
self._warm_pool.clear()
self._eviction_tombstones.clear()
self._evictions_in_progress.clear()
self._remote_ops_in_progress.clear()
self._unowned_remote_ops_in_progress.clear()
self._thread_sandboxes.clear()
self._thread_locks.clear()
self._reserved_slots = 0
self._transitioning_slots = 0
self._capacity_cond.notify_all()
executor = getattr(self, "_acquire_executor", None)
if executor is not None:
executor.shutdown(wait=False, cancel_futures=True)
logger.info(
"Shutting down E2BSandboxProvider: %d active + %d warm sandboxes",

23
backend/packages/harness/deerflow/config/sandbox_config.py
View File

@ -3,6 +3,7 @@ from typing import Literal
from pydantic import BaseModel, ConfigDict, Field
SandboxOwnershipType = Literal["memory", "redis"]
SandboxOverflowPolicy = Literal["wait", "reject", "burst"]
class SandboxOwnershipConfig(BaseModel):
@ -73,9 +74,10 @@ class SandboxConfig(BaseModel):
allow_host_bash: Enable host-side bash execution for LocalSandboxProvider.
Dangerous and intended only for fully trusted local workflows.
AioSandboxProvider and BoxliteProvider shared options:
AioSandboxProvider, BoxliteProvider, and E2BSandboxProvider shared options:
image: Sandbox image to use (Docker/AIO image or BoxLite OCI image)
replicas: Maximum active + warm sandboxes/VMs per gateway process (default: 3). When the limit is reached, warm/least-recently-used sandboxes are evicted to make room; active sandboxes are not forcibly stopped.
replicas: Positive provider capacity per gateway process. Each provider
defines which lifecycle states count toward this limit.
idle_timeout: Idle timeout in seconds before released warm sandboxes/VMs are stopped (default: 600 = 10 minutes). Set to 0 to disable.
environment: Environment variables to inject into the sandbox (values starting with $ are resolved from host env)
@ -108,7 +110,22 @@ class SandboxConfig(BaseModel):
)
replicas: int | None = Field(
default=None,
description="Maximum active + warm sandboxes/VMs per gateway process (default: 3). Warm/least-recently-used entries are evicted to make room; active sandboxes are not forcibly stopped.",
gt=0,
description="Positive provider capacity per gateway process. Each provider defines which lifecycle states count toward this limit.",
)
overflow_policy: SandboxOverflowPolicy = Field(
default="wait",
description="E2B capacity policy. Use wait, reject, or burst.",
)
acquire_timeout: int = Field(
default=30,
gt=0,
description="Seconds that E2B wait policy waits for capacity.",
)
burst_limit: int = Field(
default=0,
ge=0,
description="Extra E2B capacity slots when overflow_policy is burst.",
)
container_prefix: str | None = Field(
default=None,

2
backend/packages/harness/deerflow/config/subagents_config.py
View File

@ -11,7 +11,7 @@ logger = logging.getLogger(__name__)
DEFAULT_MAX_TOTAL_SUBAGENTS_PER_RUN = 6
MIN_TOTAL_SUBAGENTS_PER_RUN = 1
MAX_TOTAL_SUBAGENTS_PER_RUN = 50
MIN_CONCURRENT_SUBAGENT_CALLS = 2
MIN_CONCURRENT_SUBAGENT_CALLS = 1
MAX_CONCURRENT_SUBAGENT_CALLS = 4

130
backend/packages/harness/deerflow/persistence/migrations/versions/0007_scheduled_run_active_index.py Normal file
View File

@ -0,0 +1,130 @@
"""scheduled task run active uniqueness.
Revision ID: 0007_scheduled_run_active_index
Revises: 0006_agents
Create Date: 2026-07-11
"""
from __future__ import annotations
import logging
from collections.abc import Sequence
from datetime import UTC, datetime
import sqlalchemy as sa
from alembic import op
logger = logging.getLogger(__name__)
revision: str = "0007_scheduled_run_active_index"
down_revision: str | Sequence[str] | None = "0006_agents"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _dedupe_active_scheduled_runs_per_task() -> None:
"""Supersede duplicate active rows so the partial unique index can be built.
``uq_scheduled_task_run_active`` enforces at most one queued/running row per
``task_id``. A DB that already has two+ active rows for the same task (the
exact TOCTOU this PR closes: two concurrent ``dispatch_task`` calls both
passed ``has_active_runs`` and both inserted a "queued" row) would fail
``CREATE UNIQUE INDEX`` and abort the alembic upgrade, blocking gateway
startup.
Keep the newest active row per ``task_id`` (by ``created_at`` DESC, ``id``
DESC as a deterministic tiebreaker) and mark the rest ``interrupted`` with
an explanatory ``error`` and a ``finished_at`` — the same orphan semantics
``ScheduledTaskRunRepository.mark_stale_active_runs`` uses for runs whose
process is gone.
"""
bind = op.get_bind()
superseded_message = "interrupted during migration 0007_scheduled_run_active_index: superseded by a newer active run for the same scheduled task (partial unique index uq_scheduled_task_run_active)"
find_dupe_rows = sa.text(
"""
SELECT id, task_id
FROM scheduled_task_runs AS r1
WHERE r1.status IN ('queued', 'running')
AND EXISTS (
SELECT 1 FROM scheduled_task_runs AS r2
WHERE r2.task_id = r1.task_id
AND r2.status IN ('queued', 'running')
AND r2.id <> r1.id
AND (
r2.created_at > r1.created_at
OR (r2.created_at = r1.created_at AND r2.id > r1.id)
)
)
"""
)
rows = list(bind.execute(find_dupe_rows).fetchall())
if not rows:
return
for run_id, task_id in rows:
logger.warning(
"migration 0007_scheduled_run_active_index: superseding duplicate active scheduled run %s on task %s",
run_id,
task_id,
)
update_stmt = sa.text(
"""
UPDATE scheduled_task_runs
SET status = 'interrupted',
error = :error_message,
finished_at = :finished_at
WHERE status IN ('queued', 'running')
AND EXISTS (
SELECT 1 FROM scheduled_task_runs AS r2
WHERE r2.task_id = scheduled_task_runs.task_id
AND r2.status IN ('queued', 'running')
AND r2.id <> scheduled_task_runs.id
AND (
r2.created_at > scheduled_task_runs.created_at
OR (r2.created_at = scheduled_task_runs.created_at AND r2.id > scheduled_task_runs.id)
)
)
"""
).bindparams(
sa.bindparam("error_message"),
# Typed so SQLAlchemy applies the dialect's DateTime bind processor
# (SQLite string format / Postgres timestamptz) instead of handing a
# raw datetime to the DBAPI (Python 3.12 dropped sqlite3's default
# datetime adapter).
sa.bindparam("finished_at", type_=sa.DateTime(timezone=True)),
)
bind.execute(
update_stmt,
{"error_message": superseded_message, "finished_at": datetime.now(UTC)},
)
def upgrade() -> None:
# Idempotent index creation: the legacy/empty bootstrap path runs
# create_all (which creates the index from the ORM __table_args__) before
# upgrade head, so the migration must not fail when the index already
# exists.
insp = sa.inspect(op.get_bind())
existing = {ix["name"] for ix in insp.get_indexes("scheduled_task_runs")}
if "uq_scheduled_task_run_active" not in existing:
# Supersede duplicate active rows first so the partial UNIQUE index can
# be built on DBs that already violate the invariant. No-op on clean
# DBs (the common path -- create_all already created the index, so this
# branch only runs on legacy DBs that pre-date the index).
_dedupe_active_scheduled_runs_per_task()
with op.batch_alter_table("scheduled_task_runs", schema=None) as batch_op:
batch_op.create_index(
"uq_scheduled_task_run_active",
["task_id"],
unique=True,
sqlite_where=sa.text("status IN ('queued', 'running')"),
postgresql_where=sa.text("status IN ('queued', 'running')"),
)
def downgrade() -> None:
bind = op.get_bind()
insp = sa.inspect(bind)
existing = {ix["name"] for ix in insp.get_indexes("scheduled_task_runs")}
if "uq_scheduled_task_run_active" in existing:
with op.batch_alter_table("scheduled_task_runs", schema=None) as batch_op:
batch_op.drop_index("uq_scheduled_task_run_active")

8
backend/packages/harness/deerflow/persistence/migrations/versions/0007_feedback_tags.py → backend/packages/harness/deerflow/persistence/migrations/versions/0008_feedback_tags.py
View File

@ -1,7 +1,7 @@
"""feedback tags.
Revision ID: 0007_feedback_tags
Revises: 0006_agents
Revision ID: 0008_feedback_tags
Revises: 0007_scheduled_run_active_index
Create Date: 2026-07-23
"""
@ -12,8 +12,8 @@ from collections.abc import Sequence
import sqlalchemy as sa
from alembic import op
revision: str = "0007_feedback_tags"
down_revision: str | Sequence[str] | None = "0006_agents"
revision: str = "0008_feedback_tags"
down_revision: str | Sequence[str] | None = "0007_scheduled_run_active_index"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None

10
backend/packages/harness/deerflow/persistence/run/sql.py
View File

@ -461,8 +461,16 @@ class RunRepository(RunStore):
*,
grace_seconds: int,
error: str,
stop_reason: str | None = None,
) -> bool:
cutoff = datetime.now(UTC) - timedelta(seconds=grace_seconds)
values: dict[str, Any] = {
"status": "error",
"error": error,
"updated_at": datetime.now(UTC),
}
if stop_reason is not None:
values["stop_reason"] = stop_reason
async with self._sf() as session:
result = await session.execute(
update(RunRow)
@ -471,7 +479,7 @@ class RunRepository(RunStore):
RunRow.status.in_(("pending", "running")),
_lease_expired_or_null(RunRow.lease_expires_at, cutoff),
)
.values(status="error", error=error, updated_at=datetime.now(UTC))
.values(**values)
)
await session.commit()
return result.rowcount != 0

4
backend/packages/harness/deerflow/persistence/scheduled_task_runs/__init__.py
View File

@ -1,4 +1,4 @@
from .model import ScheduledTaskRunRow
from .sql import ScheduledTaskRunRepository
from .sql import ActiveScheduledRunConflict, ScheduledTaskRunRepository
__all__ = ["ScheduledTaskRunRow", "ScheduledTaskRunRepository"]
__all__ = ["ActiveScheduledRunConflict", "ScheduledTaskRunRow", "ScheduledTaskRunRepository"]

31
backend/packages/harness/deerflow/persistence/scheduled_task_runs/model.py
View File

@ -2,7 +2,7 @@ from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import DateTime, String, Text
from sqlalchemy import DateTime, Index, String, Text, text
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
@ -22,3 +22,32 @@ class ScheduledTaskRunRow(Base):
started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC))
__table_args__ = (
# At most one active (queued/running) run per task. This is the atomic
# arbiter for the ``dispatch_task`` skip policy: the non-atomic
# ``has_active_runs`` check-then-create is a fast path, but two
# concurrent dispatches (double-click / client retry / manual trigger
# racing the poller) can both pass it, so the DB must reject the second
# active insert. Sibling of the ``runs`` table's ``uq_runs_thread_active``
# (PR #4003); that one keys on ``thread_id`` and does not cover the
# default ``fresh_thread_per_run`` context (every dispatch gets a new
# thread), which is why the scheduled-task run row needs its own guard.
#
# Condition is status-only, not ``overlap_policy``: the policy is fixed
# to "skip" in the MVP, so a status-only predicate enforces the current
# invariant without denormalizing ``overlap_policy`` onto the run row
# for an unimplemented non-skip policy. If a non-skip policy is added
# this must become conditional (e.g. ``... AND overlap_policy = 'skip'``).
#
# Must live in ORM ``__table_args__`` (not just the migration) because
# the empty-DB bootstrap path runs ``create_all`` + ``stamp head`` and
# never executes the migration that also defines this index.
Index(
"uq_scheduled_task_run_active",
"task_id",
unique=True,
sqlite_where=text("status IN ('queued', 'running')"),
postgresql_where=text("status IN ('queued', 'running')"),
),
)

34
backend/packages/harness/deerflow/persistence/scheduled_task_runs/sql.py
View File

@ -4,6 +4,7 @@ from datetime import UTC, datetime
from typing import Any
from sqlalchemy import func, select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from deerflow.persistence.scheduled_task_runs.model import ScheduledTaskRunRow
@ -13,6 +14,26 @@ TERMINAL_RUN_STATUSES: frozenset[str] = frozenset({"success", "failed", "skipped
ACTIVE_RUN_STATUSES: tuple[str, ...] = ("queued", "running")
class ActiveScheduledRunConflict(Exception):
"""A concurrent dispatch already holds the task's single active-run slot.
Raised by :meth:`ScheduledTaskRunRepository.create` when inserting an
active (queued/running) run row would violate the partial unique index
``uq_scheduled_task_run_active`` (at most one active run per ``task_id``).
This is the atomic counterpart to the non-atomic ``has_active_runs`` check
in ``ScheduledTaskService.dispatch_task``: two dispatches can both pass that
check, but only one can insert the active row — the loser lands here.
Translating the SQLAlchemy ``IntegrityError`` into a domain exception at
the repository boundary keeps the service layer free of ``sqlalchemy.exc``
coupling (mirrors ``deerflow.runtime.ConflictError`` for the runs table).
"""
def __init__(self, task_id: str) -> None:
self.task_id = task_id
super().__init__(f"scheduled task {task_id!r} already has an active run")
class ScheduledTaskRunRepository:
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
self._sf = session_factory
@ -46,7 +67,18 @@ class ScheduledTaskRunRepository:
)
async with self._sf() as session:
session.add(row)
await session.commit()
try:
await session.commit()
except IntegrityError:
await session.rollback()
# Only active-status inserts can trip the partial unique index
# ``uq_scheduled_task_run_active``; a terminal-status row (e.g.
# a "skipped" tombstone) is outside its predicate and cannot
# conflict, so any IntegrityError there is a genuine fault and
# is re-raised untranslated.
if status in ACTIVE_RUN_STATUSES:
raise ActiveScheduledRunConflict(task_id) from None
raise
await session.refresh(row)
return self._row_to_dict(row)

4
backend/packages/harness/deerflow/runtime/__init__.py
View File

@ -7,7 +7,7 @@ directly from ``deerflow.runtime``.
from .checkpoint_state import CheckpointStateAccessor, build_state_mutation_graph
from .checkpointer import checkpointer_context, get_checkpointer, make_checkpointer, reset_checkpointer
from .runs import CancelOutcome, ConflictError, DisconnectMode, RunContext, RunManager, RunRecord, RunStatus, UnsupportedStrategyError, run_agent
from .runs import ORPHAN_RECOVERY_STOP_REASON, STARTUP_ORPHAN_RECOVERY_ERROR, CancelOutcome, ConflictError, DisconnectMode, RunContext, RunManager, RunRecord, RunStatus, UnsupportedStrategyError, run_agent
from .serialization import serialize, serialize_channel_values, serialize_channel_values_for_api, serialize_lc_object, serialize_messages_tuple, strip_data_url_image_blocks
from .store import get_store, make_store, reset_store, store_context
@ -29,10 +29,12 @@ __all__ = [
"CancelOutcome",
"ConflictError",
"DisconnectMode",
"ORPHAN_RECOVERY_STOP_REASON",
"RunContext",
"RunManager",
"RunRecord",
"RunStatus",
"STARTUP_ORPHAN_RECOVERY_ERROR",
"UnsupportedStrategyError",
"run_agent",
# serialization

4
backend/packages/harness/deerflow/runtime/runs/__init__.py
View File

@ -1,6 +1,6 @@
"""Run lifecycle management for LangGraph Platform API compatibility."""
from .manager import CancelOutcome, ConflictError, RunManager, RunRecord, UnsupportedStrategyError
from .manager import ORPHAN_RECOVERY_STOP_REASON, STARTUP_ORPHAN_RECOVERY_ERROR, CancelOutcome, ConflictError, RunManager, RunRecord, UnsupportedStrategyError
from .schemas import DisconnectMode, RunStatus
from .worker import RunContext, run_agent
@ -8,10 +8,12 @@ __all__ = [
"CancelOutcome",
"ConflictError",
"DisconnectMode",
"ORPHAN_RECOVERY_STOP_REASON",
"RunContext",
"RunManager",
"RunRecord",
"RunStatus",
"STARTUP_ORPHAN_RECOVERY_ERROR",
"UnsupportedStrategyError",
"run_agent",
]

110
backend/packages/harness/deerflow/runtime/runs/manager.py
View File

@ -27,6 +27,10 @@ if TYPE_CHECKING:
logger = logging.getLogger(__name__)
ORPHAN_RECOVERY_STOP_REASON = "orphan_recovered"
STARTUP_ORPHAN_RECOVERY_ERROR = "Gateway restarted before this run reached a durable final state."
LEASE_ORPHAN_RECOVERY_ERROR = "Run lease expired — owning worker is unreachable."
_RETRYABLE_SQLITE_MESSAGES = (
"database is locked",
"database table is locked",
@ -184,6 +188,9 @@ class RunRecord:
stop_reason: str | None = None
OrphanRecoveryCallback = Callable[[list[RunRecord]], Awaitable[None]]
class RunManager:
"""In-memory run registry with optional persistent RunStore backing.
@ -199,6 +206,7 @@ class RunManager:
persistence_retry_policy: PersistenceRetryPolicy | None = None,
worker_id: str | None = None,
run_ownership_config: RunOwnershipConfig | None = None,
on_orphans_recovered: OrphanRecoveryCallback | None = None,
) -> None:
self._runs: dict[str, RunRecord] = {}
# Secondary index: thread_id -> insertion-ordered run_id set (a dict is
@ -211,8 +219,10 @@ class RunManager:
self._persistence_retry_policy = persistence_retry_policy or PersistenceRetryPolicy()
self._worker_id = worker_id or _generate_worker_id()
self._run_ownership_config = run_ownership_config
self._on_orphans_recovered = on_orphans_recovered
self._heartbeat_task: asyncio.Task | None = None
self._heartbeat_stop: asyncio.Event | None = None
self._orphan_recovery_task: asyncio.Task[None] | None = None
def _index_run_locked(self, record: RunRecord) -> None:
"""Register *record* in the thread index. Caller must hold ``self._lock``."""
@ -580,7 +590,12 @@ class RunManager:
if self._store is None:
return sorted(memory_records, key=lambda r: r.created_at, reverse=True)[:limit]
records_by_id = {record.run_id: record for record in memory_records}
store_limit = max(0, limit - len(memory_records))
# Query enough rows to cover both the requested page and every possible
# in-memory/store duplicate. Local records can be older than persisted
# rows, so subtracting them from the store limit can hide the actual
# newest run before the merge; querying only ``limit`` can still lose a
# distinct row when that page is occupied by duplicate local records.
store_limit = limit + len(memory_records)
try:
rows = await self._store.list_by_thread(thread_id, user_id=user_id, limit=store_limit)
except Exception:
@ -1090,6 +1105,7 @@ class RunManager:
*,
error: str,
before: str | None = None,
stop_reason: str | None = None,
) -> list[RunRecord]:
"""Mark persisted active runs as failed when their lease has expired.
@ -1138,6 +1154,7 @@ class RunManager:
record.run_id,
grace_seconds=grace_seconds,
error=error,
stop_reason=stop_reason,
),
)
except Exception:
@ -1151,6 +1168,7 @@ class RunManager:
continue
record.status = RunStatus.error
record.error = error
record.stop_reason = stop_reason
record.updated_at = now
recovered.append(record)
@ -1215,21 +1233,21 @@ class RunManager:
self._heartbeat_task = task
logger.info("Run lease heartbeat started for worker %s", self._worker_id)
async def stop_heartbeat(self) -> None:
"""Stop the background heartbeat task."""
async def stop_heartbeat(self, *, timeout: float = 5.0) -> None:
"""Stop the background heartbeat task within ``timeout`` seconds."""
if self._heartbeat_stop is not None:
self._heartbeat_stop.set()
if self._heartbeat_task is not None and not self._heartbeat_task.done():
try:
await asyncio.wait_for(self._heartbeat_task, timeout=5.0)
except TimeoutError:
_, pending = await asyncio.wait(
(self._heartbeat_task,),
timeout=max(0.0, timeout),
)
if pending:
self._heartbeat_task.cancel()
try:
await self._heartbeat_task
except asyncio.CancelledError:
pass
except asyncio.CancelledError:
pass
self._heartbeat_task = None
self._heartbeat_stop = None
logger.info("Run lease heartbeat stopped for worker %s", self._worker_id)
@ -1273,10 +1291,7 @@ class RunManager:
# replacement starts before the lease expires, and the
# startup pass skips the still-valid lease.
if cycle % 3 == 0:
try:
await self._reconcile_orphans_periodic()
except Exception:
logger.warning("Periodic orphan reconciliation failed", exc_info=True)
self._schedule_orphan_reconciliation()
async def _renew_leases(self) -> None:
"""Renew the lease on every locally-owned active run."""
@ -1339,22 +1354,74 @@ class RunManager:
async def _reconcile_orphans_periodic(self) -> None:
"""Sweep for expired leases owned by dead peers.
Called from ``_heartbeat_loop`` every ``lease_seconds``. Startup
reconciliation handles the initial sweep; this periodic pass
catches orphans whose lease expires between restarts.
Scheduled as a single-flight background task by ``_heartbeat_loop``.
This keeps both the store scan/status writes and the Gateway callback
off the lease-renewal loop. Startup reconciliation handles the initial
sweep; this periodic pass catches orphans whose lease expires between
restarts.
"""
error_msg = "Run lease expired — owning worker is unreachable."
recovered = await self.reconcile_orphaned_inflight_runs(error=error_msg)
recovered = await self.reconcile_orphaned_inflight_runs(
error=LEASE_ORPHAN_RECOVERY_ERROR,
stop_reason=ORPHAN_RECOVERY_STOP_REASON,
)
if recovered:
logger.warning(
"Periodic reconciliation recovered %d orphaned run(s) as error",
len(recovered),
)
if self._on_orphans_recovered is not None:
try:
await self._on_orphans_recovered(recovered)
except Exception:
logger.warning(
"Periodic orphan recovery callback failed for %d run(s): run_ids=%s",
len(recovered),
[record.run_id for record in recovered],
exc_info=True,
)
def _schedule_orphan_reconciliation(self) -> None:
"""Start one supervised recovery pass unless one is already running."""
task = self._orphan_recovery_task
if task is not None and not task.done():
logger.debug("Skipping periodic orphan reconciliation: previous pass is still running")
return
task = asyncio.create_task(self._reconcile_orphans_periodic())
task.set_name("deerflow-periodic-orphan-recovery")
self._orphan_recovery_task = task
task.add_done_callback(self._orphan_reconciliation_done)
def _orphan_reconciliation_done(self, task: asyncio.Task[None]) -> None:
"""Clear and inspect the supervised single-flight recovery task."""
if self._orphan_recovery_task is task:
self._orphan_recovery_task = None
if task.cancelled():
return
try:
task.result()
except Exception:
logger.warning("Periodic orphan reconciliation failed", exc_info=True)
async def _drain_orphan_recovery_task(self, *, timeout: float) -> None:
"""Boundedly await the supervised recovery pass during shutdown."""
task = self._orphan_recovery_task
if task is None or task.done():
return
_, pending = await asyncio.wait((task,), timeout=max(0.0, timeout))
if pending:
task.cancel()
await asyncio.gather(task, return_exceptions=True)
logger.warning(
"Orphan recovery drain exceeded %.1fs on shutdown; cancelled the active pass",
timeout,
)
async def shutdown(self, *, timeout: float = 5.0) -> None:
"""Cancel and bounded-await all in-flight runs on process shutdown.
Stops the lease heartbeat first so no renewal races against the drain.
Signals active runs first so their cancellation/cleanup can overlap a
bounded heartbeat stop. The heartbeat may perform one final benign
lease renewal before it observes the stop event.
Chat runs execute in fire-and-forget background ``asyncio`` tasks that
write checkpoints through a shared checkpointer. On shutdown the
@ -1380,7 +1447,6 @@ class RunManager:
``app.gateway.app._SHUTDOWN_HOOK_TIMEOUT_SECONDS``. Runs still active
after ``timeout`` are logged and may still race teardown.
"""
await self.stop_heartbeat()
loop = asyncio.get_running_loop()
deadline = loop.time() + timeout
@ -1393,11 +1459,14 @@ class RunManager:
# Status is decided AFTER the drain (below), not here: a run that
# completes on its own during the drain must keep its real status.
await self.stop_heartbeat(timeout=max(0.0, deadline - loop.time()))
if not inflight:
await self._drain_orphan_recovery_task(timeout=max(0.0, deadline - loop.time()))
return
tasks = [record.task for record in inflight]
_, pending = await asyncio.wait(tasks, timeout=timeout)
_, pending = await asyncio.wait(tasks, timeout=max(0.0, deadline - loop.time()))
# Only mark/persist ``interrupted`` for runs that did not settle on their
# own (still pending after the timeout, or ended cancelled). A run that
@ -1445,6 +1514,7 @@ class RunManager:
if pending:
logger.warning("Run drain exceeded %.1fs on shutdown; %d run task(s) still active and may race checkpointer teardown", timeout, len(pending))
logger.info("Drained %d in-flight run(s) on shutdown (%d settled within %.1fs)", len(inflight), len(inflight) - len(pending), timeout)
await self._drain_orphan_recovery_task(timeout=max(0.0, deadline - loop.time()))
class CancelOutcome(StrEnum):

4
backend/packages/harness/deerflow/runtime/runs/store/base.py
View File

@ -188,13 +188,15 @@ class RunStore(abc.ABC):
*,
grace_seconds: int,
error: str,
stop_reason: str | None = None,
) -> bool:
"""Atomically mark an expired-lease active run as ``error``.
Only rows whose lease has expired past *grace_seconds* (or whose
lease is NULL — pre-ownership data) are updated. The conditional
WHERE closes the race between the caller's stale read of the lease
and a concurrent heartbeat renewal by the owning worker.
and a concurrent heartbeat renewal by the owning worker. When
provided, *stop_reason* is persisted in the same atomic update.
Returns ``False`` when:
- the run is no longer ``pending`` / ``running``,

3
backend/packages/harness/deerflow/runtime/runs/store/memory.py
View File

@ -227,6 +227,7 @@ class MemoryRunStore(RunStore):
*,
grace_seconds: int,
error: str,
stop_reason: str | None = None,
) -> bool:
from deerflow.utils.time import is_lease_expired
@ -240,6 +241,8 @@ class MemoryRunStore(RunStore):
return False
run["status"] = "error"
run["error"] = error
if stop_reason is not None:
run["stop_reason"] = stop_reason
run["updated_at"] = datetime.now(UTC).isoformat()
return True

40
backend/packages/harness/deerflow/runtime/runs/worker.py
View File

@ -7,7 +7,7 @@ Uses ``graph.astream(stream_mode=[...])`` which gives correct full-state
snapshots for ``values`` mode, proper ``{node: writes}`` for ``updates``,
and ``(chunk, metadata)`` tuples for ``messages`` mode.
Note: ``events`` mode is not supported through the gateway — it requires
Note: ``events`` mode is rejected by the gateway — it requires
``graph.astream_events()`` which cannot simultaneously produce ``values``
snapshots. The JS open-source LangGraph API server works around this via
internal checkpoint callbacks that are not exposed in the Python public API.
@ -63,6 +63,7 @@ from deerflow.runtime.goal import (
)
from deerflow.runtime.serialization import serialize
from deerflow.runtime.stream_bridge import StreamBridge
from deerflow.runtime.stream_modes import normalize_stream_modes, to_langgraph_stream_modes
from deerflow.runtime.user_context import get_effective_user_id, resolve_runtime_user_id
from deerflow.trace_context import (
DEERFLOW_TRACE_METADATA_KEY,
@ -102,8 +103,6 @@ async def _checkpoint_thread_lock(thread_id: str) -> AsyncIterator[None]:
yield
# Valid stream_mode values for LangGraph's graph.astream()
_VALID_LG_MODES = {"values", "updates", "checkpoints", "tasks", "debug", "messages", "custom"}
# Keep this streaming policy separate from middleware write-authorization sets.
_LARGE_FILE_TOOL_NAMES = frozenset({"str_replace", "write_file"})
_LARGE_FILE_TOOL_BATCH_SIZE = 32
@ -397,7 +396,6 @@ async def run_agent(
run_id = record.run_id
thread_id = record.thread_id
requested_modes: set[str] = set(stream_modes or ["values"])
pre_run_checkpoint_id: str | None = None
pre_run_workspace_snapshot: WorkspaceSnapshot | None = None
workspace_changes_user_id: str | None = None
@ -420,14 +418,10 @@ async def run_agent(
# finally is safe even if an exception fires before streaming begins.
subagent_events: _SubagentEventBuffer | None = None
# Track whether "events" was requested but skipped
if "events" in requested_modes:
logger.info(
"Run %s: 'events' stream_mode not supported in gateway (requires astream_events + checkpoint callbacks). Skipping.",
run_id,
)
try:
normalized_stream_modes = normalize_stream_modes(stream_modes)
requested_modes: set[str] = set(normalized_stream_modes)
lg_modes = to_langgraph_stream_modes(normalized_stream_modes)
await run_manager.wait_for_prior_finalizing(thread_id, run_id)
mode = ctx.checkpoint_channel_mode
inject_checkpoint_mode(config, mode)
@ -616,30 +610,6 @@ async def run_agent(
if interrupt_after:
agent.interrupt_after_nodes = interrupt_after
# 6. Build LangGraph stream_mode list
# "events" is NOT a valid astream mode — skip it
# "messages-tuple" maps to LangGraph's "messages" mode
lg_modes: list[str] = []
for m in requested_modes:
if m == "messages-tuple":
lg_modes.append("messages")
elif m == "events":
# Skipped — see log above
continue
elif m in _VALID_LG_MODES:
lg_modes.append(m)
if not lg_modes:
lg_modes = ["values"]
# Deduplicate while preserving order
seen: set[str] = set()
deduped: list[str] = []
for m in lg_modes:
if m not in seen:
seen.add(m)
deduped.append(m)
lg_modes = deduped
logger.info("Run %s: streaming with modes %s (requested: %s)", run_id, lg_modes, requested_modes)
# Buffer subagent step events and persist them in batches (#3779) instead

47
backend/packages/harness/deerflow/runtime/stream_modes.py Normal file
View File

@ -0,0 +1,47 @@
"""Supported stream modes for the LangGraph-compatible runtime boundary."""
from __future__ import annotations
from typing import Literal, get_args
type RunStreamMode = Literal[
"values",
"messages-tuple",
"updates",
"debug",
"tasks",
"checkpoints",
"custom",
]
SUPPORTED_RUN_STREAM_MODES: frozenset[str] = frozenset(get_args(RunStreamMode.__value__))
class UnsupportedStreamModeError(ValueError):
"""Raised when a caller requests a stream mode DeerFlow cannot honor."""
def __init__(self, modes: list[str]) -> None:
self.modes = tuple(dict.fromkeys(modes))
super().__init__(f"Unsupported stream mode(s): {', '.join(self.modes)}")
def normalize_stream_modes(raw: list[str] | str | None) -> list[str]:
"""Normalize and validate public run stream modes."""
if raw is None:
modes = ["values"]
elif isinstance(raw, str):
modes = [raw]
else:
modes = raw or ["values"]
unsupported = [mode if isinstance(mode, str) else type(mode).__name__ for mode in modes if not isinstance(mode, str) or mode not in SUPPORTED_RUN_STREAM_MODES]
if unsupported:
raise UnsupportedStreamModeError(unsupported)
return modes
def to_langgraph_stream_modes(raw: list[str] | str | None) -> list[str]:
"""Map public run modes to ``graph.astream`` modes without silent fallback."""
modes = normalize_stream_modes(raw)
mapped = ["messages" if mode == "messages-tuple" else mode for mode in modes]
return list(dict.fromkeys(mapped))

42
backend/packages/harness/deerflow/sandbox/exceptions.py
View File

@ -69,3 +69,45 @@ class SandboxFileNotFoundError(SandboxFileError):
"""Raised when a file or directory is not found."""
pass
class SandboxCapacityExceededError(SandboxError):
"""Raised when the sandbox provider has no available capacity.
The reason distinguishes occupied capacity from provider shutdown.
The caller controls retry scheduling. DeerFlow does not retry automatically.
"""
CODE = "SANDBOX_CAPACITY_EXCEEDED"
def __init__(
self,
message: str = "All sandbox replica slots are in use",
*,
active: int = 0,
warm: int = 0,
reserved: int = 0,
replicas: int = 0,
retry_after_seconds: float = 5.0,
reason: str = "capacity",
) -> None:
details: dict[str, object] = {
"code": self.CODE,
"reason": reason,
"replicas": replicas,
"retryable": True,
"retry_after_seconds": retry_after_seconds,
}
if active:
details["active"] = active
if warm:
details["warm"] = warm
if reserved:
details["reserved"] = reserved
super().__init__(message, details)
self.active = active
self.warm = warm
self.reserved = reserved
self.replicas = replicas
self.retry_after_seconds = retry_after_seconds
self.reason = reason

11
backend/packages/harness/deerflow/sandbox/sandbox_provider.py
View File

@ -51,7 +51,10 @@ class SandboxProvider(ABC):
pass
def reset(self) -> None:
"""Clear cached state that survives provider instance replacement."""
"""Clear cached state that survives provider instance replacement.
Provider overrides can release resources and make the instance unusable.
"""
pass
@ -115,7 +118,7 @@ def get_sandbox_provider(**kwargs) -> SandboxProvider:
def reset_sandbox_provider() -> None:
"""Reset the sandbox provider singleton.
This clears the cached instance without calling shutdown.
This clears the cached instance without calling shutdown directly.
The next call to `get_sandbox_provider()` will create a new instance.
Useful for testing or when switching configurations.
@ -124,7 +127,9 @@ def reset_sandbox_provider() -> None:
`LocalSandbox` singleton). Without it, config/mount changes would not take
effect on the next acquire().
Note: If the provider has active sandboxes, they will be orphaned.
A provider override can release active sandboxes during reset.
Otherwise, active sandboxes become orphaned.
Do not reuse the detached provider after reset.
Use `shutdown_sandbox_provider()` for proper cleanup.
"""
global _default_sandbox_provider

4
backend/packages/harness/deerflow/skills/frontmatter.py
View File

@ -57,6 +57,10 @@ def split_skill_markdown(content: str) -> tuple[SkillMarkdownParts | None, str |
if not isinstance(metadata, dict):
return None, "Frontmatter must be a YAML dictionary"
# YAML permits non-string keys, but downstream validation expects field
# names to be strings.
metadata = {str(key): value for key, value in metadata.items()}
return (
SkillMarkdownParts(
metadata=metadata,

5
backend/packages/harness/deerflow/tools/builtins/view_image_tool.py
View File

@ -22,6 +22,7 @@ _EXTENSION_TO_MIME = {
".jpeg": "image/jpeg",
".png": "image/png",
".webp": "image/webp",
".gif": "image/gif",
}
@ -36,6 +37,8 @@ def _detect_image_mime(image_data: bytes) -> str | None:
return "image/png"
if len(image_data) >= 12 and image_data.startswith(b"RIFF") and image_data[8:12] == b"WEBP":
return "image/webp"
if image_data.startswith((b"GIF87a", b"GIF89a")):
return "image/gif"
return None
@ -63,7 +66,7 @@ def view_image_tool(
- For multiple files at once (use present_files instead)
Args:
image_path: Absolute /mnt/user-data virtual path to the image file. Common formats supported: jpg, jpeg, png, webp.
image_path: Absolute /mnt/user-data virtual path to the image file. Common formats supported: jpg, jpeg, png, webp, gif.
"""
from deerflow.sandbox.exceptions import SandboxRuntimeError
from deerflow.sandbox.tools import (

65
backend/tests/blocking_io/test_skills_router.py Normal file
View File

@ -0,0 +1,65 @@
"""Regression anchor: get_custom_skill_history must not block the event loop.
``app.gateway.routers.skills.get_custom_skill_history`` is an async route handler
that probes custom-skill storage (``custom_skill_exists`` /
``get_skill_history_file().exists()``) and reads the per-skill ``.history`` file —
all blocking filesystem IO. It offloads that work via ``asyncio.to_thread``; if it
regresses back onto the event loop, the strict Blockbuster gate raises
``BlockingError`` and this test fails.
Seeding the history on disk is itself offloaded with ``asyncio.to_thread`` so only
the handler's own filesystem access is exercised on the loop.
"""
from __future__ import annotations
import asyncio
import json
from pathlib import Path
from types import SimpleNamespace
from uuid import UUID
import pytest
from fastapi import Request
from app.gateway.routers.skills import _get_user_skill_storage, get_custom_skill_history
pytestmark = pytest.mark.asyncio
def _config(skills_root: Path) -> SimpleNamespace:
return SimpleNamespace(
skills=SimpleNamespace(
get_skills_path=lambda: skills_root,
container_path="/mnt/skills",
use="deerflow.skills.storage.local_skill_storage:LocalSkillStorage",
)
)
def _admin_request() -> Request:
# The route is admin-only. ``AuthMiddleware`` normally stamps
# ``request.state.user``; supply it directly here, as
# ``test_channel_runtime_config_store`` does for the same reason.
user = SimpleNamespace(id=UUID("11111111-2222-3333-4444-555555555555"), system_role="admin")
return Request({"type": "http", "headers": [], "state": {"user": user}})
async def test_get_custom_skill_history_does_not_block_event_loop(tmp_path: Path) -> None:
config = _config(tmp_path / "skills")
def _seed() -> None:
# Seed through the same user-scoped accessor the handler uses so both
# resolve the same storage root. An existing history file is enough for
# the handler to skip the 404 branch and read it.
history_file = _get_user_skill_storage(config).get_skill_history_file("demo-skill")
history_file.parent.mkdir(parents=True, exist_ok=True)
history_file.write_text(json.dumps({"action": "human_edit", "new_content": "x"}) + "\n", encoding="utf-8")
request = await asyncio.to_thread(_admin_request)
await asyncio.to_thread(_seed)
response = await get_custom_skill_history("demo-skill", request, config)
assert len(response.history) == 1
assert response.history[-1]["action"] == "human_edit"

676
backend/tests/test_channels.py
View File

@ -1360,12 +1360,21 @@ class TestChannelManager:
_run(go())
def test_fire_and_forget_thread_busy_releases_dedupe_key(self, tmp_path):
"""Same invariant on the third swallow site: runs.create's busy branch."""
"""Same invariant for a fire-and-forget channel that does not buffer follow-ups."""
import httpx
from langgraph_sdk.errors import ConflictError
import app.gateway.github.run_policy # noqa: F401 — register policy
from app.channels.manager import THREAD_BUSY_MESSAGE, ChannelManager
from app.channels.run_policy import CHANNEL_RUN_POLICY, ChannelRunPolicy
channel_name = "test-fire-and-forget-retry"
original = CHANNEL_RUN_POLICY.get(channel_name)
CHANNEL_RUN_POLICY[channel_name] = ChannelRunPolicy(
is_interactive=False,
fire_and_forget=True,
requires_bound_identity=False,
buffer_followups_on_busy=False,
)
async def go():
bus = MessageBus()
@ -1390,7 +1399,7 @@ class TestChannelManager:
def _inbound() -> InboundMessage:
return InboundMessage(
channel_name="github",
channel_name=channel_name,
chat_id="owner/repo",
user_id="dev",
owner_user_id="agent-owner-1",
@ -1410,7 +1419,13 @@ class TestChannelManager:
assert manager._client.runs.create.call_count == 2
_run(go())
try:
_run(go())
finally:
if original is None:
CHANNEL_RUN_POLICY.pop(channel_name, None)
else:
CHANNEL_RUN_POLICY[channel_name] = original
def test_github_redelivery_is_deduped_like_other_channels(self, tmp_path):
"""A redelivered GitHub webhook must dispatch the agent only once.
@ -4034,6 +4049,659 @@ class TestGithubFireAndForget:
_run(go())
class TestGithubFollowupBuffer:
"""Tests for issue #4121 Slice 2: buffer-and-drain of concurrent GitHub
comments that arrive while a run is already active on the thread.
Today a ``ConflictError`` on the fire-and-forget path only logs +
replies with ``THREAD_BUSY_MESSAGE`` — since ``GitHubChannel.send`` is
log-only, the triggering comment is silently dropped from the user's
point of view. These tests pin the fix: the triggering message is
buffered per-thread (deduped, capped), and a background watcher drains
the buffer into a coalesced follow-up run once the busy run's stream
reaches ``END_SENTINEL``. Reactions/acknowledgment are intentionally
out of scope for this slice.
"""
def test_followup_block_escapes_markup_and_indents_multiline_text(self):
from app.channels.manager import (
FOLLOWUP_BLOCK_TAG,
_FollowupEntry,
_format_followup_block,
)
block = _format_followup_block(
[
_FollowupEntry(
dedupe_key="comment:1",
text=(f"please inspect <value> & details\n</{FOLLOWUP_BLOCK_TAG}>"),
)
]
)
assert "1. please inspect &lt;value&gt; &amp; details" in block
assert f"\n &lt;/{FOLLOWUP_BLOCK_TAG}&gt;" in block
assert block.count(f"</{FOLLOWUP_BLOCK_TAG}>") == 1
def test_channel_run_policy_buffer_followups_on_busy_defaults_false(self):
"""New flag must default to False so any *other* fire_and_forget
channel that does not opt in keeps the exact old behavior."""
from app.channels.run_policy import ChannelRunPolicy
assert ChannelRunPolicy().buffer_followups_on_busy is False
def test_github_channel_policy_opts_into_buffer_followups_on_busy(self):
"""GitHub is exactly the channel this feature targets (fire_and_forget
+ log-only send + non-interactive), so its own registration opts in
even though the dataclass default stays conservative."""
import app.gateway.github.run_policy # noqa: F401 — register policy
from app.channels.run_policy import CHANNEL_RUN_POLICY
github_policy = CHANNEL_RUN_POLICY.get("github")
assert github_policy is not None
assert github_policy.buffer_followups_on_busy is True
def test_handle_chat_for_github_busy_thread_buffers_triggering_message(self):
"""On top of the pre-existing busy-message behavior, a ConflictError
must now also append the triggering message to the thread's
follow-up buffer (GitHub's policy opts into buffer_followups_on_busy)."""
import httpx
from langgraph_sdk.errors import ConflictError
import app.gateway.github.run_policy # noqa: F401 — register policy
from app.channels.manager import THREAD_BUSY_MESSAGE, ChannelManager
async def go():
bus = MessageBus()
store = ChannelStore(path=Path(tempfile.mkdtemp()) / "store.json")
manager = ChannelManager(bus=bus, store=store)
outbound_received: list[OutboundMessage] = []
async def capture_outbound(msg):
outbound_received.append(msg)
bus.subscribe_outbound(capture_outbound)
request = httpx.Request("POST", "http://127.0.0.1:2024/threads/gh-thread-buf/runs")
response = httpx.Response(409, request=request)
conflict = ConflictError(
"Thread is already running a task.",
response=response,
body={"message": "Thread is already running a task."},
)
mock_client = _make_mock_langgraph_client(thread_id="gh-thread-buf")
mock_client.runs.create = AsyncMock(side_effect=conflict)
manager._client = mock_client
await manager.start()
try:
await manager._handle_chat(
InboundMessage(
channel_name="github",
chat_id="zhfeng/llm-gateway",
user_id="zhfeng",
owner_user_id="agent-owner-1",
text="please also update the README",
metadata={"github": {"delivery_id": "delivery-buf-1"}},
)
)
await _wait_for(lambda: any(m.text == THREAD_BUSY_MESSAGE for m in outbound_received))
finally:
await manager.stop()
assert "gh-thread-buf" in manager._followup_buffers
buffered = list(manager._followup_buffers["gh-thread-buf"].values())
assert len(buffered) == 1
assert buffered[0].text == "please also update the README"
_run(go())
def test_conflict_error_does_not_buffer_when_flag_disabled(self):
"""A fire_and_forget channel that has NOT opted into
buffer_followups_on_busy must keep the exact old silent-drop-with-log
behavior: busy message still emitted, nothing buffered."""
import httpx
from langgraph_sdk.errors import ConflictError
from app.channels.manager import THREAD_BUSY_MESSAGE, ChannelManager
from app.channels.run_policy import CHANNEL_RUN_POLICY, ChannelRunPolicy
original = CHANNEL_RUN_POLICY.get("test-fire-and-forget-no-buffer")
CHANNEL_RUN_POLICY["test-fire-and-forget-no-buffer"] = ChannelRunPolicy(
is_interactive=False,
fire_and_forget=True,
requires_bound_identity=False,
buffer_followups_on_busy=False,
)
try:
async def go():
bus = MessageBus()
store = ChannelStore(path=Path(tempfile.mkdtemp()) / "store.json")
manager = ChannelManager(bus=bus, store=store)
outbound_received: list[OutboundMessage] = []
async def capture_outbound(msg):
outbound_received.append(msg)
bus.subscribe_outbound(capture_outbound)
request = httpx.Request("POST", "http://127.0.0.1:2024/threads/no-buf-thread/runs")
response = httpx.Response(409, request=request)
conflict = ConflictError("busy", response=response, body={"message": "busy"})
mock_client = _make_mock_langgraph_client(thread_id="no-buf-thread")
mock_client.runs.create = AsyncMock(side_effect=conflict)
manager._client = mock_client
await manager.start()
try:
await manager._handle_chat(
InboundMessage(
channel_name="test-fire-and-forget-no-buffer",
chat_id="c1",
user_id="u1",
text="hello while busy",
)
)
await _wait_for(lambda: any(m.text == THREAD_BUSY_MESSAGE for m in outbound_received))
finally:
await manager.stop()
assert manager._followup_buffers == {}
_run(go())
finally:
if original is None:
CHANNEL_RUN_POLICY.pop("test-fire-and-forget-no-buffer", None)
else:
CHANNEL_RUN_POLICY["test-fire-and-forget-no-buffer"] = original
def test_duplicate_delivery_id_does_not_double_buffer(self):
"""A redelivered webhook for the same comment (same delivery_id) must
not be buffered twice."""
from app.channels.manager import ChannelManager
manager = ChannelManager(bus=MessageBus(), store=ChannelStore(path=Path(tempfile.mkdtemp()) / "store.json"))
thread_id = "gh-thread-dedup"
msg = InboundMessage(
channel_name="github",
chat_id="c",
user_id="u",
text="please look at this",
metadata={"github": {"delivery_id": "dupe-1"}},
)
manager._buffer_followup(thread_id, msg)
manager._buffer_followup(thread_id, msg) # redelivery of the same comment
assert len(manager._followup_buffers[thread_id]) == 1
def test_followup_buffer_overflow_drops_oldest_and_warns(self, caplog):
"""At the per-thread cap, overflow must drop the OLDEST buffered
comment (not the newest) and log a warning — recent activity is a
more useful signal than the stalest queued item once a thread is
deep enough in the backlog to hit the cap."""
from app.channels.manager import FOLLOWUP_BUFFER_MAX_PER_THREAD, ChannelManager
manager = ChannelManager(bus=MessageBus(), store=ChannelStore(path=Path(tempfile.mkdtemp()) / "store.json"))
thread_id = "gh-thread-overflow"
with caplog.at_level(logging.WARNING):
for i in range(FOLLOWUP_BUFFER_MAX_PER_THREAD + 5):
msg = InboundMessage(
channel_name="github",
chat_id="c",
user_id="u",
text=f"comment {i}",
metadata={"github": {"delivery_id": f"d{i}"}},
)
manager._buffer_followup(thread_id, msg)
buffer = manager._followup_buffers[thread_id]
assert len(buffer) == FOLLOWUP_BUFFER_MAX_PER_THREAD
kept_texts = {entry.text for entry in buffer.values()}
for i in range(5):
assert f"comment {i}" not in kept_texts
assert f"comment {FOLLOWUP_BUFFER_MAX_PER_THREAD + 4}" in kept_texts
assert any("overflow" in r.message.lower() for r in caplog.records)
def test_drain_batches_at_most_ten_entries_per_cycle(self):
"""A queue deeper than the drain batch size must only coalesce the
oldest N entries in one cycle, leaving the rest buffered — this is
what lets a >10 backlog chain into a second drain cycle instead of
growing one unbounded input block."""
from app.channels.manager import FOLLOWUP_DRAIN_BATCH_SIZE, ChannelManager
async def go():
bus = MessageBus()
store = ChannelStore(path=Path(tempfile.mkdtemp()) / "store.json")
manager = ChannelManager(bus=bus, store=store)
carrier_msg = InboundMessage(
channel_name="github",
chat_id="zhfeng/llm-gateway",
user_id="zhfeng",
owner_user_id="agent-owner-1",
text="carrier",
)
thread_id = "gh-thread-batch"
for i in range(15):
entry_msg = InboundMessage(
channel_name="github",
chat_id="zhfeng/llm-gateway",
user_id="zhfeng",
owner_user_id="agent-owner-1",
text=f"comment {i}",
metadata={"github": {"delivery_id": f"d{i}"}},
)
manager._buffer_followup(thread_id, entry_msg)
assert len(manager._followup_buffers[thread_id]) == 15
mock_client = _make_mock_langgraph_client(thread_id=thread_id)
mock_client.runs.create = AsyncMock(return_value={"run_id": "run-drain-1", "status": "pending"})
await manager._drain_followups_for_thread(mock_client, thread_id, carrier_msg)
mock_client.runs.create.assert_called_once()
drained_text = mock_client.runs.create.call_args[1]["input"]["messages"][0]["content"]
for i in range(FOLLOWUP_DRAIN_BATCH_SIZE):
assert f"comment {i}" in drained_text
for i in range(FOLLOWUP_DRAIN_BATCH_SIZE, 15):
assert f"comment {i}" not in drained_text
assert len(manager._followup_buffers[thread_id]) == 15 - FOLLOWUP_DRAIN_BATCH_SIZE
_run(go())
def test_drain_conflict_requeues_entries_without_losing_them(self):
"""If the drain's own runs.create hits ConflictError (a real edge
case — e.g. a manual/scheduled trigger raced onto the same thread),
the popped batch must be put back rather than lost, and the drain
must not raise."""
import httpx
from langgraph_sdk.errors import ConflictError
from app.channels.manager import ChannelManager
async def go():
bus = MessageBus()
store = ChannelStore(path=Path(tempfile.mkdtemp()) / "store.json")
manager = ChannelManager(bus=bus, store=store)
carrier_msg = InboundMessage(
channel_name="github",
chat_id="zhfeng/llm-gateway",
user_id="zhfeng",
owner_user_id="agent-owner-1",
text="queued comment",
)
thread_id = "gh-thread-9"
manager._buffer_followup(thread_id, carrier_msg)
assert len(manager._followup_buffers[thread_id]) == 1
request = httpx.Request("POST", "http://127.0.0.1:2024/threads/gh-thread-9/runs")
response = httpx.Response(409, request=request)
conflict = ConflictError("busy", response=response, body={"message": "busy"})
mock_client = MagicMock()
mock_client.runs.create = AsyncMock(side_effect=conflict)
# Must not raise.
await manager._drain_followups_for_thread(mock_client, thread_id, carrier_msg)
assert thread_id in manager._followup_buffers
assert len(manager._followup_buffers[thread_id]) == 1
mock_client.runs.create.assert_called_once()
_run(go())
def test_drain_non_conflict_error_also_requeues_without_crashing(self):
"""A non-busy exception from the drain's runs.create (network error,
5xx, ...) must also be swallowed-and-requeued rather than crashing
the watcher task or losing the buffered comments."""
from app.channels.manager import ChannelManager
async def go():
bus = MessageBus()
store = ChannelStore(path=Path(tempfile.mkdtemp()) / "store.json")
manager = ChannelManager(bus=bus, store=store)
carrier_msg = InboundMessage(
channel_name="github",
chat_id="zhfeng/llm-gateway",
user_id="zhfeng",
owner_user_id="agent-owner-1",
text="queued comment",
)
thread_id = "gh-thread-neterr"
manager._buffer_followup(thread_id, carrier_msg)
mock_client = MagicMock()
mock_client.runs.create = AsyncMock(side_effect=RuntimeError("connection reset"))
await manager._drain_followups_for_thread(mock_client, thread_id, carrier_msg)
assert len(manager._followup_buffers[thread_id]) == 1
_run(go())
def test_drain_resolve_run_params_failure_requeues_entries_without_losing_them(self, monkeypatch):
"""If a step BETWEEN the buffer pop and runs.create raises -- e.g.
_resolve_run_params blows up because the target agent config was
removed mid-run -- the already-popped batch must still end up back
in the buffer instead of vanishing, and the drain must not raise
(mirrors the existing runs.create requeue-on-failure guarantee)."""
from app.channels.manager import ChannelManager
async def go():
bus = MessageBus()
store = ChannelStore(path=Path(tempfile.mkdtemp()) / "store.json")
manager = ChannelManager(bus=bus, store=store)
carrier_msg = InboundMessage(
channel_name="github",
chat_id="zhfeng/llm-gateway",
user_id="zhfeng",
owner_user_id="agent-owner-1",
text="queued comment",
)
thread_id = "gh-thread-resolve-fail"
manager._buffer_followup(thread_id, carrier_msg)
assert len(manager._followup_buffers[thread_id]) == 1
def _boom(*args, **kwargs):
raise RuntimeError("agent config missing mid-run")
monkeypatch.setattr(manager, "_resolve_run_params", _boom)
mock_client = MagicMock()
mock_client.runs.create = AsyncMock(return_value={"run_id": "should-not-be-created", "status": "pending"})
# Must not raise -- the failure must be swallowed and requeued.
await manager._drain_followups_for_thread(mock_client, thread_id, carrier_msg)
assert thread_id in manager._followup_buffers
assert len(manager._followup_buffers[thread_id]) == 1
mock_client.runs.create.assert_not_called()
_run(go())
def test_drain_apply_channel_policy_failure_requeues_entries_without_losing_them(self, monkeypatch):
"""Same guarantee one step later: if _apply_channel_policy raises
(e.g. channel-policy/credential resolution blows up instead of
following its documented degrade-and-continue path), the popped
batch must still be requeued rather than lost."""
from app.channels.manager import ChannelManager
async def go():
bus = MessageBus()
store = ChannelStore(path=Path(tempfile.mkdtemp()) / "store.json")
manager = ChannelManager(bus=bus, store=store)
carrier_msg = InboundMessage(
channel_name="github",
chat_id="zhfeng/llm-gateway",
user_id="zhfeng",
owner_user_id="agent-owner-1",
text="queued comment",
)
thread_id = "gh-thread-apply-fail"
manager._buffer_followup(thread_id, carrier_msg)
async def _boom(*args, **kwargs):
raise RuntimeError("channel policy blew up")
monkeypatch.setattr(manager, "_apply_channel_policy", _boom)
mock_client = MagicMock()
mock_client.runs.create = AsyncMock(return_value={"run_id": "should-not-be-created", "status": "pending"})
await manager._drain_followups_for_thread(mock_client, thread_id, carrier_msg)
assert thread_id in manager._followup_buffers
assert len(manager._followup_buffers[thread_id]) == 1
mock_client.runs.create.assert_not_called()
_run(go())
def test_run_watcher_drains_buffer_on_end_sentinel(self):
"""End-to-end mechanism test: a busy-thread follow-up gets buffered,
and once the ORIGINAL run's stream reaches END_SENTINEL, the watcher
drains the buffer into a new coalesced runs.create call. That
drained run is itself watched too, so an empty buffer at its own
END_SENTINEL is a clean no-op (the chain terminates)."""
import httpx
from langgraph_sdk.errors import ConflictError
import app.gateway.github.run_policy # noqa: F401 — register policy
from app.channels.manager import FOLLOWUP_BLOCK_TAG, ChannelManager
from deerflow.runtime import MemoryStreamBridge
async def go():
bus = MessageBus()
store = ChannelStore(path=Path(tempfile.mkdtemp()) / "store.json")
bridge = MemoryStreamBridge()
manager = ChannelManager(bus=bus, store=store, get_stream_bridge=lambda: bridge)
request = httpx.Request("POST", "http://127.0.0.1:2024/threads/gh-thread-watch/runs")
response = httpx.Response(409, request=request)
conflict = ConflictError("busy", response=response, body={"message": "busy"})
mock_client = _make_mock_langgraph_client(thread_id="gh-thread-watch")
mock_client.runs.create = AsyncMock(
side_effect=[
{"run_id": "run-1", "status": "pending"},
conflict,
{"run_id": "run-2", "status": "pending"},
]
)
manager._client = mock_client
# First message: no active run yet -> succeeds, watcher spawned for run-1.
await manager._handle_chat(
InboundMessage(
channel_name="github",
chat_id="zhfeng/llm-gateway",
user_id="zhfeng",
owner_user_id="agent-owner-1",
text="first comment",
metadata={"github": {"delivery_id": "d1"}},
)
)
# Second message: thread is busy -> ConflictError -> buffered.
await manager._handle_chat(
InboundMessage(
channel_name="github",
chat_id="zhfeng/llm-gateway",
user_id="zhfeng",
owner_user_id="agent-owner-1",
text="second comment while busy",
metadata={"github": {"delivery_id": "d2"}},
)
)
assert len(manager._followup_buffers["gh-thread-watch"]) == 1
# The busy run completes -> watcher observes END_SENTINEL -> drains.
await bridge.publish_end("run-1")
await _wait_for(lambda: mock_client.runs.create.call_count == 3, timeout=2.0)
drain_call = mock_client.runs.create.call_args_list[2]
assert drain_call[0][0] == "gh-thread-watch"
coalesced_text = drain_call[1]["input"]["messages"][0]["content"]
assert "second comment while busy" in coalesced_text
assert f"<{FOLLOWUP_BLOCK_TAG}>" in coalesced_text
await _wait_for(lambda: "gh-thread-watch" not in manager._followup_buffers)
# The drained run (run-2) also gets watched. Ending it with an
# empty buffer must be a clean no-op — no 4th runs.create call.
await bridge.publish_end("run-2")
await asyncio.sleep(0.2)
assert mock_client.runs.create.call_count == 3
_run(go())
def test_stop_cancels_inflight_followup_watcher_task(self):
"""A follow-up watcher spawned for a run that is still active must be
tracked and actually cancelled+awaited by manager.stop() rather than
left running as an orphaned task -- otherwise a run that ends AFTER
shutdown would still fire a brand new runs.create() into a manager
that has already been stopped."""
from app.channels.manager import ChannelManager
from deerflow.runtime import MemoryStreamBridge
async def go():
bus = MessageBus()
store = ChannelStore(path=Path(tempfile.mkdtemp()) / "store.json")
bridge = MemoryStreamBridge()
manager = ChannelManager(bus=bus, store=store, get_stream_bridge=lambda: bridge)
carrier_msg = InboundMessage(
channel_name="github",
chat_id="zhfeng/llm-gateway",
user_id="zhfeng",
owner_user_id="agent-owner-1",
text="carrier",
)
thread_id = "gh-thread-stop-watcher"
# Something buffered, so a slipped-through drain would have a
# non-empty batch to (wrongly) fire a run for.
manager._buffer_followup(
thread_id,
InboundMessage(
channel_name="github",
chat_id="c",
user_id="u",
text="queued while busy",
metadata={"github": {"delivery_id": "d-stop-1"}},
),
)
mock_client = _make_mock_langgraph_client(thread_id=thread_id)
mock_client.runs.create = AsyncMock(return_value={"run_id": "run-should-not-fire", "status": "pending"})
manager._client = mock_client
await manager.start()
# Spawn a watcher for a run whose stream never ends -- it sits
# suspended awaiting stream_bridge.subscribe(), exactly like a
# real in-flight watcher for a long-running GitHub coding run.
manager._maybe_spawn_followup_watcher(thread_id, {"run_id": "run-being-watched"}, carrier_msg)
await asyncio.sleep(0.05)
assert len(manager._followup_watcher_tasks) == 1
watcher_task = next(iter(manager._followup_watcher_tasks))
assert not watcher_task.done()
await manager.stop()
assert watcher_task.done()
assert watcher_task.cancelled()
assert watcher_task not in manager._followup_watcher_tasks
# A late "run completed" signal for the (cancelled) watched run
# must not resurrect a drain -- nothing is subscribed anymore.
await bridge.publish_end("run-being-watched")
await asyncio.sleep(0.1)
mock_client.runs.create.assert_not_called()
_run(go())
def test_drain_after_stop_does_not_create_run(self):
"""Belt-and-suspenders guard: even if a drain call reaches
_drain_followups_for_thread after the manager has been stopped (e.g.
a watcher that had already slipped past its own cancellation point
mid-drain), it must not fire client.runs.create against the stopped
manager, and the buffered entries must remain untouched (not popped,
not lost)."""
from app.channels.manager import ChannelManager
async def go():
bus = MessageBus()
store = ChannelStore(path=Path(tempfile.mkdtemp()) / "store.json")
manager = ChannelManager(bus=bus, store=store)
carrier_msg = InboundMessage(
channel_name="github",
chat_id="zhfeng/llm-gateway",
user_id="zhfeng",
owner_user_id="agent-owner-1",
text="carrier",
)
thread_id = "gh-thread-post-stop-drain"
manager._buffer_followup(
thread_id,
InboundMessage(
channel_name="github",
chat_id="c",
user_id="u",
text="queued",
metadata={"github": {"delivery_id": "d-post-stop-1"}},
),
)
await manager.start()
await manager.stop()
assert manager._running is False
mock_client = MagicMock()
mock_client.runs.create = AsyncMock(return_value={"run_id": "should-not-fire", "status": "pending"})
await manager._drain_followups_for_thread(mock_client, thread_id, carrier_msg)
mock_client.runs.create.assert_not_called()
assert len(manager._followup_buffers[thread_id]) == 1
_run(go())
def test_channel_manager_get_stream_bridge_threaded_from_service(self):
"""The app.py -> service.py -> manager.py plumbing: ChannelService
must forward get_stream_bridge through to its ChannelManager."""
from app.channels.service import ChannelService
sentinel = object()
service = ChannelService(channels_config={}, get_stream_bridge=lambda: sentinel)
assert service.manager._get_stream_bridge() is sentinel
def test_start_channel_service_forwards_get_stream_bridge(self):
"""The module-level singleton entrypoint must also thread the
callable through to ChannelService.from_app_config."""
import app.channels.service as service_module
captured: dict[str, object] = {}
class _FakeService:
async def start(self):
return None
def get_status(self):
return {}
def fake_from_app_config(app_config=None, *, get_stream_bridge=None):
captured["get_stream_bridge"] = get_stream_bridge
return _FakeService()
async def go():
service_module._channel_service = None
with patch.object(service_module.ChannelService, "from_app_config", staticmethod(fake_from_app_config)):
sentinel = object()
await service_module.start_channel_service(get_stream_bridge=lambda: sentinel)
assert captured["get_stream_bridge"]() is sentinel
try:
_run(go())
finally:
service_module._channel_service = None
class _BoundIdentityRepo:
def __init__(self, connections: list[dict[str, str | None]] | None = None) -> None:
self.connections = list(connections or [])

70
backend/tests/test_client.py
View File

@ -691,6 +691,47 @@ class TestStream:
assert tool_events[0].data["content"] == "file.txt"
assert tool_events[0].data["name"] == "bash"
assert tool_events[0].data["tool_call_id"] == "tc-1"
assert "artifact" not in tool_events[0].data
def test_messages_mode_tool_message_preserves_human_input_artifact(self, client):
"""Structured clarification data survives both embedded stream paths."""
artifact = {
"human_input": {
"request_id": "request-1",
"tool_call_id": "tc-1",
"question": "Which environment should be used?",
"options": [{"label": "Production", "value": "production"}],
}
}
tool_message = ToolMessage(
content="Which environment should be used?",
id="tm-1",
tool_call_id="tc-1",
name="ask_clarification",
artifact=artifact,
)
agent = MagicMock()
agent.stream.return_value = iter(
[
("messages", (tool_message, {})),
("values", {"messages": [HumanMessage(content="deploy", id="h-1"), tool_message]}),
]
)
with (
patch.object(client, "_ensure_agent"),
patch.object(client, "_agent", agent),
):
events = list(client.stream("deploy", thread_id="t-tool-artifact"))
tool_event = next(event for event in events if event.type == "messages-tuple" and event.data.get("type") == "tool")
values_event = next(event for event in events if event.type == "values")
serialized_tool_message = next(message for message in values_event.data["messages"] if message["type"] == "tool")
assert tool_event.data["artifact"] == artifact
assert serialized_tool_message["artifact"] == artifact
assert tool_event.data["artifact"] is artifact
assert serialized_tool_message["artifact"] is artifact
def test_list_content_blocks(self, client):
"""stream() handles AIMessage with list-of-blocks content."""
@ -3658,6 +3699,35 @@ class TestSerializeMessage:
result = DeerFlowClient._serialize_message(msg)
assert result["type"] == "tool"
assert isinstance(result["content"], str)
assert "artifact" not in result
def test_tool_message_event_preserves_native_artifact(self):
marker = object()
msg = ToolMessage(
content="result",
id="tm-1",
tool_call_id="tc-1",
name="tool",
artifact={"payload": marker},
)
result = DeerFlowClient._tool_message_event(msg)
assert result.data["artifact"] is msg.artifact
def test_tool_message_values_serialization_preserves_native_artifact(self):
marker = object()
msg = ToolMessage(
content="result",
id="tm-1",
tool_call_id="tc-1",
name="tool",
artifact={"payload": marker},
)
result = DeerFlowClient._serialize_message(msg)
assert result["artifact"] is msg.artifact
# ===========================================================================

1337
backend/tests/test_e2b_sandbox_provider.py
View File

File diff suppressed because it is too large Load Diff

8
backend/tests/test_gateway_lifespan_shutdown.py
View File

@ -42,7 +42,7 @@ async def _run_lifespan_with_hanging_stop() -> float:
fake_service = MagicMock()
fake_service.get_status = MagicMock(return_value={})
async def fake_start(_startup_config):
async def fake_start(_startup_config, **_kwargs):
return fake_service
close_oidc_service = AsyncMock()
@ -90,7 +90,7 @@ async def _run_lifespan_with_upload_staging_cleanup():
close_oidc_service = AsyncMock()
stop_channel_service = AsyncMock()
async def fake_start(_startup_config):
async def fake_start(_startup_config, **_kwargs):
return fake_service
with (
@ -142,7 +142,7 @@ async def _run_lifespan_with_memory_flush(*, enabled: bool, flush_return: bool)
close_oidc_service = AsyncMock()
stop_channel_service = AsyncMock()
async def fake_start(_startup_config):
async def fake_start(_startup_config, **_kwargs):
return fake_service
manager = MagicMock()
@ -217,7 +217,7 @@ async def _run_lifespan_with_warm_return(warm_return: bool | None) -> MagicMock:
close_oidc_service = AsyncMock()
stop_channel_service = AsyncMock()
async def fake_start(_startup_config):
async def fake_start(_startup_config, **_kwargs):
return fake_service
manager = MagicMock()

120
backend/tests/test_gateway_run_recovery.py
View File

@ -2,7 +2,9 @@
from __future__ import annotations
import asyncio
from contextlib import asynccontextmanager
from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
import anyio
@ -11,10 +13,13 @@ from fastapi import FastAPI
import deerflow.runtime as runtime_module
from app.gateway import deps as gateway_deps
from deerflow.config.run_ownership_config import RunOwnershipConfig
from deerflow.persistence import engine as engine_module
from deerflow.persistence import thread_meta as thread_meta_module
from deerflow.runtime import END_SENTINEL, MemoryStreamBridge, RunManager
from deerflow.runtime.checkpointer import async_provider as checkpointer_module
from deerflow.runtime.events import store as event_store_module
from deerflow.runtime.runs.store.memory import MemoryRunStore
@asynccontextmanager
@ -29,16 +34,23 @@ class _FakeRunManager:
recovered_runs = [SimpleNamespace(run_id="run-1", thread_id="thread-1")]
latest_by_thread: dict[str, list[SimpleNamespace]] = {}
def __init__(self, *, store, run_ownership_config=None):
def __init__(self, *, store, run_ownership_config=None, on_orphans_recovered=None):
self.store = store
self.run_ownership_config = run_ownership_config
self.on_orphans_recovered = on_orphans_recovered
self.reconcile_calls: list[dict] = []
self.list_by_thread_calls: list[dict] = []
self.shutdown_calls: int = 0
_FakeRunManager.instances.append(self)
async def reconcile_orphaned_inflight_runs(self, *, error: str, before: str | None = None):
self.reconcile_calls.append({"error": error, "before": before})
async def reconcile_orphaned_inflight_runs(
self,
*,
error: str,
before: str | None = None,
stop_reason: str | None = None,
):
self.reconcile_calls.append({"error": error, "before": before, "stop_reason": stop_reason})
return self.recovered_runs
async def list_by_thread(self, thread_id: str, *, user_id=None, limit: int = 100):
@ -81,6 +93,35 @@ class _FakeStreamBridge:
self.cleanup_calls.append((run_id, delay))
class _RetainedMemoryStreamBridge(MemoryStreamBridge):
"""Memory bridge that records cleanup without deleting test history."""
def __init__(self) -> None:
super().__init__()
self.cleanup_calls: list[tuple[str, float]] = []
async def cleanup(self, run_id: str, *, delay: float = 0) -> None:
self.cleanup_calls.append((run_id, delay))
class _DelayedCleanupStreamBridge(_FakeStreamBridge):
def __init__(self) -> None:
super().__init__(existing_streams={"run-1"})
self.delayed_cleanup_started = asyncio.Event()
self.delayed_cleanup_cancelled = asyncio.Event()
async def cleanup(self, run_id: str, *, delay: float = 0) -> None:
self.cleanup_calls.append((run_id, delay))
if delay <= 0:
return
self.delayed_cleanup_started.set()
try:
await asyncio.Event().wait()
except asyncio.CancelledError:
self.delayed_cleanup_cancelled.set()
raise
@pytest.mark.anyio
async def test_recovered_run_stream_end_skips_expired_stream():
"""Startup recovery should not recreate an already-expired retained stream."""
@ -95,6 +136,78 @@ async def test_recovered_run_stream_end_skips_expired_stream():
assert stream_bridge.cleanup_calls == []
@pytest.mark.anyio
async def test_shutdown_flushes_delayed_recovered_stream_cleanup_immediately():
"""Bridge shutdown must not abandon a delayed cleanup until the stream TTL."""
stream_bridge = _DelayedCleanupStreamBridge()
cleanups = await gateway_deps._publish_recovered_run_stream_end(
stream_bridge,
[SimpleNamespace(run_id="run-1", thread_id="thread-1")],
cleanup_delay=60.0,
)
cleanup_tasks = {task: run_id for run_id, task in cleanups}
await asyncio.wait_for(stream_bridge.delayed_cleanup_started.wait(), timeout=0.5)
await gateway_deps._flush_recovered_stream_cleanups(
stream_bridge,
cleanup_tasks,
)
assert stream_bridge.delayed_cleanup_cancelled.is_set()
assert stream_bridge.cleanup_calls == [("run-1", 60.0), ("run-1", 0)]
@pytest.mark.anyio
async def test_periodic_recovery_terminalizes_stream_without_thread_projection():
"""Periodic recovery must close streams without racing thread projection."""
store = MemoryRunStore()
stream_bridge = _RetainedMemoryStreamBridge()
thread_store = _FakeThreadStore()
expired = (datetime.now(UTC) - timedelta(seconds=60)).isoformat()
await store.put(
"periodic-orphan",
thread_id="thread-1",
status="running",
owner_worker_id="dead-worker",
lease_expires_at=expired,
created_at=expired,
)
await stream_bridge.publish("periodic-orphan", "values", {"step": 1})
async def terminalize(recovered_runs):
await gateway_deps._terminalize_recovered_runs(
stream_bridge,
recovered_runs,
cleanup_delay=60.0,
)
manager = RunManager(
store=store,
worker_id="live-worker",
run_ownership_config=RunOwnershipConfig(
heartbeat_enabled=True,
lease_seconds=30,
grace_seconds=10,
),
on_orphans_recovered=terminalize,
)
await manager._reconcile_orphans_periodic()
await anyio.sleep(0)
received = [
entry
async for entry in stream_bridge.subscribe(
"periodic-orphan",
heartbeat_interval=0.01,
)
]
await anyio.sleep(0)
assert received[-1] is END_SENTINEL
assert stream_bridge.cleanup_calls == [("periodic-orphan", 60.0)]
assert thread_store.status_updates == []
@pytest.mark.anyio
async def test_sqlite_runtime_reconciles_orphaned_runs_on_startup(monkeypatch):
"""SQLite startup should recover stale active runs before serving requests."""
@ -133,6 +246,7 @@ async def test_sqlite_runtime_reconciles_orphaned_runs_on_startup(monkeypatch):
assert len(_FakeRunManager.instances) == 1
assert _FakeRunManager.instances[0].reconcile_calls
assert _FakeRunManager.instances[0].reconcile_calls[0]["error"]
assert _FakeRunManager.instances[0].reconcile_calls[0]["stop_reason"] == runtime_module.ORPHAN_RECOVERY_STOP_REASON
assert _FakeRunManager.instances[0].list_by_thread_calls == [{"thread_id": "thread-1", "user_id": None, "limit": 1}]
assert thread_store.status_updates == [("thread-1", "error", None)]
assert stream_bridge.publish_end_calls == ["run-1"]

77
backend/tests/test_gateway_services.py
View File

@ -79,6 +79,28 @@ def test_normalize_stream_modes_empty_list():
assert normalize_stream_modes([]) == ["values"]
@pytest.mark.parametrize("raw", ["messages", "events", "tools", ["values", "events"]])
def test_normalize_stream_modes_rejects_unsupported_modes(raw):
from app.gateway.services import normalize_stream_modes
with pytest.raises(ValueError, match="Unsupported stream mode"):
normalize_stream_modes(raw)
@pytest.mark.parametrize(
("raw", "expected"),
[
("messages-tuple", ["messages"]),
(["values", "messages-tuple", "messages-tuple", "values"], ["values", "messages"]),
(["updates", "custom"], ["updates", "custom"]),
],
)
def test_to_langgraph_stream_modes_maps_alias_and_deduplicates(raw, expected):
from deerflow.runtime.stream_modes import to_langgraph_stream_modes
assert to_langgraph_stream_modes(raw) == expected
def test_normalize_input_none():
from app.gateway.services import normalize_input
@ -1438,15 +1460,19 @@ def test_launch_scheduled_thread_run_marks_context_non_interactive(_stub_app_con
from types import SimpleNamespace
from unittest.mock import patch
from app.gateway.routers.thread_runs import RunCreateRequest
from app.gateway.services import launch_scheduled_thread_run
async def _scenario():
captured: dict[str, object] = {}
async def fake_start_run(body, thread_id, request):
captured["body"] = body
captured["thread_id"] = thread_id
captured["context"] = body.context
captured["metadata"] = body.metadata
captured["if_not_exists"] = body.if_not_exists
captured["on_completion"] = body.on_completion
return SimpleNamespace(run_id="run-1", thread_id=thread_id)
with patch("app.gateway.services.start_run", side_effect=fake_start_run):
@ -1463,8 +1489,11 @@ def test_launch_scheduled_thread_run_marks_context_non_interactive(_stub_app_con
captured, result = asyncio.run(_scenario())
assert captured["thread_id"] == "thread-scheduled"
assert isinstance(captured["body"], RunCreateRequest)
assert captured["context"] == {"non_interactive": True, "user_id": "user-1"}
assert captured["metadata"] == {"scheduled_task_id": "task-1"}
assert captured["if_not_exists"] == "create"
assert captured["on_completion"] is None
assert result == {"run_id": "run-1", "thread_id": "thread-scheduled"}
@ -1757,6 +1786,54 @@ class TestInjectAuthenticatedUserContextAuthz:
inject_authenticated_user_context(config, _make_request_with_auth_source("session"))
@pytest.mark.asyncio
async def test_run_agent_invalid_stream_mode_finalizes_run_before_graph_invocation():
import asyncio
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock
from deerflow.runtime.runs.manager import RunManager
from deerflow.runtime.runs.schemas import RunStatus
from deerflow.runtime.runs.worker import RunContext, run_agent
run_manager = RunManager()
record = await run_manager.create("thread-invalid-stream-mode")
bridge = SimpleNamespace(
publish=AsyncMock(),
publish_end=AsyncMock(),
cleanup=AsyncMock(),
)
agent_factory = MagicMock()
await run_agent(
bridge,
run_manager,
record,
ctx=RunContext(checkpointer=None),
agent_factory=agent_factory,
graph_input={"messages": []},
config={"configurable": {"thread_id": record.thread_id}},
stream_modes=["events"],
)
await asyncio.sleep(0)
assert record.status == RunStatus.error
assert record.error == "Unsupported stream mode(s): events"
agent_factory.assert_not_called()
bridge.publish.assert_awaited_once_with(
record.run_id,
"error",
{
"message": "Unsupported stream mode(s): events",
"name": "UnsupportedStreamModeError",
},
)
bridge.publish_end.assert_awaited_once_with(record.run_id)
bridge.cleanup.assert_awaited_once_with(record.run_id, delay=60)
replacement = await run_manager.create_or_reject(record.thread_id)
assert replacement.run_id != record.run_id
@pytest.mark.asyncio
async def test_run_agent_full_mode_rejects_delta_before_graph_invocation():
import asyncio

42
backend/tests/test_lead_agent_prompt.py
View File

@ -262,6 +262,48 @@ def test_apply_prompt_template_clamps_subagent_limits_to_enforced_bounds(monkeyp
assert "MAXIMUM 50 `task` CALLS PER RUN" in prompt
def test_apply_prompt_template_single_subagent_limit_matches_middleware(monkeypatch):
"""Regression test for single-subagent mode (MIN_CONCURRENT_SUBAGENT_CALLS = 1).
Before the floor was lowered to 1, a user-configured limit of 1 was silently
bumped to 2 by both the prompt path and the middleware. This renders the real
system prompt with max_concurrent_subagents=1 and asserts the advertised
HARD LIMITS value equals the middleware-enforced max_concurrent, so the two
paths cannot drift apart on the newly-allowed value.
"""
from deerflow.agents.middlewares.subagent_limit_middleware import SubagentLimitMiddleware
explicit_config = SimpleNamespace(
sandbox=SimpleNamespace(
use="deerflow.sandbox.local:LocalSandboxProvider",
allow_host_bash=False,
mounts=[],
),
subagents=SubagentsAppConfig(),
skills=SimpleNamespace(container_path="/mnt/skills", use="deerflow.skills.storage.local_skill_storage:LocalSkillStorage", get_skills_path=lambda: Path("/tmp/skills")),
skill_evolution=SimpleNamespace(enabled=False),
tool_search=SimpleNamespace(enabled=False),
memory=SimpleNamespace(enabled=False, injection_enabled=True, max_injection_tokens=2000),
acp_agents={},
)
monkeypatch.setattr(prompt_module, "get_or_new_skill_storage", lambda app_config=None: SimpleNamespace(load_skills=lambda enabled_only=True: []))
monkeypatch.setattr(prompt_module, "get_agent_soul", lambda agent_name=None: "")
enforced = SubagentLimitMiddleware(max_concurrent=1).max_concurrent
assert enforced == 1 # 1 must pass through, not be bumped to 2
prompt = prompt_module.apply_prompt_template(
subagent_enabled=True,
max_concurrent_subagents=1,
max_total_subagents=6,
app_config=explicit_config,
)
assert f"MAXIMUM {enforced} `task` CALLS PER RESPONSE" in prompt
assert f"HARD LIMITS: max {enforced} `task` calls per response" in prompt
def test_build_acp_section_uses_explicit_app_config_without_global_config(monkeypatch):
explicit_config = SimpleNamespace(acp_agents={"codex": object()})

3
backend/tests/test_migration_0004_run_ownership_dedupe.py
View File

@ -156,7 +156,8 @@ async def test_migration_dedupes_duplicate_active_rows_before_unique_index(tmp_p
with sqlite3.connect(db_path) as raw:
version_row = raw.execute("SELECT version_num FROM alembic_version").fetchone()
assert version_row[0] == "0007_feedback_tags"
# Bootstrap upgrades through the later revisions after 0004.
assert version_row[0] == "0008_feedback_tags"
# Sanity: the invariant the index enforces is now true — at most one
# active row per thread.

180
backend/tests/test_migration_0007_scheduled_run_active_dedupe.py Normal file
View File

@ -0,0 +1,180 @@
"""Regression test for migration ``0007_scheduled_run_active_index`` dedupe pass.
End-to-end shape (mirrors ``test_migration_0004_run_ownership_dedupe``):
1. Hand-build a SQLite DB that mirrors a real pre-0007 deployment that ran the
racy ``dispatch_task`` check-then-insert and accumulated duplicate active
rows per ``task_id`` (the exact dirty state the partial unique index
targets).
2. Stamp it at ``0006_agents`` so ``bootstrap_schema`` takes the
versioned branch and runs ``alembic upgrade head``.
3. Insert two+ queued/running rows for the same ``task_id`` (only possible
because the partial unique index does not exist yet).
4. Run ``init_engine`` (the FastAPI lifespan entry point), which routes through
``bootstrap_schema`` -> ``upgrade head`` -> ``0007.upgrade()``.
5. Verify the migration superseded the older duplicates (set them to
``interrupted`` with an explanatory message + ``finished_at``), kept the
newest active row, and successfully built the ``uq_scheduled_task_run_active``
partial unique index.
Pre-fix codepath would have raised ``UNIQUE constraint failed`` (SQLite) /
``could not create unique index`` (Postgres) on step 5, aborting the alembic
upgrade and blocking gateway startup.
"""
from __future__ import annotations
import sqlite3
from datetime import UTC, datetime, timedelta
from pathlib import Path
import pytest
import sqlalchemy as sa
from sqlalchemy.orm import Session
import deerflow.persistence.models # noqa: F401 -- registers ORM models
from deerflow.persistence.base import Base
from deerflow.persistence.engine import close_engine, init_engine
from deerflow.persistence.scheduled_task_runs.model import ScheduledTaskRunRow
pytestmark = pytest.mark.asyncio
def _seed_pre_0007_with_duplicates(db_path: Path) -> None:
"""Build a DB at revision 0006 with duplicate active rows per task_id.
``Base.metadata.create_all`` produces the full current schema (including
the partial unique index), so we drop just that index to land in the dirty
state the migration's dedupe pass targets, then stamp at 0005 and insert
the duplicates via the ORM so Python-side defaults populate.
"""
db_path.parent.mkdir(parents=True, exist_ok=True)
sync_engine = sa.create_engine(f"sqlite:///{db_path.as_posix()}")
try:
Base.metadata.create_all(sync_engine)
with sync_engine.begin() as conn:
# Drop only the partial unique index — its absence is what permits
# duplicate active rows to exist in the first place.
conn.execute(sa.text("DROP INDEX IF EXISTS uq_scheduled_task_run_active"))
# Stamp at 0006 so bootstrap takes the versioned branch and runs
# ``alembic upgrade head`` (which is what executes 0007.upgrade()).
conn.execute(sa.text("CREATE TABLE IF NOT EXISTS alembic_version (version_num VARCHAR(32) NOT NULL)"))
conn.execute(sa.text("DELETE FROM alembic_version"))
conn.execute(sa.text("INSERT INTO alembic_version (version_num) VALUES ('0006_agents')"))
base = datetime.now(UTC)
with Session(sync_engine) as session:
session.add_all(
[
# Task with three active rows: two older duplicates + newest.
ScheduledTaskRunRow(
id="run-old-a",
task_id="task-dup",
thread_id="thread-a",
scheduled_for=base,
trigger="scheduled",
status="queued",
created_at=base,
),
ScheduledTaskRunRow(
id="run-old-b",
task_id="task-dup",
thread_id="thread-b",
scheduled_for=base,
trigger="manual",
status="running",
created_at=base + timedelta(seconds=10),
),
ScheduledTaskRunRow(
id="run-newest",
task_id="task-dup",
thread_id="thread-c",
scheduled_for=base,
trigger="scheduled",
status="queued",
created_at=base + timedelta(seconds=60),
),
# A single-active-row task: must be left untouched.
ScheduledTaskRunRow(
id="run-solo",
task_id="task-solo",
thread_id="thread-solo",
scheduled_for=base,
trigger="scheduled",
status="running",
created_at=base,
),
# A terminal row: must stay terminal.
ScheduledTaskRunRow(
id="run-done",
task_id="task-done",
thread_id="thread-done",
scheduled_for=base,
trigger="scheduled",
status="success",
created_at=base,
),
]
)
session.commit()
finally:
sync_engine.dispose()
def _fetch_runs(db_path: Path) -> dict[str, tuple[str, str | None, str | None]]:
"""Map id -> (status, error, finished_at) for assertions."""
with sqlite3.connect(db_path) as raw:
rows = raw.execute("SELECT id, status, error, finished_at FROM scheduled_task_runs").fetchall()
return {run_id: (status, error, finished_at) for run_id, status, error, finished_at in rows}
def _index_exists(db_path: Path, index_name: str) -> bool:
with sqlite3.connect(db_path) as raw:
row = raw.execute(
"SELECT 1 FROM sqlite_master WHERE type='index' AND name=?",
(index_name,),
).fetchone()
return row is not None
async def test_migration_supersedes_duplicate_active_runs_before_unique_index(tmp_path: Path) -> None:
db_path = tmp_path / "dirty.db"
_seed_pre_0007_with_duplicates(db_path)
url = f"sqlite+aiosqlite:///{db_path.as_posix()}"
await init_engine(backend="sqlite", url=url, sqlite_dir=str(tmp_path))
try:
runs = _fetch_runs(db_path)
# Newest active row on the duplicated task survives unchanged.
assert runs["run-newest"][0] == "queued"
assert runs["run-newest"][1] is None
# Older duplicate active rows are superseded with an explanatory error
# and a finished_at timestamp (mark_stale_active_runs orphan semantics).
for run_id in ("run-old-a", "run-old-b"):
status, error, finished_at = runs[run_id]
assert status == "interrupted", (run_id, status)
assert "uq_scheduled_task_run_active" in (error or ""), (run_id, error)
assert finished_at is not None, run_id
# Untouched tasks: single active row stays active, terminal stays terminal.
assert runs["run-solo"][0] == "running"
assert runs["run-done"][0] == "success"
# The partial unique index was successfully created — the upgrade did
# not abort with ``UNIQUE constraint failed``.
assert _index_exists(db_path, "uq_scheduled_task_run_active")
with sqlite3.connect(db_path) as raw:
version_row = raw.execute("SELECT version_num FROM alembic_version").fetchone()
assert version_row[0] == "0007_scheduled_run_active_index"
# Sanity: the invariant the index enforces now holds — at most one
# active row per task_id.
with sqlite3.connect(db_path) as raw:
dupes = raw.execute("SELECT task_id, COUNT(*) FROM scheduled_task_runs WHERE status IN ('queued', 'running') GROUP BY task_id HAVING COUNT(*) > 1").fetchall()
assert dupes == []
finally:
await close_engine()

4
backend/tests/test_monocle_tracing.py
View File

@ -330,7 +330,7 @@ def test_gateway_lifespan_initializes_monocle():
fake_service = MagicMock()
fake_service.get_status = MagicMock(return_value={})
async def fake_start(_startup_config):
async def fake_start(_startup_config, **_kwargs):
return fake_service
setup_spy = MagicMock(return_value=False)
@ -372,7 +372,7 @@ def test_gateway_lifespan_survives_monocle_setup_failure(caplog):
fake_service = MagicMock()
fake_service.get_status = MagicMock(return_value={})
async def fake_start(_startup_config):
async def fake_start(_startup_config, **_kwargs):
return fake_service
setup_spy = MagicMock(side_effect=ValueError("MONOCLE_EXPORTERS has unknown exporter(s): fle."))

267
backend/tests/test_multi_worker_run_ownership.py
View File

@ -5,6 +5,7 @@ Coverage:
- create_or_reject with interrupt strategy claims and cancels old runs
- create_run_atomic refuses to interrupt a run owned by another live worker
- reconcile_orphaned_inflight_runs uses lease-based detection
- periodic reconciliation notifies Gateway recovery orchestration
- Worker reconciliation skips runs with unexpired leases
- Lease heartbeat renews active run leases
- GATEWAY_WORKERS=1 + heartbeat_enabled=false behaviour unchanged
@ -19,7 +20,7 @@ from unittest.mock import AsyncMock
import pytest
from deerflow.config.run_ownership_config import RunOwnershipConfig
from deerflow.runtime import RunManager, RunStatus
from deerflow.runtime import ORPHAN_RECOVERY_STOP_REASON, RunManager, RunStatus
from deerflow.runtime.runs.manager import CancelOutcome, ConflictError, _generate_worker_id
from deerflow.runtime.runs.store.memory import MemoryRunStore
@ -389,6 +390,262 @@ async def test_reconciliation_returns_empty_when_no_orphaned_runs():
assert recovered == []
@pytest.mark.anyio
async def test_periodic_reconciliation_notifies_recovery_callback():
"""Periodic recovery must hand terminalized rows to Gateway orchestration."""
store = MemoryRunStore()
on_orphans_recovered = AsyncMock()
manager = _make_manager(
store=store,
on_orphans_recovered=on_orphans_recovered,
)
expired_lease = (datetime.now(UTC) - timedelta(seconds=60)).isoformat()
await store.put(
"periodic-orphan",
thread_id="thread-1",
status="running",
owner_worker_id="dead-worker",
lease_expires_at=expired_lease,
created_at=(datetime.now(UTC) - timedelta(seconds=120)).isoformat(),
)
await manager._reconcile_orphans_periodic()
await asyncio.sleep(0)
on_orphans_recovered.assert_awaited_once()
recovered = on_orphans_recovered.await_args.args[0]
assert [record.run_id for record in recovered] == ["periodic-orphan"]
assert recovered[0].status == RunStatus.error
assert recovered[0].stop_reason == ORPHAN_RECOVERY_STOP_REASON
stored = await store.get("periodic-orphan")
assert stored is not None
assert stored["stop_reason"] == ORPHAN_RECOVERY_STOP_REASON
@pytest.mark.anyio
async def test_periodic_reconciliation_logs_recovered_run_ids_when_callback_fails(caplog):
"""Callback failures must identify every recovered run in the warning."""
store = MemoryRunStore()
on_orphans_recovered = AsyncMock(side_effect=RuntimeError("callback failed"))
manager = _make_manager(
store=store,
on_orphans_recovered=on_orphans_recovered,
)
expired_lease = (datetime.now(UTC) - timedelta(seconds=60)).isoformat()
created_at = (datetime.now(UTC) - timedelta(seconds=120)).isoformat()
for run_id in ("periodic-orphan-1", "periodic-orphan-2"):
await store.put(
run_id,
thread_id=f"thread-{run_id}",
status="running",
owner_worker_id="dead-worker",
lease_expires_at=expired_lease,
created_at=created_at,
)
with caplog.at_level("WARNING", logger="deerflow.runtime.runs.manager"):
await manager._reconcile_orphans_periodic()
await asyncio.sleep(0)
assert "Periodic orphan recovery callback failed for 2 run(s)" in caplog.text
assert "periodic-orphan-1" in caplog.text
assert "periodic-orphan-2" in caplog.text
@pytest.mark.anyio
async def test_periodic_terminalization_does_not_block_lease_renewal_or_shutdown():
"""The real heartbeat loop must keep renewing during a slow callback."""
store = MemoryRunStore()
callback_started = asyncio.Event()
callback_release = asyncio.Event()
callback_finished = asyncio.Event()
async def on_orphans_recovered(_recovered):
callback_started.set()
await callback_release.wait()
callback_finished.set()
manager = _make_manager(
store=store,
run_ownership_config=_lease_config(heartbeat_enabled=True, lease_seconds=5),
on_orphans_recovered=on_orphans_recovered,
)
active = await manager.create_or_reject("active-thread")
await manager.set_status(active.run_id, RunStatus.running)
original_expiry = active.lease_expires_at
expired_lease = (datetime.now(UTC) - timedelta(seconds=60)).isoformat()
await store.put(
"periodic-orphan",
thread_id="orphan-thread",
status="running",
owner_worker_id="dead-worker",
lease_expires_at=expired_lease,
created_at=expired_lease,
)
await manager.start_heartbeat()
await asyncio.wait_for(callback_started.wait(), timeout=4.5)
expiry_during_callback = active.lease_expires_at
await asyncio.sleep(1.2)
assert expiry_during_callback != original_expiry
assert active.lease_expires_at != expiry_during_callback
assert callback_finished.is_set() is False
shutdown_task = asyncio.create_task(manager.shutdown(timeout=1.0))
await asyncio.sleep(0)
assert shutdown_task.done() is False
callback_release.set()
await asyncio.wait_for(shutdown_task, timeout=1.0)
assert callback_finished.is_set() is True
@pytest.mark.anyio
async def test_periodic_store_scan_does_not_block_real_heartbeat_loop():
"""A slow orphan scan must not stall later lease-renewal cycles."""
class SlowScanStore(MemoryRunStore):
def __init__(self):
super().__init__()
self.scan_started = asyncio.Event()
self.scan_release = asyncio.Event()
async def list_inflight_with_expired_lease(
self,
*,
before=None,
grace_seconds=10,
):
self.scan_started.set()
await self.scan_release.wait()
return await super().list_inflight_with_expired_lease(
before=before,
grace_seconds=grace_seconds,
)
store = SlowScanStore()
manager = _make_manager(
store=store,
run_ownership_config=_lease_config(heartbeat_enabled=True, lease_seconds=5),
)
active = await manager.create_or_reject("active-thread")
await manager.set_status(active.run_id, RunStatus.running)
await manager.start_heartbeat()
await asyncio.wait_for(store.scan_started.wait(), timeout=4.5)
expiry_during_scan = active.lease_expires_at
await asyncio.sleep(1.2)
assert active.lease_expires_at != expiry_during_scan
store.scan_release.set()
await manager.stop_heartbeat()
await manager._drain_orphan_recovery_task(timeout=0.5)
@pytest.mark.anyio
async def test_periodic_recovery_is_single_flight():
"""A second trigger must not create another recovery pipeline."""
store = MemoryRunStore()
callback_started = asyncio.Event()
callback_release = asyncio.Event()
callback_calls = 0
async def on_orphans_recovered(_recovered):
nonlocal callback_calls
callback_calls += 1
callback_started.set()
await callback_release.wait()
manager = _make_manager(
store=store,
on_orphans_recovered=on_orphans_recovered,
)
expired_lease = (datetime.now(UTC) - timedelta(seconds=60)).isoformat()
await store.put(
"periodic-orphan",
thread_id="orphan-thread",
status="running",
owner_worker_id="dead-worker",
lease_expires_at=expired_lease,
created_at=expired_lease,
)
manager._schedule_orphan_reconciliation()
await asyncio.wait_for(callback_started.wait(), timeout=0.5)
first_task = manager._orphan_recovery_task
manager._schedule_orphan_reconciliation()
assert manager._orphan_recovery_task is first_task
assert callback_calls == 1
callback_release.set()
await asyncio.wait_for(first_task, timeout=0.5)
@pytest.mark.anyio
async def test_shutdown_cancels_recovery_that_exceeds_drain_budget():
"""The pending -> cancel -> gather branch must observe callback cancellation."""
store = MemoryRunStore()
callback_started = asyncio.Event()
callback_cancelled = asyncio.Event()
async def on_orphans_recovered(_recovered):
callback_started.set()
try:
await asyncio.Event().wait()
except asyncio.CancelledError:
callback_cancelled.set()
raise
manager = _make_manager(
store=store,
on_orphans_recovered=on_orphans_recovered,
)
expired_lease = (datetime.now(UTC) - timedelta(seconds=60)).isoformat()
await store.put(
"periodic-orphan",
thread_id="orphan-thread",
status="running",
owner_worker_id="dead-worker",
lease_expires_at=expired_lease,
created_at=expired_lease,
)
manager._schedule_orphan_reconciliation()
await asyncio.wait_for(callback_started.wait(), timeout=0.5)
await manager.shutdown(timeout=0.01)
assert callback_cancelled.is_set()
assert manager._orphan_recovery_task is None
@pytest.mark.anyio
async def test_shutdown_applies_shared_deadline_to_heartbeat_stop():
"""A stuck heartbeat must not receive a separate five-second budget."""
manager = _make_manager(
run_ownership_config=_lease_config(heartbeat_enabled=True),
)
heartbeat_cancelled = asyncio.Event()
async def stuck_heartbeat():
try:
await asyncio.Event().wait()
except asyncio.CancelledError:
heartbeat_cancelled.set()
raise
manager._heartbeat_stop = asyncio.Event()
manager._heartbeat_task = asyncio.create_task(stuck_heartbeat())
started = asyncio.get_running_loop().time()
await manager.shutdown(timeout=0.01)
elapsed = asyncio.get_running_loop().time() - started
assert heartbeat_cancelled.is_set()
assert elapsed < 0.5
# ---------------------------------------------------------------------------
# Lease heartbeat
# ---------------------------------------------------------------------------
@ -1023,13 +1280,19 @@ async def test_claim_for_takeover_succeeds_with_expired_lease():
expired_lease = (datetime.now(UTC) - timedelta(seconds=grace + 5)).isoformat()
await store.put("run-1", thread_id="t1", status="running", created_at=datetime.now(UTC).isoformat(), owner_worker_id="w-a", lease_expires_at=expired_lease)
ok = await store.claim_for_takeover("run-1", grace_seconds=grace, error="claimed")
ok = await store.claim_for_takeover(
"run-1",
grace_seconds=grace,
error="claimed",
stop_reason=ORPHAN_RECOVERY_STOP_REASON,
)
assert ok is True
row = await store.get("run-1")
assert row is not None
assert row["status"] == "error"
assert row["error"] == "claimed"
assert row["stop_reason"] == ORPHAN_RECOVERY_STOP_REASON
@pytest.mark.anyio

2
backend/tests/test_persistence_bootstrap.py
View File

@ -48,7 +48,7 @@ from deerflow.persistence.migrations._helpers import _normalize_default
asyncio_test = pytest.mark.asyncio
HEAD = "0007_feedback_tags"
HEAD = "0008_feedback_tags"
BASELINE = "0001_baseline"

2
backend/tests/test_persistence_bootstrap_concurrency.py
View File

@ -28,7 +28,7 @@ from deerflow.persistence.bootstrap import bootstrap_schema
pytestmark = pytest.mark.asyncio
HEAD = "0007_feedback_tags"
HEAD = "0008_feedback_tags"
def _url(tmp_path: Path) -> str:

4
backend/tests/test_persistence_bootstrap_regression.py
View File

@ -76,7 +76,7 @@ async def test_legacy_database_recovers_token_usage_column(tmp_path: Path) -> No
cols = {row[1] for row in raw.execute("PRAGMA table_info(runs)").fetchall()}
assert "token_usage_by_model" in cols
version_row = raw.execute("SELECT version_num FROM alembic_version").fetchone()
assert version_row[0] == "0007_feedback_tags"
assert version_row[0] == "0008_feedback_tags"
# And the read path that originally 500'd must now succeed.
sf = get_session_factory()
@ -116,6 +116,6 @@ async def test_legacy_database_with_manual_alter_still_bootstraps(tmp_path: Path
# No duplicate column -- list, not set, to catch dupes.
assert cols.count("token_usage_by_model") == 1
version_row = raw.execute("SELECT version_num FROM alembic_version").fetchone()
assert version_row[0] == "0007_feedback_tags"
assert version_row[0] == "0008_feedback_tags"
finally:
await close_engine()

21
backend/tests/test_run_manager.py
View File

@ -68,7 +68,7 @@ class FailingTakeoverRunStore(MemoryRunStore):
super().__init__()
self.takeover_attempts = 0
async def claim_for_takeover(self, run_id, *, grace_seconds, error):
async def claim_for_takeover(self, run_id, *, grace_seconds, error, stop_reason=None):
self.takeover_attempts += 1
raise sqlite3.OperationalError("database is locked")
@ -576,6 +576,25 @@ async def test_list_by_thread_merges_store_runs_newest_first():
assert runs[0] is memory_record
@pytest.mark.anyio
async def test_list_by_thread_limit_does_not_let_old_memory_hide_new_store_run():
"""A local row must not consume the store query's newest-run limit."""
store = MemoryRunStore()
manager = RunManager(store=store)
old_memory = await manager.create("thread-1")
old_memory.created_at = "2026-01-01T00:00:00+00:00"
await store.put(
"new-store",
thread_id="thread-1",
status="success",
created_at="2026-01-02T00:00:00+00:00",
)
runs = await manager.list_by_thread("thread-1", limit=1)
assert [run.run_id for run in runs] == ["new-store"]
@pytest.mark.anyio
async def test_create_defaults(manager: RunManager):
"""Create with no optional args should use defaults."""

210
backend/tests/test_run_request_validation.py Normal file
View File

@ -0,0 +1,210 @@
"""Contract tests for the LangGraph-compatible run request boundary."""
from __future__ import annotations
from typing import Any
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
from app.gateway.routers import runs
SUPPORTED_STREAM_MODES = {
"values",
"messages-tuple",
"updates",
"debug",
"tasks",
"checkpoints",
"custom",
}
@pytest.fixture
def client() -> TestClient:
app = FastAPI()
app.include_router(runs.router)
return TestClient(app, raise_server_exceptions=False)
@pytest.mark.parametrize(
("field", "value"),
[
("webhook", "https://example.com/callback"),
("stream_resumable", False),
("on_completion", "complete"),
("on_completion", "continue"),
("on_completion", "keep"),
("on_completion", "delete"),
("after_seconds", 1.5),
("if_not_exists", "reject"),
("feedback_keys", []),
("multitask_strategy", "enqueue"),
],
)
def test_run_request_rejects_each_unsupported_option_with_exact_422(
client: TestClient,
field: str,
value: Any,
) -> None:
response = client.post("/api/runs/stream", json={field: value})
assert response.status_code == 422
assert response.json() == {
"detail": [
{
"type": "unsupported_run_option",
"loc": ["body", field],
"msg": f"Run option '{field}' is not supported by DeerFlow",
"input": value,
"ctx": {"option": field},
}
]
}
@pytest.mark.parametrize(
"stream_mode",
[
"messages",
"events",
"tools",
["values", "events"],
["events", "tools"],
],
)
def test_run_request_rejects_unsupported_stream_modes_with_exact_422(
client: TestClient,
stream_mode: str | list[str],
) -> None:
response = client.post("/api/runs/stream", json={"stream_mode": stream_mode})
requested = [stream_mode] if isinstance(stream_mode, str) else stream_mode
unsupported = list(dict.fromkeys(mode for mode in requested if mode not in SUPPORTED_STREAM_MODES))
mode_list = ", ".join(unsupported)
assert response.status_code == 422
assert response.json() == {
"detail": [
{
"type": "unsupported_stream_mode",
"loc": ["body", "stream_mode"],
"msg": f"Unsupported stream mode(s): {mode_list}",
"input": stream_mode,
"ctx": {"modes": mode_list},
}
]
}
def test_run_request_keeps_supported_modes_and_compatibility_defaults() -> None:
from app.gateway.routers.thread_runs import RunCreateRequest
body = RunCreateRequest(
stream_mode=list(SUPPORTED_STREAM_MODES),
webhook=None,
stream_resumable=None,
on_completion=None,
after_seconds=None,
if_not_exists="create",
feedback_keys=None,
)
assert set(body.stream_mode or []) == SUPPORTED_STREAM_MODES
assert body.on_completion is None
assert body.if_not_exists == "create"
@pytest.mark.parametrize(
"payload",
[
{"multitask_strategy": []},
{"stream_mode": [{}]},
],
)
def test_malformed_option_types_remain_validation_errors(client: TestClient, payload: dict[str, Any]) -> None:
response = client.post("/api/runs/stream", json=payload)
assert response.status_code == 422
@pytest.mark.parametrize(
"field",
[
"multitask_strategy",
"if_not_exists",
],
)
def test_string_run_options_keep_native_type_errors(client: TestClient, field: str) -> None:
response = client.post("/api/runs/stream", json={field: []})
assert response.status_code == 422
assert response.json()["detail"][0]["type"] == "literal_error"
@pytest.mark.parametrize(
("field", "value"),
[
("checkpoint_during", False),
("durability", "sync"),
],
)
def test_run_request_rejects_unknown_sdk_options_with_exact_422(
client: TestClient,
field: str,
value: Any,
) -> None:
response = client.post("/api/runs/stream", json={field: value})
assert response.status_code == 422
assert response.json() == {
"detail": [
{
"type": "extra_forbidden",
"loc": ["body", field],
"msg": "Extra inputs are not permitted",
"input": value,
}
]
}
def test_openapi_stream_mode_enum_matches_runtime_support() -> None:
app = FastAPI()
app.include_router(runs.router)
openapi = app.openapi()
schemas = openapi["components"]["schemas"]
stream_mode_schema = schemas["RunCreateRequest"]["properties"]["stream_mode"]
enums: list[str] = []
def collect_enums(value: Any) -> None:
if isinstance(value, dict):
enum = value.get("enum")
if isinstance(enum, list):
enums.extend(item for item in enum if isinstance(item, str))
for child in value.values():
collect_enums(child)
elif isinstance(value, list):
for child in value:
collect_enums(child)
collect_enums(stream_mode_schema)
collect_enums(schemas["RunStreamMode"])
assert set(enums) == SUPPORTED_STREAM_MODES
assert "events" not in enums
def test_openapi_run_option_schema_exposes_only_supported_values() -> None:
app = FastAPI()
app.include_router(runs.router)
schema = app.openapi()["components"]["schemas"]["RunCreateRequest"]
properties = schema["properties"]
assert schema["additionalProperties"] is False
for field in ("webhook", "stream_resumable", "after_seconds", "feedback_keys"):
assert properties[field]["type"] == "null"
assert properties["on_completion"]["type"] == "null"
assert properties["if_not_exists"]["const"] == "create"
assert properties["multitask_strategy"]["enum"] == ["reject", "rollback", "interrupt"]

219
backend/tests/test_scheduled_task_dispatch_race.py Normal file
View File

@ -0,0 +1,219 @@
"""Concurrency regression tests for the scheduled-task dispatch TOCTOU.
``ScheduledTaskService.dispatch_task`` guards the "at most one active run per
task when overlap_policy=skip" invariant with a non-atomic
``has_active_runs`` check followed by a separate ``create(status="queued")``
insert. Two concurrent dispatches (double-click, client retry, or a manual
trigger racing the poller) can both pass the check and both launch. The fix
makes the database the atomic arbiter via the partial unique index
``uq_scheduled_task_run_active`` (``task_id WHERE status IN
('queued','running')``); the losing insert is translated to the typed
``ActiveScheduledRunConflict`` and collapsed to the same outcome as the
fast-path check.
These tests drive the REAL ``ScheduledTaskRunRepository`` + ``ScheduledTaskService``
against a real file-backed ``sqlite+aiosqlite`` DB (so the index is actually
enforced), with a fake ``launch_run`` that only records launches.
"""
from __future__ import annotations
import asyncio
from datetime import UTC, datetime
import pytest
from app.scheduler.service import ScheduledTaskService
from deerflow.config.database_config import DatabaseConfig
from deerflow.persistence.engine import close_engine, get_session_factory, init_engine_from_config
from deerflow.persistence.scheduled_task_runs import ActiveScheduledRunConflict, ScheduledTaskRunRepository
from deerflow.persistence.scheduled_tasks import ScheduledTaskRepository
pytestmark = pytest.mark.asyncio
_ACTIVE_STATUSES = {"queued", "running"}
class _BarrierRunRepo(ScheduledTaskRunRepository):
"""Real repository that only releases both dispatchers past
``has_active_runs`` once both have read it, so their ``create()`` calls
genuinely race for the task's single active slot — a deterministic
reproduction of the check-then-insert TOCTOU."""
def __init__(self, session_factory, barrier: asyncio.Barrier | None) -> None:
super().__init__(session_factory)
self._barrier = barrier
async def has_active_runs(self, task_id: str) -> bool:
result = await super().has_active_runs(task_id)
if self._barrier is not None:
await self._barrier.wait()
return result
def _make_service(task_repo, run_repo, launched: list) -> ScheduledTaskService:
async def fake_launch(**kwargs):
# Yield so a truly-concurrent sibling can interleave, then record.
await asyncio.sleep(0)
launched.append(kwargs)
return {"run_id": f"run-{len(launched)}", "thread_id": kwargs["thread_id"]}
return ScheduledTaskService(
task_repo=task_repo,
task_run_repo=run_repo,
launch_run=fake_launch,
poll_interval_seconds=5,
lease_seconds=120,
max_concurrent_runs=10,
)
async def _seed_task(task_repo: ScheduledTaskRepository, task_id: str) -> dict:
# fresh_thread_per_run: every dispatch gets a NEW thread_id, so #4003's
# per-thread uq_runs_thread_active can never fire for two dispatches of the
# same task — this is precisely the gap the per-task index closes.
await task_repo.create(
task_id=task_id,
user_id="user-1",
thread_id=None,
context_mode="fresh_thread_per_run",
assistant_id="lead_agent",
title=task_id,
prompt="do the thing",
schedule_type="cron",
schedule_spec={"cron": "*/5 * * * *"},
timezone="UTC",
next_run_at=None,
)
task = await task_repo.get(task_id, user_id="user-1")
assert task is not None
assert task["overlap_policy"] == "skip"
return task
async def _active_run_count(run_repo: ScheduledTaskRunRepository, task_id: str) -> int:
rows = await run_repo.list_by_task(task_id, limit=100)
return sum(1 for row in rows if row["status"] in _ACTIVE_STATUSES)
async def test_two_concurrent_manual_dispatches_launch_exactly_once(tmp_path):
await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path)))
try:
sf = get_session_factory()
assert sf is not None
task_repo = ScheduledTaskRepository(sf)
run_repo = _BarrierRunRepo(sf, asyncio.Barrier(2))
launched: list = []
service = _make_service(task_repo, run_repo, launched)
task = await _seed_task(task_repo, "task-race-manual")
now = datetime.now(UTC)
results = await asyncio.gather(
service.dispatch_task(dict(task), now=now, trigger="manual"),
service.dispatch_task(dict(task), now=now, trigger="manual"),
)
outcomes = sorted(result["outcome"] for result in results)
# Exactly one wins the active slot; the loser is a 409-style conflict.
assert outcomes == ["conflict", "launched"], outcomes
assert len(launched) == 1, launched
assert await _active_run_count(run_repo, "task-race-manual") == 1
# The manual loser records no run-history row (nothing was scheduled).
conflict = next(r for r in results if r["outcome"] == "conflict")
assert conflict["task_run_id"] is None
finally:
await close_engine()
async def test_scheduled_and_manual_dispatch_launch_exactly_once(tmp_path):
await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path)))
try:
sf = get_session_factory()
assert sf is not None
task_repo = ScheduledTaskRepository(sf)
run_repo = _BarrierRunRepo(sf, asyncio.Barrier(2))
launched: list = []
service = _make_service(task_repo, run_repo, launched)
task = await _seed_task(task_repo, "task-race-mixed")
now = datetime.now(UTC)
results = await asyncio.gather(
service.dispatch_task(dict(task), now=now, trigger="scheduled"),
service.dispatch_task(dict(task), now=now, trigger="manual"),
)
outcomes = sorted(result["outcome"] for result in results)
# Whichever won launched; the loser is conflict (manual) or skipped
# (scheduled). Which one wins is timing-dependent, but exactly one runs.
assert outcomes.count("launched") == 1, outcomes
assert set(outcomes) <= {"launched", "conflict", "skipped"}, outcomes
assert len(launched) == 1, launched
assert await _active_run_count(run_repo, "task-race-mixed") == 1
finally:
await close_engine()
async def test_natural_timing_concurrent_dispatch_launches_exactly_once(tmp_path):
# No barrier: exercise the fix under the same natural interleaving that
# reproduced the bug (5/5 both-launch on main). The fix must hold whether
# the second dispatch is caught by the has_active_runs fast path or by the
# index-violation path.
await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path)))
try:
sf = get_session_factory()
assert sf is not None
task_repo = ScheduledTaskRepository(sf)
run_repo = ScheduledTaskRunRepository(sf)
for i in range(5):
launched: list = []
service = _make_service(task_repo, run_repo, launched)
task_id = f"task-natural-{i}"
task = await _seed_task(task_repo, task_id)
now = datetime.now(UTC)
results = await asyncio.gather(
service.dispatch_task(dict(task), now=now, trigger="manual"),
service.dispatch_task(dict(task), now=now, trigger="manual"),
)
outcomes = sorted(result["outcome"] for result in results)
assert outcomes.count("launched") == 1, (i, outcomes)
assert len(launched) == 1, (i, launched)
assert await _active_run_count(run_repo, task_id) == 1, i
finally:
await close_engine()
async def test_partial_unique_index_enforces_one_active_run_per_task(tmp_path):
# Focused repository-level test of the index semantics + the typed conflict.
await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path)))
try:
sf = get_session_factory()
assert sf is not None
run_repo = ScheduledTaskRunRepository(sf)
now = datetime(2026, 7, 2, 1, 0, tzinfo=UTC)
await run_repo.create(run_record_id="r1", task_id="t1", thread_id="th1", scheduled_for=now, trigger="scheduled", status="queued")
# queued -> running is a same-row UPDATE: keeps the one active slot, no
# violation (this is the normal launch transition).
await run_repo.update_status("r1", status="running", run_id="run-1", started_at=now)
assert await run_repo.has_active_runs("t1") is True
# A second active insert for the same task is a domain conflict.
with pytest.raises(ActiveScheduledRunConflict):
await run_repo.create(run_record_id="r2", task_id="t1", thread_id="th2", scheduled_for=now, trigger="manual", status="queued")
# Terminal-status rows for the same task are outside the index predicate.
await run_repo.create(run_record_id="r3", task_id="t1", thread_id="th3", scheduled_for=now, trigger="scheduled", status="skipped")
# A different task's active row is independent.
await run_repo.create(run_record_id="r4", task_id="t2", thread_id="th4", scheduled_for=now, trigger="scheduled", status="queued")
# Finishing the active run frees the slot; a fresh active row is allowed.
await run_repo.update_status("r1", status="success", run_id="run-1", finished_at=now)
assert await run_repo.has_active_runs("t1") is False
await run_repo.create(run_record_id="r5", task_id="t1", thread_id="th5", scheduled_for=now, trigger="scheduled", status="queued")
assert await run_repo.has_active_runs("t1") is True
finally:
await close_engine()

14
backend/tests/test_scheduled_task_repository.py
View File

@ -67,6 +67,10 @@ async def test_mark_stale_active_runs_fails_orphaned_runs(tmp_path):
assert sf is not None
repo = ScheduledTaskRunRepository(sf)
# The queued and running rows live on different tasks: mark_stale_active_runs
# is a global sweep (no task filter), and the uq_scheduled_task_run_active
# partial unique index forbids two active rows on one task_id, so the pair
# that proves both active statuses get swept must be spread across tasks.
await repo.create(
run_record_id="task-run-queued",
task_id="task-1",
@ -77,12 +81,14 @@ async def test_mark_stale_active_runs_fails_orphaned_runs(tmp_path):
)
await repo.create(
run_record_id="task-run-running",
task_id="task-1",
thread_id="thread-1",
task_id="task-2",
thread_id="thread-2",
scheduled_for=datetime(2026, 7, 2, 1, 0, tzinfo=UTC),
trigger="scheduled",
status="running",
)
# A terminal row on task-1 (outside the index predicate) coexists with the
# active queued row and must be left untouched by the sweep.
await repo.create(
run_record_id="task-run-success",
task_id="task-1",
@ -95,8 +101,8 @@ async def test_mark_stale_active_runs_fails_orphaned_runs(tmp_path):
swept = await repo.mark_stale_active_runs(error="interrupted: gateway restarted")
assert swept == 2
history = await repo.list_by_task("task-1")
by_id = {entry["id"]: entry for entry in history}
by_id = {entry["id"]: entry for entry in await repo.list_by_task("task-1")}
by_id.update({entry["id"]: entry for entry in await repo.list_by_task("task-2")})
assert by_id["task-run-queued"]["status"] == "interrupted"
assert by_id["task-run-running"]["status"] == "interrupted"
assert by_id["task-run-success"]["status"] == "success"

6
backend/tests/test_scheduled_task_service.py
View File

@ -447,7 +447,11 @@ async def test_skip_policy_applies_to_fresh_thread_runs():
assert result["outcome"] == "skipped"
assert launched == []
assert run_repo.created["status"] == "queued"
# The skip tombstone is created directly as terminal "skipped" (not the
# transient "queued" the launch path uses): a queued row is active and would
# itself trip the uq_scheduled_task_run_active partial unique index against
# the pre-existing run still holding the task's single active slot.
assert run_repo.created["status"] == "skipped"
assert run_repo.updated[-1][1]["status"] == "skipped"
assert task_repo.updated[1]["status"] == "enabled"
assert task_repo.updated[1]["increment_run_count"] is False

30
backend/tests/test_skill_review_core.py
View File

@ -56,6 +56,22 @@ def test_review_core_reports_missing_description_blocker(tmp_path):
assert any(f["rule_id"] == "structure.missing-description" for f in facts["findings"])
def test_review_core_reports_non_string_frontmatter_key_as_unknown_field(tmp_path):
_write(
tmp_path / "SKILL.md",
"---\nname: demo-skill\ndescription: Demo skill. Invoke when testing review.\n42: stray-value\nunexpected-field: another-value\n---\n\n# Demo\n\nFollow the steps and stop.\n",
)
facts = analyze_skill_package(LocalDirectoryReader(tmp_path).read())
assert facts["summary"]["blockers"] == 0
finding = next(f for f in facts["findings"] if f["rule_id"] == "structure.unknown-frontmatter-field")
assert finding["severity"] == "warning"
assert finding["evidence"] == ["42", "unexpected-field"]
assert "42" in finding["message"]
assert "unexpected-field" in finding["message"]
def test_resource_graph_reports_unreferenced_resource(tmp_path):
_write(tmp_path / "SKILL.md", _valid_skill())
_write(tmp_path / "references" / "unused.md", "# Unused\n")
@ -277,6 +293,20 @@ def test_cli_fail_on_error(tmp_path, capsys):
assert "structure.missing-description" in output
def test_cli_reports_non_string_frontmatter_key_without_crashing(tmp_path, capsys):
_write(
tmp_path / "SKILL.md",
"---\nname: demo-skill\ndescription: Demo skill. Invoke when testing review.\n42: stray-value\nunexpected-field: another-value\n---\n\n# Demo\n\nFollow the steps and stop.\n",
)
exit_code = review_cli_main([str(tmp_path), "--format", "text", "--fail-on", "error", "--fail-on-incomplete"])
output = capsys.readouterr().out
assert exit_code == 0
assert "structure.unknown-frontmatter-field" in output
assert "Unknown frontmatter field(s): 42, unexpected-field" in output
def test_cli_fail_on_incomplete_package(tmp_path, capsys):
_write(tmp_path / "SKILL.md", _valid_skill())
_write(tmp_path / "references" / "large.md", "x" * 32)

11
backend/tests/test_skills_validation.py
View File

@ -116,6 +116,17 @@ class TestValidateSkillFrontmatter:
assert valid is False
assert "custom-field" in msg
def test_non_string_frontmatter_key_reports_cleanly_instead_of_crashing(self, tmp_path):
skill_dir = _write_skill(
tmp_path,
"---\nname: my-skill\ndescription: test\n42: bad\ncustom-field: bad\n---\n\nBody\n",
)
valid, msg, name = _validate_skill_frontmatter(skill_dir)
assert valid is False
assert "custom-field" in msg
assert "42" in msg
assert name is None
def test_name_must_be_hyphen_case(self, tmp_path):
skill_dir = _write_skill(
tmp_path,

20
backend/tests/test_subagent_limit_middleware.py
View File

@ -52,11 +52,23 @@ def _raw_tool_call(call_id: str, name: str = "task") -> dict:
class TestClampSubagentLimit:
def test_below_min_clamped_to_min(self):
assert _clamp_subagent_limit(0) == MIN_SUBAGENT_LIMIT
assert _clamp_subagent_limit(1) == MIN_SUBAGENT_LIMIT
def test_min_limit_is_one(self):
# MIN lowered from 2 to 1 so a user asking for a single subagent gets 1.
# Both consumers (SubagentLimitMiddleware.__init__ and the prompt path)
# share this floor via clamp_subagent_concurrency in subagents_config.py.
assert MIN_SUBAGENT_LIMIT == 1
assert MAX_SUBAGENT_LIMIT == 4
def test_above_max_clamped_to_max(self):
def test_below_min_clamped_to_one(self):
assert _clamp_subagent_limit(0) == 1
assert _clamp_subagent_limit(-5) == 1
def test_one_is_allowed_not_bumped_to_two(self):
# Previously 1 clamped up to 2; it must now pass through as 1.
assert _clamp_subagent_limit(1) == 1
def test_above_max_clamped_to_four(self):
assert _clamp_subagent_limit(5) == 4
assert _clamp_subagent_limit(10) == MAX_SUBAGENT_LIMIT
assert _clamp_subagent_limit(100) == MAX_SUBAGENT_LIMIT

61
backend/tests/test_thread_messages_page.py
View File

@ -63,14 +63,15 @@ def test_thread_page_orders_across_runs_and_paginates_without_gaps():
assert older.json()["next_before_seq"] is None
def test_thread_page_scans_past_middleware_chunks_to_fill_visible_page(monkeypatch):
@pytest.mark.parametrize("caller", ["middleware:title", "subagent:general-purpose"])
def test_thread_page_scans_past_internal_ai_chunks_to_fill_visible_page(monkeypatch, caller):
monkeypatch.setattr(thread_runs, "THREAD_MESSAGE_PAGE_SCAN_BATCH", 3)
store = MemoryRunEventStore()
async def seed():
await _put_message(store, "run-1", "human", "visible-old")
for index in range(3):
await _put_message(store, "run-1", "ai", f"middleware-{index}", caller="middleware:title")
await _put_message(store, "run-1", "ai", f"internal-{index}", caller=caller)
await _put_message(store, "run-2", "human", "visible-new-human")
await _put_message(store, "run-2", "ai", "visible-new-ai")
@ -85,6 +86,62 @@ def test_thread_page_scans_past_middleware_chunks_to_fill_visible_page(monkeypat
assert body["next_before_seq"] == 5
def test_thread_page_hides_subagent_ai_but_keeps_root_task_result():
store = MemoryRunEventStore()
async def seed():
await _put_message(store, "run-1", "human", "user-prompt")
await store.put(
thread_id="thread-1",
run_id="run-1",
event_type="llm.ai.response",
category="message",
content={
"type": "ai",
"id": "lead-task-call",
"content": "",
"tool_calls": [{"id": "task-0", "name": "task", "args": {}}],
"additional_kwargs": {},
},
metadata={"caller": "lead_agent"},
)
await _put_message(
store,
"run-1",
"ai",
"internal-subagent-answer",
caller="subagent:general-purpose",
)
await store.put(
thread_id="thread-1",
run_id="run-1",
event_type="llm.tool.result",
category="message",
content={
"type": "tool",
"id": "root-task-result",
"tool_call_id": "task-0",
"content": "Task Succeeded. Result: poem",
"additional_kwargs": {"subagent_status": "completed"},
},
metadata={},
)
await _put_message(store, "run-1", "ai", "lead-final-answer")
asyncio.run(seed())
app = _make_app(store)
with TestClient(app) as client:
response = client.get("/api/threads/thread-1/messages/page")
assert response.status_code == 200
assert [row["content"]["id"] for row in response.json()["data"]] == [
"user-prompt",
"lead-task-call",
"root-task-result",
"lead-final-answer",
]
def test_thread_page_scans_large_middleware_only_region_with_production_batch_size():
store = MemoryRunEventStore()

20
backend/tests/test_view_image_tool.py
View File

@ -12,6 +12,9 @@ view_image_module = importlib.import_module("deerflow.tools.builtins.view_image_
PNG_BYTES = base64.b64decode("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==")
# Minimal 1x1 transparent GIF (starts with the "GIF89a" magic bytes).
GIF_BYTES = base64.b64decode("R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7")
def _make_thread_data(tmp_path: Path) -> dict[str, str]:
user_data = tmp_path / "threads" / "thread-1" / "user-data"
@ -73,6 +76,23 @@ def test_view_image_reads_virtual_uploads_path(tmp_path: Path) -> None:
assert viewed_image["actual_path"] == str(image_path)
def test_view_image_reads_gif(tmp_path: Path) -> None:
thread_data = _make_thread_data(tmp_path)
image_path = Path(thread_data["uploads_path"]) / "animation.gif"
image_path.write_bytes(GIF_BYTES)
result = view_image_tool.func(
runtime=_make_runtime(thread_data),
image_path="/mnt/user-data/uploads/animation.gif",
tool_call_id="tc-gif",
)
assert _message_content(result) == "Successfully read image"
viewed_image = result.update["viewed_images"]["/mnt/user-data/uploads/animation.gif"]
assert viewed_image["mime_type"] == "image/gif"
assert viewed_image["size"] == len(GIF_BYTES)
def test_view_image_rejects_spoofed_extension(tmp_path: Path) -> None:
thread_data = _make_thread_data(tmp_path)
image_path = Path(thread_data["uploads_path"]) / "not-really.png"

165
backend/tests/test_wait_disconnect_handling.py
View File

@ -1,4 +1,4 @@
"""Regression tests for issue #3265.
"""Regression tests for issues #3265 and #3932.
The non-streaming ``/wait`` endpoints used to ``await record.task`` with no
disconnect handling and silently swallow ``CancelledError``. When a long
@ -10,7 +10,10 @@ The fix introduces ``wait_for_run_completion`` in ``app.gateway.services``:
it subscribes to the stream bridge until ``END_SENTINEL``, polls
``request.is_disconnected()`` on every wake-up, and honours the record's
``on_disconnect`` mode by cancelling the background run on real client
disconnect.
disconnect. Store-only consumers wait for the bridge's real terminal marker
instead of treating an ordinary durable terminal status as proof that all tail
events have already been published. A durable ``orphan_recovered`` stop reason
provides the narrow heartbeat fallback when the publisher is known to be gone.
"""
from __future__ import annotations
@ -19,7 +22,7 @@ import asyncio
from dataclasses import dataclass, field
from typing import Any
from deerflow.runtime import RunManager, RunRecord, RunStatus
from deerflow.runtime import ORPHAN_RECOVERY_STOP_REASON, RunManager, RunRecord, RunStatus
from deerflow.runtime.runs.schemas import DisconnectMode
from deerflow.runtime.stream_bridge.memory import MemoryStreamBridge
@ -69,6 +72,17 @@ class _MissingStreamBridge:
return None
class _FastHeartbeatBridge(MemoryStreamBridge):
"""Memory bridge with a short heartbeat for durable-status refresh tests."""
def subscribe(self, run_id, *, last_event_id=None, heartbeat_interval=15.0):
return super().subscribe(
run_id,
last_event_id=last_event_id,
heartbeat_interval=0.01,
)
async def _create_running_record(mgr: RunManager, *, on_disconnect: DisconnectMode) -> Any:
record = await mgr.create_or_reject(
THREAD_ID,
@ -249,3 +263,148 @@ class TestWaitForRunCompletion:
assert bridge.subscribed is False
asyncio.run(run())
def test_sse_consumer_preserves_tail_events_after_durable_terminal_status(self) -> None:
"""A durable terminal row must not overtake delayed error and END events."""
from app.gateway.services import sse_consumer
from deerflow.runtime.runs.store.memory import MemoryRunStore
async def run() -> None:
store = MemoryRunStore()
await store.put(
"periodic-orphan",
thread_id=THREAD_ID,
status="running",
)
mgr = RunManager(store=store)
record = await mgr.get("periodic-orphan")
assert record is not None
assert record.store_only is True
bridge = _FastHeartbeatBridge()
await bridge.publish(record.run_id, "values", {"step": 1})
request = _FakeRequest()
consumer = sse_consumer(bridge, record, request, mgr)
first_frame = await anext(consumer)
assert first_frame.startswith("event: values\n")
await store.update_status(record.run_id, "error", error="lease expired")
async def publish_tail() -> None:
await asyncio.sleep(0.05)
await bridge.publish(record.run_id, "error", {"message": "late error"})
await bridge.publish_end(record.run_id)
publisher = asyncio.create_task(publish_tail())
tail_frames = [frame async for frame in consumer]
await publisher
error_index = next(index for index, frame in enumerate(tail_frames) if frame.startswith("event: error\n"))
end_index = next(index for index, frame in enumerate(tail_frames) if frame.startswith("event: end\n"))
assert error_index < end_index
assert record.status == RunStatus.running
asyncio.run(run())
def test_wait_preserves_tail_events_after_durable_terminal_status(self) -> None:
"""The wait path must remain blocked until the real END is published."""
from app.gateway.services import wait_for_run_completion
from deerflow.runtime.runs.store.memory import MemoryRunStore
async def run() -> None:
store = MemoryRunStore()
await store.put(
"periodic-orphan",
thread_id=THREAD_ID,
status="running",
)
mgr = RunManager(store=store)
record = await mgr.get("periodic-orphan")
assert record is not None
assert record.store_only is True
bridge = _FastHeartbeatBridge()
await bridge.publish(record.run_id, "values", {"step": 1})
await store.update_status(record.run_id, "error", error="lease expired")
wait_task = asyncio.create_task(wait_for_run_completion(bridge, record, _FakeRequest(), mgr))
await asyncio.sleep(0.05)
assert wait_task.done() is False
await bridge.publish(record.run_id, "error", {"message": "late error"})
await asyncio.sleep(0)
assert wait_task.done() is False
await bridge.publish_end(record.run_id)
completed = await asyncio.wait_for(wait_task, timeout=1.0)
assert completed is True
assert record.status == RunStatus.running
asyncio.run(run())
def test_sse_consumer_uses_explicit_orphan_recovery_liveness_boundary(
self,
) -> None:
"""A recovered orphan may synthesize END when its publisher is gone."""
from app.gateway.services import sse_consumer
from deerflow.runtime.runs.store.memory import MemoryRunStore
async def run() -> None:
store = MemoryRunStore()
await store.put(
"periodic-orphan",
thread_id=THREAD_ID,
status="running",
)
mgr = RunManager(store=store)
record = await mgr.get("periodic-orphan")
assert record is not None
bridge = _FastHeartbeatBridge()
await bridge.publish(record.run_id, "values", {"step": 1})
consumer = sse_consumer(bridge, record, _FakeRequest(), mgr)
assert (await anext(consumer)).startswith("event: values\n")
await store.update_status(
record.run_id,
"error",
error="lease expired",
stop_reason=ORPHAN_RECOVERY_STOP_REASON,
)
end_frame = await asyncio.wait_for(anext(consumer), timeout=1.0)
assert end_frame == "event: end\ndata: null\n\n"
asyncio.run(run())
def test_wait_uses_explicit_orphan_recovery_liveness_boundary(self) -> None:
"""The non-streaming consumer shares the recovered-orphan boundary."""
from app.gateway.services import wait_for_run_completion
from deerflow.runtime.runs.store.memory import MemoryRunStore
async def run() -> None:
store = MemoryRunStore()
await store.put(
"periodic-orphan",
thread_id=THREAD_ID,
status="running",
)
mgr = RunManager(store=store)
record = await mgr.get("periodic-orphan")
assert record is not None
bridge = _FastHeartbeatBridge()
await bridge.publish(record.run_id, "values", {"step": 1})
await store.update_status(
record.run_id,
"error",
error="lease expired",
stop_reason=ORPHAN_RECOVERY_STOP_REASON,
)
completed = await asyncio.wait_for(
wait_for_run_completion(bridge, record, _FakeRequest(), mgr),
timeout=1.0,
)
assert completed is True
asyncio.run(run())

1
frontend/AGENTS.md
View File

@ -89,6 +89,7 @@ Tool-calling AI messages can contain user-visible text as well as `tool_calls`.
- **Thread hooks** (`useThreadStream`, `useSubmitThread`, `useThreads`) are the primary API interface
- **Thread routes** — construct Web UI chat paths through `core/threads/utils.ts::pathOfThread()`, which percent-encodes both custom agent names and thread IDs before inserting them into route segments
- **LangGraph client** is a singleton obtained via `getAPIClient()` in `core/api/`
- **Run stream options** are sanitized by `core/api/stream-mode.ts`: the Gateway-supported set is `values`, `messages-tuple`, `updates`, `debug`, `tasks`, `checkpoints`, and `custom`; any request containing an unsupported mode throws before HTTP instead of being partially forwarded or silently defaulting to `values`. `streamResumable` is retained by thread hooks only for SDK-side reconnect bookkeeping but stripped before the HTTP request because the Gateway does not implement resumable SSE. Keep this boundary aligned with the backend request schema; `messages` and `events` are not supported and must not be forwarded.
- **Streaming Markdown rendering** is owned by `core/streamdown`: Streamdown's `animated` / `isAnimating` API handles incremental word animation, while the shared `streamdownRenderingPlugins` config registers the named code-highlighting and Mermaid plugins required by Streamdown 2.5. Keep wrappers and derived configs wired to that shared object; do not reintroduce a rehype plugin that wraps every word, because reparsing a growing block remounts old words and replays their animation.
- **Environment validation** uses `@t3-oss/env-nextjs` with Zod schemas (`src/env.js`). Skip with `SKIP_ENV_VALIDATION=1`
- **Subtask step history and runtime metadata** (`core/tasks/`) — the subtask card shows a subagent's full step timeline (#3779): its assistant reasoning turns interleaved with the tools it ran. `Subtask.steps[]` is accumulated live from `task_running` events (appended via `mergeSteps`, not overwritten) and backfilled on expand for historical runs by `fetchSubtaskSteps`, which pages the events endpoint scoped to one task (GET `/runs/{runId}/events?event_types=subagent.step&task_id=…&after_seq=…`) until a short page, so the run-wide limit can't truncate the timeline. `task_started` carries the effective `model_name`; `task_running` carries a cumulative usage snapshot after each completed LLM call. `core/tasks/lifecycle.ts` normalizes these additive events, and `computeNextSubtask` keeps the largest cumulative total so replayed or late SSE frames cannot double-count or roll the folded card backward. Terminal ToolMessage metadata (`subagent_model_name` / `subagent_token_usage`) restores the same values from normal history after reload; no per-card event fetch is needed. `core/tasks/steps.ts` is the pure step model: `messageToStep` (live), `eventsToSteps` (reload), `mergeSteps` (dedup by `message_index`), and `stepsForDisplay` (what the card renders — keeps tool steps + AI steps with text, drops the trailing final-answer AI step when completed since it's shown as `result`). `core/tasks/context.tsx`'s `useUpdateSubtask` applies updates against a `tasksRef` mirroring the latest state (not a closure snapshot), so a late-resolving `fetchSubtaskSteps` backfill merges into current state instead of clobbering SSE steps or sibling subtasks that arrived meanwhile. The owning `run_id` is carried onto history content messages in `buildVisibleHistoryMessages` so the card can resolve the events endpoint.

15
frontend/src/app/workspace/agents/new/page.tsx
View File

@ -46,7 +46,7 @@ import {
} from "@/core/messages/human-input";
import { isHiddenFromUIMessage } from "@/core/messages/utils";
import { safeLocalStorage } from "@/core/settings/local";
import { useThreadStream } from "@/core/threads/hooks";
import { hasToolResult, useThreadStream } from "@/core/threads/hooks";
import { uuid } from "@/core/utils/uuid";
import { isIMEComposing } from "@/lib/ime";
import { cn } from "@/lib/utils";
@ -100,13 +100,14 @@ export default function NewAgentPage() {
mode: "flash",
is_bootstrap: true,
},
onFinish() {
if (!agent && setupAgentStatus === "requested") {
setSetupAgentStatus("idle");
onFinish(state) {
if (agent || setupAgentStatus !== "requested") {
return;
}
if (!agentName || !hasToolResult(state.messages, "setup_agent")) {
setSetupAgentStatus("idle");
return;
}
},
onToolEnd({ name }) {
if (name !== "setup_agent" || !agentName) return;
setSetupAgentStatus("completed");
void getAgentWithRetry(agentName).then((fetched) => {
if (fetched) {

12
frontend/src/components/workspace/messages/message-list-item.tsx
View File

@ -241,17 +241,22 @@ function MessageImage({
}) {
if (!src) return null;
const imgClassName = cn("overflow-hidden rounded-lg", `max-w-[${maxWidth}]`);
// `maxWidth` is applied inline rather than through a `max-w-[${maxWidth}]`
// class: Tailwind's JIT only generates utilities it can find as literal
// source tokens, so an interpolated arbitrary value would never be emitted.
const imgClassName = cn("overflow-hidden rounded-lg", props.className);
const imgStyle: React.CSSProperties = { maxWidth, ...props.style };
if (typeof src !== "string") {
return (
<img
{...props}
className={imgClassName}
style={imgStyle}
src={src}
alt={alt}
loading="lazy"
decoding="async"
{...props}
/>
);
}
@ -261,12 +266,13 @@ function MessageImage({
return (
<a href={url} target="_blank" rel="noopener noreferrer">
<img
{...props}
className={imgClassName}
style={imgStyle}
src={url}
alt={alt}
loading="lazy"
decoding="async"
{...props}
/>
</a>
);

12
frontend/src/content/en/harness/middlewares.mdx
View File

@ -17,6 +17,14 @@ Every time the Lead Agent calls the LLM, it runs through a **middleware chain**
This design keeps the agent core simple and stable while allowing rich, composable behaviors to be layered in.
<Callout type="info">
Each subagent runs its own agent loop and gets its own middleware chain. The
loop-detection, token-budget, and summarization guards below are mirrored on
the subagent chain (#3875); the other middlewares are Lead-Agent-specific
(e.g. memory, title generation, clarification). See
[Subagents → Runaway guards](/docs/harness/subagents#runaway-guards).
</Callout>
## How the chain works
The middleware chain is built once per agent invocation, based on the current configuration and request parameters. The middlewares run in a defined order:
@ -58,6 +66,8 @@ Detects when the agent is making the same tool call repeatedly without making pr
Warning interventions are queued per thread and run, then drained on the next model call as a single hidden `HumanMessage(name="loop_warning")` appended after existing tool results. This keeps provider tool-call pairing valid. Run start/end hooks clear stale or undelivered warnings, and hard stops still strip tool calls before forcing a final text response.
This middleware is also attached to the subagent chain, where only the tool-loop heuristic can fire (subagents disallow `task`), so a degenerate subagent loop is broken the same way.
**Configuration**: built-in, no user configuration.
---
@ -145,6 +155,8 @@ The backend stamps bounded structured task-result and skill-read metadata before
When the conversation grows long, summarizes older messages to reduce context size. The generated summary is stored in thread state and projected into later model calls as hidden durable context data, preserving meaning without keeping the original messages in the active transcript.
The same middleware and the same `summarization.enabled` switch also compact subagent transcripts, so a single config covers both the Lead Agent and subagent chains (#3875).
**Configuration**: `summarization:` section in `config.yaml`. See detailed configuration below.
---

43
frontend/src/content/en/harness/subagents.mdx
View File

@ -30,15 +30,15 @@ DeerFlow ships with two built-in subagents:
A general-purpose reasoning and execution agent. Suitable for delegating complex subtasks that require multi-step reasoning, web search, file operations, and artifact production.
- **Default timeout**: 900 seconds (15 minutes)
- **Default max turns**: 160
- **Default timeout**: 1800 seconds (30 minutes)
- **Default max turns**: 150
### bash
A subagent specialized for command-line task execution inside the sandbox. Suitable for scripting, data processing, file transformation, and environment setup tasks.
- **Default timeout**: 900 seconds (15 minutes)
- **Default max turns**: 80
- **Default timeout**: 1800 seconds (30 minutes)
- **Default max turns**: 60
- **Availability**: only exposed when the sandbox's `bash` tool is available (either `allow_host_bash: true` or a container sandbox is configured)
## Delegation flow
@ -57,32 +57,45 @@ The runtime then:
1. Looks up the subagent configuration from the registry, applying any `config.yaml` overrides.
2. Creates a new agent invocation with the subagent's own prompt and tools.
3. Runs the subagent to completion (or until timeout / max turns).
3. Runs the subagent to completion — bounded by `max_turns`, the `timeout`, and a **middleware guard chain** that mirrors the Lead Agent's (loop detection, token budget, summarization). See [Runaway guards](#runaway-guards) below.
4. Returns the subagent's final output to the Lead Agent as the tool result.
## Configuration
Subagent timeouts and max turns are controlled through the `subagents:` section in `config.yaml`:
Subagent timeouts, max turns, and the per-run token budget are controlled through the `subagents:` section in `config.yaml`:
```yaml
subagents:
# Default timeout in seconds for all subagents (default: 900 = 15 minutes)
timeout_seconds: 900
# Default timeout in seconds for all subagents (default: 1800 = 30 minutes)
timeout_seconds: 1800
# Optional: override max turns for all subagents
# Optional: override max turns for all subagents.
# Built-in defaults: general-purpose=150, bash=60. Leave unset to keep them.
# max_turns: 120
# Per-run token ceiling — a backstop against a subagent burning tokens on
# trivial work. At the hard-stop the in-flight turn is capped (tool calls
# stripped, finish_reason forced to "stop") so the run completes naturally;
# the result is stamped completed + subagent_stop_reason=token_capped so the
# lead/UI can tell a budget-capped run from a clean one (#3875).
token_budget:
enabled: true
max_tokens: 2000000 # generous default — lower it to tighten cost controls
warn_threshold: 0.7 # log a warning once this fraction of the budget is spent
# Optional: per-agent overrides
agents:
general-purpose:
timeout_seconds: 1800 # 30 minutes for complex tasks
max_turns: 160
# token_budget: # per-agent override of the global token_budget above
# max_tokens: 3000000
bash:
timeout_seconds: 300 # 5 minutes for quick commands
max_turns: 80
```
Per-agent overrides take priority over the global `timeout_seconds` and `max_turns` settings.
Per-agent overrides take priority over the global `timeout_seconds`, `max_turns`, and `token_budget` settings.
## Delegation limits
@ -94,6 +107,16 @@ The `SubagentLimitMiddleware` controls how many subagents the Lead Agent can inv
If the agent tries to call more subagents than the limits allow, the middleware trims the excess calls. When the total cap is exhausted, it stops new `task` calls for that run and lets the agent synthesize from already collected results.
## Runaway guards
A subagent runs its own agent loop, so it needs the same runaway backstops the Lead Agent has. The subagent middleware chain mirrors three Lead Agent guards (#3875):
- **`LoopDetectionMiddleware`** — breaks a subagent that repeats the same tool call without making progress. Subagents disallow `task`, so only the tool-loop heuristic can fire here. A hard-stop stamps the result `completed` + `subagent_stop_reason=loop_capped`, symmetric to the token budget below. Controlled by the existing `loop_detection` config.
- **`TokenBudgetMiddleware`** — enforces the per-run `subagents.token_budget` ceiling. When the budget is hit the in-flight turn is capped (a final answer is forced) and the result is stamped `completed` + `subagent_stop_reason=token_capped` so the Lead Agent can tell a capped completion from a clean one. Reaching `max_turns` is likewise surfaced as `turn_capped`.
- **`SummarizationMiddleware`** — compacts a long subagent transcript the same way it compacts the Lead Agent's, gated on the same `summarization.enabled` switch so a single config covers both chains.
These guards engage in addition to the `max_turns` and `timeout` limits. The default `max_tokens` for the token budget is coupled to `summarization.enabled` — 1M when compaction is on, 2M when off — but an explicit `subagents.token_budget.max_tokens` (global or per-agent) always wins, so flipping the summarization switch never silently changes a value you pinned.
## ACP agents (external agents)
In addition to the built-in subagents, DeerFlow supports delegating to external agents through the **Agent Client Protocol (ACP)**. ACP allows DeerFlow to invoke agents running as separate processes (including third-party CLI tools wrapped with an ACP adapter).

8
frontend/src/content/zh/harness/middlewares.mdx
View File

@ -17,6 +17,10 @@ import { Callout } from "nextra/components";
这种设计使 Agent 核心保持简单稳定，同时允许丰富的可组合行为分层叠加。
<Callout type="info">
每个子 Agent 运行各自的 Agent 循环，并拥有自己的中间件链。下方的循环检测、token 预算和摘要压缩防护已镜像到子 Agent 链（#3875）；其余中间件为 Lead Agent 专属（如记忆、标题生成、澄清）。参见[子 Agent → 失控行为防护](/docs/harness/subagents#失控行为防护)。
</Callout>
## 链的工作方式
中间件链在每次 Agent 调用时根据当前配置和请求参数构建一次。中间件按定义的顺序运行：
@ -58,6 +62,8 @@ import { Callout } from "nextra/components";
Warning 介入会按 thread 和 run 排队，并在下一次模型调用时合并为一条隐藏的 `HumanMessage(name="loop_warning")`，追加到已有工具结果之后。这样不会破坏 provider 对 tool-call/tool-message 配对的校验。Run 开始和结束时会清理过期或未送达的 warning；达到 hard stop 时仍会清空 tool calls 并强制生成最终文本回复。
此中间件同样挂载到子 Agent 链——子 Agent 不允许调用 `task`，因此这里只会触发工具循环启发式判定，退化的子 Agent 循环会被同样地打破。
**配置**：内置，无需用户配置。
---
@ -137,6 +143,8 @@ token_usage:
当对话变长时，对旧消息进行摘要以减少上下文大小。生成的摘要存入线程状态，并在后续模型调用中作为隐藏的 durable context 数据投影进去，在不把旧消息继续留在活跃 transcript 中的情况下保留含义。
同一个中间件与同一个 `summarization.enabled` 开关也会压缩子 Agent transcript，使一份配置同时覆盖 Lead Agent 与子 Agent 两条链（#3875）。
**配置**：`config.yaml` 中的 `summarization:` 部分。详见下方详细配置。
---

42
frontend/src/content/zh/harness/subagents.mdx
View File

@ -29,15 +29,15 @@ DeerFlow 内置两个子 Agent：
通用推理和执行 Agent，适合委派需要多步骤推理、网络搜索、文件操作和产出物生成的复杂子任务。
- **默认超时**：900 秒（15 分钟）
- **默认最大轮次**：160
- **默认超时**：1800 秒（30 分钟）
- **默认最大轮次**：150
### bash
专门用于在沙箱内执行命令行任务的子 Agent，适合脚本编写、数据处理、文件转换和环境设置任务。
- **默认超时**：900 秒（15 分钟）
- **默认最大轮次**：80
- **默认超时**：1800 秒（30 分钟）
- **默认最大轮次**：60
- **可用性**：仅当沙箱的 `bash` 工具可用时才暴露（`allow_host_bash: true` 或配置了容器沙箱）
## 委派流程
@ -56,32 +56,44 @@ task(
1. 从注册表查找子 Agent 配置，应用任何 `config.yaml` 覆盖。
2. 用子 Agent 自己的提示词和工具创建新的 Agent 调用。
3. 将子 Agent 运行到完成（或直到超时/最大轮次）。
3. 将子 Agent 运行到完成——受 `max_turns`、`timeout` 以及镜像 Lead Agent 的**中间件防护链**（循环检测、token 预算、摘要压缩）共同约束。详见下方[失控行为防护](#失控行为防护)。
4. 将子 Agent 的最终输出作为工具结果返回给 Lead Agent。
## 配置
子 Agent 超时和最大轮次通过 `config.yaml` 中的 `subagents:` 部分控制：
子 Agent 的超时、最大轮次以及单次运行的 token 预算通过 `config.yaml` 中的 `subagents:` 部分控制：
```yaml
subagents:
# 所有子 Agent 的默认超时（秒，默认：900 = 15 分钟）
timeout_seconds: 900
# 所有子 Agent 的默认超时（秒，默认：1800 = 30 分钟）
timeout_seconds: 1800
# 可选：覆盖所有子 Agent 的最大轮次
# 可选：覆盖所有子 Agent 的最大轮次。
# 内置默认值：general-purpose=150、bash=60。留空则保持默认。
# max_turns: 120
# 单次运行的 token 上限——防止子 Agent 在琐碎任务上烧 token 的兜底。
# 触达硬停止时，进行中的那轮会被截断（清空 tool calls，强制 finish_reason
# 为 "stop"），使运行自然完成；结果会被标记为 completed +
# subagent_stop_reason=token_capped，以便 Lead/UI 区分预算截断与正常完成（#3875）。
token_budget:
enabled: true
max_tokens: 2000000 # 宽松的默认值——调低以收紧成本控制
warn_threshold: 0.7 # 预算消耗达到此比例时记录一条 warning
# 可选：按 Agent 覆盖
agents:
general-purpose:
timeout_seconds: 1800 # 复杂任务 30 分钟
max_turns: 160
# token_budget: # 对上面全局 token_budget 的按 Agent 覆盖
# max_tokens: 3000000
bash:
timeout_seconds: 300 # 快速命令 5 分钟
max_turns: 80
```
按 Agent 覆盖优先于全局 `timeout_seconds` 和 `max_turns` 设置。
按 Agent 覆盖优先于全局 `timeout_seconds`、`max_turns` 和 `token_budget` 设置。
## 委派限制
@ -93,6 +105,16 @@ subagents:
如果 Agent 尝试调用超过限制的子 Agent，中间件会裁剪多余的调用。当总量上限耗尽时，它会停止本次 run 的新 `task` 调用，让 Agent 基于已收集结果进行综合。
## 失控行为防护
子 Agent 运行各自的 Agent 循环，因此需要和 Lead Agent 一样的失控兜底。子 Agent 中间件链镜像了 Lead Agent 的三个防护（#3875）：
- **`LoopDetectionMiddleware`** —— 打破子 Agent 不取得进展却反复调用同一工具的循环。子 Agent 不允许调用 `task`，因此这里只会触发工具循环启发式判定。硬停止会将结果标记为 `completed` + `subagent_stop_reason=loop_capped`，与下方 token 预算对称。由现有的 `loop_detection` 配置控制。
- **`TokenBudgetMiddleware`** —— 强制执行 `subagents.token_budget` 单次运行上限。预算触达时，进行中的那轮会被截断（强制产出最终答案），结果标记为 `completed` + `subagent_stop_reason=token_capped`，使 Lead Agent 能区分预算截断与正常完成。触达 `max_turns` 同样以 `turn_capped` 上报。
- **`SummarizationMiddleware`** —— 以和 Lead Agent 相同的方式压缩较长的子 Agent transcript，由同一个 `summarization.enabled` 开关控制，使一份配置同时覆盖两条链。
这些防护与 `max_turns`、`timeout` 限制叠加生效。token 预算的默认 `max_tokens` 与 `summarization.enabled` 联动——压缩开启时为 1M，关闭时为 2M；但显式设置的 `subagents.token_budget.max_tokens`（全局或按 Agent）始终优先，因此切换摘要开关绝不会悄悄改动你已固定的值。
## ACP Agent（外部 Agent）
除内置子 Agent 外，DeerFlow 还通过 **Agent Client Protocol (ACP)** 支持委派给外部 Agent。ACP 允许 DeerFlow 调用作为独立进程运行的 Agent（包括用 ACP 适配器包装的第三方 CLI 工具）。

51
frontend/src/core/api/stream-mode.ts
View File

@ -1,9 +1,7 @@
const SUPPORTED_RUN_STREAM_MODES = new Set([
"values",
"messages",
"messages-tuple",
"updates",
"events",
"debug",
"tasks",
"checkpoints",
@ -11,6 +9,7 @@ const SUPPORTED_RUN_STREAM_MODES = new Set([
] as const);
const warnedUnsupportedStreamModes = new Set<string>();
let warnedUnsupportedStreamResumable = false;
export function warnUnsupportedStreamModes(
modes: string[],
@ -29,40 +28,48 @@ export function warnUnsupportedStreamModes(
}
warn(
`[deer-flow] Dropped unsupported LangGraph stream mode(s): ${unseenModes.join(", ")}`,
`[deer-flow] Rejected unsupported LangGraph stream mode(s): ${unseenModes.join(", ")}`,
);
}
export function sanitizeRunStreamOptions<T>(options: T): T {
if (
typeof options !== "object" ||
options === null ||
!("streamMode" in options)
) {
if (typeof options !== "object" || options === null) {
return options;
}
let sanitizedOptions: T = options;
if ("streamResumable" in options) {
const withoutStreamResumable = { ...options };
delete withoutStreamResumable.streamResumable;
sanitizedOptions = withoutStreamResumable as T;
if (!warnedUnsupportedStreamResumable) {
warnedUnsupportedStreamResumable = true;
console.warn(
"[deer-flow] Dropped unsupported LangGraph run option: streamResumable",
);
}
}
if (!("streamMode" in options)) {
return sanitizedOptions;
}
const streamMode = options.streamMode;
if (streamMode == null) {
return options;
return sanitizedOptions;
}
const requestedModes = Array.isArray(streamMode) ? streamMode : [streamMode];
const sanitizedModes = requestedModes.filter((mode) =>
SUPPORTED_RUN_STREAM_MODES.has(mode),
);
if (sanitizedModes.length === requestedModes.length) {
return options;
}
const droppedModes = requestedModes.filter(
(mode) => !SUPPORTED_RUN_STREAM_MODES.has(mode),
);
warnUnsupportedStreamModes(droppedModes);
if (droppedModes.length > 0) {
warnUnsupportedStreamModes(droppedModes);
throw new Error(
`[deer-flow] Unsupported LangGraph stream mode(s): ${droppedModes.join(", ")}`,
);
}
return {
...options,
streamMode: Array.isArray(streamMode) ? sanitizedModes : sanitizedModes[0],
};
return sanitizedOptions;
}

41
frontend/src/core/threads/hooks.ts
View File

@ -44,11 +44,6 @@ import type {
ThreadTokenUsageResponse,
} from "./types";
export type ToolEndEvent = {
name: string;
data: unknown;
};
export type ThreadStreamOptions = {
threadId?: string | null | undefined;
displayThreadId?: string | null | undefined;
@ -57,7 +52,6 @@ export type ThreadStreamOptions = {
onSend?: (threadId: string) => void;
onStart?: (threadId: string, runId: string) => void;
onFinish?: (state: AgentThreadState) => void;
onToolEnd?: (event: ToolEndEvent) => void;
};
type SendMessageOptions = {
@ -96,6 +90,27 @@ type RegeneratePrepareResponse = {
target_run_id: string;
};
export function hasToolResult(messages: Message[], toolName: string): boolean {
const matchingToolCallIds = new Set<string>();
for (const message of messages) {
if (message.type !== "ai") {
continue;
}
for (const toolCall of message.tool_calls ?? []) {
if (toolCall.name === toolName && toolCall.id) {
matchingToolCallIds.add(toolCall.id);
}
}
}
return messages.some(
(message) =>
message.type === "tool" &&
(message.name === toolName ||
matchingToolCallIds.has(message.tool_call_id)),
);
}
export function buildThreadSubmitMessages({
text,
additionalKwargs,
@ -962,7 +977,6 @@ export function useThreadStream({
onSend,
onStart,
onFinish,
onToolEnd,
}: ThreadStreamOptions) {
const { t } = useI18n();
const currentViewThreadId = displayThreadId ?? threadId ?? null;
@ -993,7 +1007,6 @@ export function useThreadStream({
onSend,
onStart,
onFinish,
onToolEnd,
});
const {
@ -1008,8 +1021,8 @@ export function useThreadStream({
// Keep listeners ref updated with latest callbacks
useEffect(() => {
listeners.current = { onSend, onStart, onFinish, onToolEnd };
}, [onSend, onStart, onFinish, onToolEnd]);
listeners.current = { onSend, onStart, onFinish };
}, [onSend, onStart, onFinish]);
useEffect(() => {
const normalizedThreadId = threadId ?? null;
@ -1100,14 +1113,6 @@ export function useThreadStream({
.catch(() => ({}));
}
},
onLangChainEvent(event) {
if (event.event === "on_tool_end") {
listeners.current.onToolEnd?.({
name: event.name,
data: event.data,
});
}
},
onUpdateEvent(data) {
const _messages = getSummarizationMiddlewareMessages(data);
if (_messages && _messages.length >= 2) {

71
frontend/tests/unit/core/api/stream-mode.test.ts
View File

@ -2,33 +2,34 @@ import { expect, test } from "@rstest/core";
import { sanitizeRunStreamOptions } from "@/core/api/stream-mode";
test("drops unsupported stream modes from array payloads", () => {
const sanitized = sanitizeRunStreamOptions({
streamMode: [
"values",
"messages-tuple",
"custom",
"updates",
"events",
"tools",
],
});
expect(sanitized.streamMode).toEqual([
"values",
"messages-tuple",
"custom",
"updates",
"events",
]);
test("rejects mixed supported and unsupported stream modes", () => {
expect(() =>
sanitizeRunStreamOptions({
streamMode: ["values", "events", "tools"],
}),
).toThrow("Unsupported LangGraph stream mode(s): events, tools");
});
test("drops unsupported stream modes from scalar payloads", () => {
const sanitized = sanitizeRunStreamOptions({
streamMode: "tools",
});
test("rejects payloads when every requested stream mode is unsupported", () => {
expect(() =>
sanitizeRunStreamOptions({
streamMode: ["events", "tools"],
}),
).toThrow("Unsupported LangGraph stream mode(s): events, tools");
expect(sanitized.streamMode).toBeUndefined();
expect(() =>
sanitizeRunStreamOptions({
streamMode: "tools",
}),
).toThrow("Unsupported LangGraph stream mode(s): tools");
});
test("rejects messages because the Gateway only supports messages-tuple framing", () => {
expect(() =>
sanitizeRunStreamOptions({
streamMode: "messages",
}),
).toThrow("Unsupported LangGraph stream mode(s): messages");
});
test("keeps payloads without streamMode untouched", () => {
@ -38,3 +39,25 @@ test("keeps payloads without streamMode untouched", () => {
expect(sanitizeRunStreamOptions(options)).toBe(options);
});
test("strips streamResumable before sending run options to the API", () => {
const sanitized = sanitizeRunStreamOptions({
streamResumable: true,
streamSubgraphs: true,
});
expect(sanitized).toEqual({
streamSubgraphs: true,
});
});
test("sanitizes streamResumable while preserving valid stream modes", () => {
const sanitized = sanitizeRunStreamOptions({
streamResumable: true,
streamMode: ["values", "custom"],
});
expect(sanitized).toEqual({
streamMode: ["values", "custom"],
});
});

95
frontend/tests/unit/core/threads/stream-options.test.ts Normal file
View File

@ -0,0 +1,95 @@
import { afterEach, expect, test, rs } from "@rstest/core";
async function captureThreadStreamOptions() {
let capturedOptions: Record<string, unknown> | undefined;
rs.resetModules();
rs.doMock("react", () => ({
useCallback: <T extends (...args: never[]) => unknown>(callback: T) =>
callback,
useEffect: () => undefined,
useMemo: <T>(factory: () => T) => factory(),
useRef: <T>(initialValue: T) => ({ current: initialValue }),
useState: <T>(initialValue: T | (() => T)) => [
typeof initialValue === "function"
? (initialValue as () => T)()
: initialValue,
rs.fn(),
],
}));
rs.doMock("@tanstack/react-query", () => ({
useInfiniteQuery: () => ({
data: { pages: [] },
error: null,
fetchNextPage: rs.fn(),
hasNextPage: false,
isFetchingNextPage: false,
isLoading: false,
}),
useMutation: rs.fn(),
useQuery: rs.fn(),
useQueryClient: () => ({
invalidateQueries: rs.fn(),
setQueriesData: rs.fn(),
}),
}));
rs.doMock("@langchain/langgraph-sdk/react", () => ({
useStream: (options: Record<string, unknown>) => {
capturedOptions = options;
return {
isLoading: false,
messages: [],
stop: rs.fn(),
submit: rs.fn(),
values: { title: "", messages: [] },
};
},
}));
rs.doMock("@/core/api", () => ({
getAPIClient: () => ({}),
}));
rs.doMock("@/core/i18n/hooks", () => ({
useI18n: () => ({
t: {
pages: { newChat: "New chat" },
uploads: { uploadingFiles: "Uploading files" },
},
}),
}));
rs.doMock("@/core/tasks/context", () => ({
useUpdateSubtask: () => rs.fn(),
}));
const { useThreadStream } = await import("@/core/threads/hooks");
function ThreadStreamCapture() {
useThreadStream({
context: {
mode: "flash",
},
isMock: true,
} as never);
return null;
}
ThreadStreamCapture();
return capturedOptions;
}
afterEach(() => {
rs.doUnmock("react");
rs.doUnmock("@tanstack/react-query");
rs.doUnmock("@langchain/langgraph-sdk/react");
rs.doUnmock("@/core/api");
rs.doUnmock("@/core/i18n/hooks");
rs.doUnmock("@/core/tasks/context");
rs.resetModules();
});
test("does not subscribe to unsupported LangGraph events mode", async () => {
const options = await captureThreadStreamOptions();
expect(options).toBeDefined();
expect(options).not.toHaveProperty("onLangChainEvent");
expect(options).toHaveProperty("onUpdateEvent");
expect(options).toHaveProperty("onCustomEvent");
});

60
frontend/tests/unit/core/threads/tool-result.test.ts Normal file
View File

@ -0,0 +1,60 @@
import type { Message } from "@langchain/langgraph-sdk";
import { expect, test } from "@rstest/core";
import { hasToolResult } from "@/core/threads/hooks";
test("recognizes a completed tool from its ToolMessage", () => {
const messages = [
{
type: "ai",
content: "",
tool_calls: [{ id: "call-1", name: "setup_agent", args: {} }],
},
{
type: "tool",
content: "Agent saved",
name: "setup_agent",
tool_call_id: "call-1",
},
] as Message[];
expect(hasToolResult(messages, "setup_agent")).toBe(true);
});
test("does not treat a pending call or another tool result as completed", () => {
const pending = [
{
type: "ai",
content: "",
tool_calls: [{ id: "call-1", name: "setup_agent", args: {} }],
},
] as Message[];
const otherTool = [
{
type: "tool",
content: "Done",
name: "web_search",
tool_call_id: "call-2",
},
] as Message[];
expect(hasToolResult(pending, "setup_agent")).toBe(false);
expect(hasToolResult(otherTool, "setup_agent")).toBe(false);
});
test("matches a ToolMessage without a name through its tool call id", () => {
const messages = [
{
type: "ai",
content: "",
tool_calls: [{ id: "call-1", name: "setup_agent", args: {} }],
},
{
type: "tool",
content: "Agent saved",
tool_call_id: "call-1",
},
] as Message[];
expect(hasToolResult(messages, "setup_agent")).toBe(true);
});