github/deer-flow
1
0
Fork 0
mirror of https://github.com/bytedance/deer-flow.git synced 2026-07-25 15:37:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' into rayhpeng/hexagonal-feedback-slice

Browse Source 
Resolve the one import conflict in thread_runs.py: keep main's new
checkpoint_lineage imports (#4358 regenerate-in-branched-threads fix)
alongside this branch's get_feedback_service (feedback domain-service
refactor). All other files auto-merged. Verified: backend feedback/
regenerate/branch/lineage tests (112 passed) and the full frontend suite.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
rayhpeng 2026-07-24 09:55:36 +08:00
commit 02a0a57609
84 changed files with 8327 additions and 471 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

134
.agent/skills/engineer-system-change/SKILL.md Normal file
View File

@ -0,0 +1,134 @@
---
name: engineer-system-change
description: Evaluate and carry out non-trivial software-system changes from first principles. Use when assessing RFCs, issues, designs, features, refactors, migrations, dependency changes, or proposed fields, events, APIs, modules, and services whose need, consumers, system fit, validation, or rollback require scrutiny. Read the actual system, identify the concrete problem and named semantic consumers, choose the smallest sufficient solution, reject pseudo-requirements and speculative abstractions, and require evidence proportional to risk. Do not use for mechanical edits, source-code explanation, or a dedicated review of an already-complete diff.
---
# Engineer System Change
Treat every proposed change as a hypothesis about a real system, not as an implementation checklist. Establish whether the change should exist before designing or building it, then keep the solution and the process proportional to the risk.
## Preserve the Task Boundary
- If asked only to assess, review, or plan, make no project or external-state changes.
- If explicitly asked to implement, including after an assessment, pass the decision gates before editing and verify the result afterward.
- Treat implementation permission as separate from permission to commit, push, deploy, publish, or update issues and pull requests.
- If repository truth matters, inspect the current target revision and relevant discussion. Do not rely on a stale checkout, an RFC alone, or remembered architecture.
- Separate verified facts, inferences, and unknowns. Do not turn missing evidence into a confident conclusion.
## Apply the Decision Gates
### 1. Ground the Problem
- Trace the current user workflow, failure, or code path before proposing a solution.
- State the undesirable observable behavior and the invariant or outcome that should replace it.
- Identify who is affected and which concrete decision or action changes.
- Check whether the existing system, configuration, documentation, or operating procedure already solves the problem.
- Enumerate adjacent product paths and workarounds, not only the proposed target surface. Explain precisely which accepted outcome each alternative fails; do not claim “the only option” from one missing UI control or code path.
- Treat an absent field, interface, abstraction, or standard as an observation, not proof of a requirement.
Return `STOP` only when evidence affirmatively shows that no change is needed or the affected workflow already achieves the outcome. Return `NEEDS_EVIDENCE` when an unverified fact prevents the decision.
### 2. Name Semantic Consumers
For every proposed durable field, event, API, table, store, module, service, or workflow, establish:
| Question | Required answer |
| --- | --- |
| Producer or lifecycle owner | What creates, updates, or owns it? |
| Committed consumer | Which named caller, component, operator, or user reads or acts on it now or as part of this same accepted slice? |
| Semantic use | What behavior, decision, or externally visible result changes after consumption? |
| Reachable path | Where does production reach consumption in the current system or proposed slice? |
| Absence test | Which verified scenario or accepted outcome fails if the addition is removed? |
Accept a proposed consumer only when it is tied to a verified current need and committed integration in the same change. Do not accept a roadmap, possible future evaluator, generic read/debug API, storage alone, or “future flexibility” as a semantic consumer. If an addition has no such consumer, remove or defer it.
Treat a public or externally consumed contract as a compatibility boundary even when no in-repository caller is visible. Absence of a discoverable caller is uncertainty, not proof that no consumer exists.
### 3. Choose the Smallest Sufficient Change
Consider solutions in this order and stop at the first one that fully satisfies the verified outcome and invariants without shifting disproportionate recurring cost, coupling, or risk downstream:
1. No product or code change
2. Documentation, configuration, or operating procedure
3. Reuse an existing capability
4. Make a local behavior fix
5. Extend an existing abstraction
6. Introduce a new abstraction
7. Introduce a new subsystem or migration path
- Minimize concepts, states, interfaces, irreversible decisions, and maintenance surface, not literal line count.
- Require a second current consumer, a demonstrated variation, or a hard boundary before generalizing a local solution.
- Prefer independently reversible slices over a comprehensive architecture rollout.
- Distinguish a real problem from an oversized solution. A valid verdict is: “The problem is real; reduce the proposal to this smaller change.”
- Apply these gates recursively to your own recommendation. Do not propose a new field, contract, abstraction, migration, or validation system without naming its consumer, checking existing mechanisms, and showing why a smaller change is insufficient.
### 4. Map Consequences and Verification Proportionally
Inspect only relevant dimensions, but do not omit a dimension merely because the proposal omits it:
- callers and downstream consumers
- API, data, event, and UI contracts
- authorization, ownership, privacy, and trust boundaries
- persistence, migrations, replay, and side effects
- concurrency, ordering, retries, idempotency, and failure recovery
- compatibility, dependencies, performance, deployment, and operations
- observability and rollback
For state replay or retry features, explicitly distinguish restored application state from external side effects that cannot be undone.
Label material risk claims as `VERIFIED`, `INFERENCE`, or `UNKNOWN`. Use an inference to request a focused check, not to require new architecture as though the claim were already proven.
Evidence labels classify individual claims; verdicts classify the overall decision. An `UNKNOWN` requires `NEEDS_EVIDENCE` only when the unknown blocks a material decision.
Before implementation, require observed evidence for the current-system claims that justify the decision and a proportional, executable verification plan. Treat proposed checks as a verification plan, not observed evidence.
### 5. Implement Only the Justified Slice
When implementation is authorized:
- Reproduce the baseline first. Encode it as a failing behavioral test when executable; otherwise state why and record a reproducible check.
- Change only the paths required by the accepted outcome and consumers.
- Reuse existing execution paths and contracts when they preserve the required semantics.
- Avoid speculative compatibility layers, selectors, shadow systems, canaries, or dual stacks unless an irreversible or high-risk transition requires them.
- Update repository guidance only when architecture, commands, or durable conventions actually change.
### 6. Prove the Result
- Map each important result claim to observed evidence: tests, contract checks, static analysis, runtime traces, benchmarks, or a reproducible manual check.
- Do not use the agent's own summary as proof.
- Verify negative boundaries and failure behavior, not only the happy path.
- State what remains unverified and how that uncertainty affects the verdict.
- Use focused regression checks for local reversible changes.
- Add targeted integration and adversarial checks for contract, persistence, security, concurrency, replay, or cross-component changes.
- Require a production-like rehearsal plus executable containment or rollback for irreversible changes or materially high-risk external side effects.
## Use Explicit Verdicts
- `STOP`: evidence affirmatively shows that no current change is needed, or that existing capability already achieves the accepted outcome.
- `REDUCE`: the problem is real, but the proposed scope or abstraction exceeds the evidence.
- `REVISE`: the problem and approximate scope are justified, but a correctness, contract, or failure-semantics defect must change before proceeding.
- `PROCEED`: the problem, consumers, minimum solution, consequences, and proportional verification plan are sufficiently established.
- `NEEDS_EVIDENCE`: a decision would be guesswork until a specific fact, code path, incident, or consumer is verified.
Do not force a binary approve/reject judgment when evidence is incomplete.
Choose the verdict from the condition blocking the earliest gate, not from the gate number: affirmative evidence that no change is needed maps to `STOP`; a decision-blocking unknown maps to `NEEDS_EVIDENCE`; a real problem with unsupported scope or no committed consumer maps to `REDUCE`; and a confirmed correctness, contract, or failure-semantics defect maps to `REVISE`. Use `PROCEED` only when no gate is blocked.
When more than one condition applies, give one primary verdict and list the other required changes without inventing a compound status.
A verdict records the decision gate and never expands authorization. After authorized implementation, separately report the implemented slice, observed verification, and remaining uncertainty.
## Keep the Output Proportional
For a simple change, report only:
1. Problem and desired behavior
2. Named semantic consumer
3. Smallest sufficient change
4. Observed evidence and, before implementation, the verification plan
5. Verdict
For a cross-boundary change or RFC, add:
- verified current-system facts
- existing alternatives and why they do not meet the accepted outcome
- consumer ledger for proposed additions
- affected contracts and side effects
- rejected or deferred scope with reasons
- failure, observation, and rollback strategy
Do not create ceremonial documents or exhaustive matrices when a short evidence-backed answer is sufficient. The workflow itself must not become overengineering.

8
README.md
View File

@ -332,6 +332,8 @@ DeerFlow runs the agent runtime inside the Gateway API. Development mode enables
Gateway owns `/api/langgraph/*` and translates those public LangGraph-compatible paths to its native `/api/*` routers behind nginx.
DeerFlow's built-in custom events are available through both LangGraph streaming interfaces: native clients can continue subscribing to `stream_mode="custom"`, while callback-based integrations can consume the same payloads as `on_custom_event` records from `astream_events(version="v2")`. The callback event name matches the payload's `type` field.
#### Docker Production Deployment
`deploy.sh` supports building and starting separately:
@ -672,6 +674,8 @@ uv run python -m deerflow.skills.review.cli ../skills/public/data-analysis --for
Tools follow the same philosophy. DeerFlow comes with a core toolset — web search, web fetch, rendered web capture, file operations, bash execution — and supports custom tools via MCP servers and Python functions. Swap anything. Add anything.
Advanced deployments can enable pluggable tool authorization with `authorization.enabled` in `config.yaml`. A configured `AuthorizationProvider` filters denied tools before they reach the model or deferred-tool catalog, then the same provider is checked again before every business-tool execution through the existing guardrail middleware. A generated `tool_search` may bypass that second check only when it fronts the current build's already-filtered deferred catalog. The built-in RBAC provider supports per-role tool allow/deny policies and validates that `default_role` names a configured role; authorization is disabled by default. See `config.example.yaml` and the [authorization RFC](docs/plans/2026-07-10-pluggable-authorization-rfc.md).
Advanced deployments can also extend the agent runtime itself by declaring zero-argument `AgentMiddleware` classes under `extensions.middlewares` in `config.yaml` or `extensions_config.json`. DeerFlow loads the same configured class list into the lead-agent and subagent pipelines after their built-in runtime middlewares and loop/token guards, but before the terminal-response/safety/clarification tail, so enterprise forks can add domain guardrails, tool-call governance, or observability hooks without patching the built-in middleware builders. Missing packages, invalid classes, and broken modules fail loudly at agent creation. Treat `config.yaml` and `extensions_config.json` as trusted operator-controlled files: middleware paths are code execution, just like custom tool, model, sandbox, guardrail, MCP server, and MCP interceptor declarations. Gateway skill/MCP toggle endpoints preserve this field but do not expose an API write path for `extensions.middlewares`. Per-context parameterization and separate lead-only/subagent-only middleware lists are not supported yet.
Gateway-generated follow-up suggestions now normalize both plain-string model output and block/list-style rich content before parsing the JSON array response, so provider-specific content wrappers do not silently drop suggestions.
@ -689,7 +693,9 @@ Interrupted first-turn runs still persist a fallback conversation title, so stop
Streaming Markdown responses animate only newly arrived words; text that is already visible is not faded out and replayed when the next chunk extends the same block.
In the Web UI, completed assistant turns can be branched into a new main conversation. The new thread starts from that turn's checkpoint. Because workspace files are not checkpointed, the branch only receives a best-effort copy of the current workspace when you branch from the latest turn; branching from an older turn keeps just the restored message history so the branch never inherits files that were created in a later part of the conversation.
In the Web UI, completed assistant turns can be branched into a new main conversation. The new thread starts from that turn's checkpoint and keeps the preceding replay checkpoint, so the branched response can be regenerated immediately. Legacy or imported histories without checkpoint parent links use a bounded chronological fallback; if no earlier replay checkpoint exists, branching still succeeds with the legacy single-checkpoint shape, while regeneration remains unavailable for that inherited response. Existing single-checkpoint branches are left unchanged rather than attempting an unsafe checkpoint copy. Because workspace files are not checkpointed, the branch only receives a best-effort copy of the current workspace when you branch from the latest turn; branching from an older turn keeps just the restored message history so the branch never inherits files that were created in a later part of the conversation.
The Web UI reports completed task time once per run. This is total wall-clock time—including model reasoning, tool calls, and waiting—not a per-step or model-only thinking duration. Reasoning content remains available through its own separate disclosure.
Web UI chat links percent-encode custom thread identifiers before placing them in route segments, so reserved URL characters such as `#` and `?` do not change which conversation is opened.

2
README_zh.md
View File

@ -548,7 +548,7 @@ Tools 也是同样的思路。DeerFlow 自带一组核心工具：网页搜索
Gateway 生成后续建议时，现在会先把普通字符串输出和 block/list 风格的富文本内容统一归一化，再去解析 JSON 数组响应，因此不同 provider 的内容包装方式不会再悄悄把建议吞掉。
Web UI 支持从已完成的 assistant 回复分叉出一个新的主对话。新 thread 会从该回复对应的 checkpoint 开始，并尽力复制当前 thread 的工作区文件。
Web UI 支持从已完成的 assistant 回复分叉出一个新的主对话。新 thread 会保留该轮回复的 checkpoint 以及用户消息之前的重放 checkpoint，因此分叉后可以立即重新生成该回复。对于缺少 checkpoint 父链接的旧历史或导入历史，Gateway 会进行有界的时间顺序查找；如果不存在更早的重放 checkpoint，分叉仍会按旧版单-checkpoint 形态成功创建，但无法重新生成继承的回复。已有的单-checkpoint 分叉会保持不变，不会通过不安全的 checkpoint 复制尝试修复。只有从最新回合分叉时才会尽力复制当前 thread 的工作区文件；从历史回合分叉不会带入后续时间线创建的文件。
```text
# sandbox 容器内的路径

66
backend/AGENTS.md
View File

@ -15,6 +15,7 @@ DeerFlow is a LangGraph-based AI super agent system with a full-stack architectu
**Runtime**:
- `make dev`, Docker dev, and production all run the agent runtime in Gateway via `RunManager` + `run_agent()` + `StreamBridge` (`packages/harness/deerflow/runtime/`). Nginx exposes that runtime at `/api/langgraph/*` and rewrites it to Gateway's native `/api/*` routers.
- Gateway streams `write_file` and `str_replace` argument deltas in bounded batches when clients also subscribe to `values`; messages-only consumers retain the original per-chunk contract, while `values` preserves the complete tool call.
- With `stream_subgraphs`, subgraph frames keep their namespace in the SSE event name (`values|<ns>`, LangGraph Platform style) instead of impersonating root frames — a delegated subagent inherits the parent checkpoint namespace, so publishing its `values` snapshot as bare `values` replaces the whole thread view in SDK clients (#4399). Root-only consumers (file-tool chunk batcher, subagent event persistence, LLM error-fallback detection) ignore namespaced frames. The web frontend does not request subgraph streaming; subtask progress rides root-namespace `task_*` custom events.
- Scheduled-task executions must reuse that same Gateway run lifecycle. The scheduler may decide *when* work runs, but it must dispatch through the existing run path rather than introducing a parallel execution stack.
**Project Structure**:
@ -315,13 +316,13 @@ Lead-agent middlewares are assembled in strict order across three functions: the
6. **SandboxMiddleware** - Acquires sandbox, stores `sandbox_id` in state
7. **DanglingToolCallMiddleware** - Injects placeholder ToolMessages for AIMessage tool_calls that lack responses (e.g., user interruption), preserving raw provider tool-call payloads in `additional_kwargs["tool_calls"]`; malformed tool-call names and arguments are sanitized in the model-bound request so strict OpenAI-compatible providers do not reject the next request
8. **LLMErrorHandlingMiddleware** - Normalizes provider/model invocation failures into recoverable assistant-facing errors before later stages run
9. **GuardrailMiddleware** - *(optional, if `guardrails.enabled`)* Pre-tool-call authorization via pluggable `GuardrailProvider`; returns an error ToolMessage on deny. Providers: built-in `AllowlistProvider` (zero deps), OAP policy providers (e.g. `aport-agent-guardrails`), or custom. See [docs/GUARDRAILS.md](docs/GUARDRAILS.md)
9. **Authorization / GuardrailMiddleware** - Up to two independent pre-tool-call gates run here. When `authorization.enabled`, the `AuthorizationProvider` instance already used for Layer 1 capability filtering is wrapped by `GuardrailAuthorizationAdapter` and reused for Layer 2 execution checks. A generated `tool_search` bypasses the adapter's second provider call only when the current build has a concrete deferred setup; its catalog was already filtered by Layer 1, and an ordinary same-named tool without that deferred setup receives no exemption. When `guardrails.enabled`, the explicitly configured `GuardrailProvider` is appended after authorization and still evaluates every call, including `tool_search`. Authorization therefore runs outermost and can deny before an external guardrail call; both use the existing middleware's fail-closed, audit, sync/async, and error-`ToolMessage` behavior. See the authorization RFC and [docs/GUARDRAILS.md](docs/GUARDRAILS.md).
10. **SandboxAuditMiddleware** - Audits sandboxed shell/file operations for security logging before tool execution
11. **ReadBeforeWriteMiddleware** - *(optional, if `read_before_write.enabled`, default on)* Outermost write gate (issue #3857): `read_file` stamps a content hash onto its ToolMessage; `write_file` (append/overwrite-existing) and `str_replace` are blocked unless the newest mark for that path matches the file's current hash. Sits outside ToolProgressMiddleware and ToolErrorHandlingMiddleware so a blocked write returns immediately without consuming a ToolProgress slot. Blocked results call `normalize_tool_result` directly to stamp `deerflow_tool_meta` (`recoverable_by_model=True`) before returning, keeping the result well-formed for any outer consumer. Marks live on messages, so summarization dropping the read result invalidates the gate automatically; writes never refresh marks, forcing a re-read between consecutive edits. Gate check + tool execution are serialized per (thread, path) so same-turn parallel writes cannot reuse one stale mark; on sandboxes whose `read_file` reports failures as `"Error: ..."` strings instead of raising (AIO/E2B), uninspectable targets fail open (creation proceeds, no mark stamped)
12. **ToolProgressMiddleware** - *(optional, if `tool_progress.enabled`)* State-machine-based stagnation guard (RFC #3177). Outer wrapper around ToolErrorHandlingMiddleware so its `wrap_tool_call` receives results already stamped with `deerflow_tool_meta`. Tracks per-(thread, tool) consecutive "no-new-info" calls across three error categories: (a) `recoverable_by_model=True` (no_results, not_found, permission, Jaccard-duplicate success): ACTIVE → WARNED (terminal — hint re-injected on each subsequent problem); (b) `recoverable_by_model=False, action≠stop` (rate_limited, transient): ACTIVE → WARNED → BLOCKED after `warn_escalation_count` more problems; (c) `recoverable_by_model=False, action=stop` (auth, config, internal): immediately BLOCKED on first occurrence. **Division of labor with LoopDetectionMiddleware:** ToolProgressMiddleware is a result-quality guard — fires after tool execution and blocks specific tools that stop producing new information; LoopDetectionMiddleware is a call-pattern guard — fires after the model responds and hard-stops the whole turn when the model repeatedly issues identical tool_calls. Both can inject HumanMessage hints in the same model call without conflict; neither reads the other's internal state.
13. **ToolErrorHandlingMiddleware** - Receives `AppConfig`, converts tool exceptions into error `ToolMessage`s so the run can continue instead of aborting, stamps every result with `deerflow_tool_meta` (status / error_type / recoverable_by_model / recommended_next_action / source) via `tool_result_meta.normalize_tool_result`, stamps structured metadata for task exception wrappers, and stamps skill-read metadata for downstream durable-context capture. Task tool result text is generated from the same status/result/error inputs as the structured metadata so callers do not hand-write a second protocol string.
Authorization identity plumbing is independent of whether authorization enforcement is enabled. Gateway removes client-supplied `is_internal` / `authz_attributes` / `channel_user_id`, derives `is_internal` only from the server-owned `request.state.auth_source`, and accepts `channel_user_id` only from an internally authenticated IM caller's top-level `body.context`; free-form `body.config` can never supply it. `build_principal_from_context` is the shared Principal builder for assembly-time authorization and `GuardrailAuthorizationAdapter`; it applies `default_role`, strict-boolean internal provenance, and copy-on-read `authz_attributes`. Task delegation carries `is_internal` plus copied attributes through `SubagentExecutor`, while `GuardrailMiddleware` maps the same runtime fields into `GuardrailRequest`. Phase 1A provides this trusted identity chain only; automatic Layer 1 filtering and Layer 2 authorization middleware wiring remain separate enforcement work.
Authorization identity plumbing is independent of whether authorization enforcement is enabled. Gateway removes client-supplied `is_internal` / `authz_attributes` / `channel_user_id`, derives `is_internal` only from the server-owned `request.state.auth_source`, and accepts `channel_user_id` only from an internally authenticated IM caller's top-level `body.context`; free-form `body.config` can never supply it. `build_principal_from_context` is the shared Principal builder for assembly-time authorization and `GuardrailAuthorizationAdapter`; it applies `default_role`, strict-boolean internal provenance, and copy-on-read `authz_attributes`. The built-in RBAC provider validates `authorization.default_role` during provider resolution so an unknown fallback role fails agent construction instead of degrading into an empty tool set. Task delegation carries `is_internal` plus copied attributes through `SubagentExecutor`, while `GuardrailMiddleware` maps the same runtime fields into `GuardrailRequest`. Phase 1B applies Layer 1 before deferred-tool assembly on the lead, native-subagent, and embedded-client paths, then passes the same provider instance into Layer 2. Framework-provided `describe_skill` and memory tools are included in Layer 1 but restored to their legacy post-`tool_search` ordering afterward. `DeerFlowClient.stream()` treats its in-process caller as trusted and accepts the same identity fields as keyword overrides; it includes the complete Principal in its agent cache key and deep-copies nested attributes so caller mutation cannot make a stale tool set look current.
Before changing a later authorization phase, read the [authorization RFC](../docs/plans/2026-07-10-pluggable-authorization-rfc.md) and its [implementation notes](../docs/plans/2026-07-10-pluggable-authorization-implementation-notes.md). The notes are the cumulative handoff record for merged PR behavior, reviewer feedback, trust-boundary decisions, deferred scope, and required regression coverage.
@ -346,7 +347,7 @@ Before changing a later authorization phase, read the [authorization RFC](../doc
30. **Custom middlewares** - *(optional)* Any `custom_middlewares` passed to `build_middlewares` are injected here, before config-declared extensions and the terminal-response/safety/clarification tail
31. **Configured extension middlewares** - *(optional, if `extensions.middlewares` is set in `config.yaml` or `extensions_config.json`)* Zero-argument `AgentMiddleware` classes loaded from `module.path:ClassName` entries via `deerflow.reflection.resolve_class`. Missing packages, invalid classes, and broken modules fail loudly at agent creation. These run after built-ins/programmatic custom middleware and after the lead/subagent loop/token guards, but before the terminal-response/safety/clarification tail; subagents receive the same configured extension middleware class list before their safety tail. Treat these files as trusted operator config because middleware paths instantiate arbitrary code. Gateway skill/MCP toggle endpoints preserve this field through `to_file_dict()` but must not add a write path for `extensions.middlewares` without an explicit trust-boundary review. Lead-only vs subagent-only middleware lists and per-context constructor parameters are not expressible in this MVP.
32. **TerminalResponseMiddleware** - When a provider returns an empty terminal `AIMessage` after tool execution, injects a hidden recovery prompt and retries the model once; a second empty response is replaced in checkpoint state by a visible error fallback marked for the run worker, so the run finishes as an error instead of a silent success
33. **SafetyFinishReasonMiddleware** - *(optional, if `safety_finish_reason.enabled`)* Suppresses tool execution when the provider safety-terminated the response (e.g. `finish_reason=content_filter`); registered after terminal-response/custom/configured middlewares so LangChain's reverse-order `after_model` dispatch runs it first
33. **SafetyFinishReasonMiddleware** - *(optional, if `safety_finish_reason.enabled`)* Repairs AIMessages the provider safety-terminated (e.g. `finish_reason=content_filter`): strips truncated tool calls so they are not executed (#3028), and — when the response is otherwise blank (no tool calls, no visible content) — backfills a user-facing explanation so the empty message is not persisted and then rejected by strict OpenAI-compatible providers on the next request (`message ... with role 'assistant' must not be empty`), which would otherwise strand the whole thread (#4393). A safety-terminated response that still carries visible text is left untouched. Registered after terminal-response/custom/configured middlewares so LangChain's reverse-order `after_model` dispatch runs it first
34. **ClarificationMiddleware** - Intercepts `ask_clarification` tool calls, writes a readable `ToolMessage.content` fallback plus structured `ToolMessage.artifact.human_input` request payload, and interrupts via `Command(goto=END)` (must be last). Because this middleware can short-circuit tool execution before LangChain emits `on_tool_end`, `RunJournal` performs a root-run final reconciliation for allowlisted clarification `ToolMessage`s whose `tool_call_id` was produced by the current run, so human-input request cards remain recoverable from `run_events` after checkpoint compaction. Human Input Card replies are submitted as `hide_from_ui` `HumanMessage`s with `additional_kwargs.human_input_response`; `RunJournal` persists only allowlisted hidden response sources (currently `ask_clarification`) as `llm.human.input`, which preserves answered-card state after compaction without exposing generic internal hidden context.
### Configuration System
@ -416,7 +417,7 @@ Localhost persistence deliberately reads the direct request `Host` and ignores `
| **Skills** (`/api/skills`) | `GET /` - list skills; `GET /{name}` - details; `PUT /{name}` - update enabled; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`); `POST /reload` - admin-only process-local prompt-cache invalidation after trusted external filesystem changes |
| **Memory** (`/api/memory`) | `GET /` - memory data; `POST /reload` - force reload; `GET /config` - config; `GET /status` - config + data |
| **Uploads** (`/api/threads/{id}/uploads`) | `POST /` - upload files (auto-converts PDF/PPT/Excel/Word); `GET /list` - list; `DELETE /{filename}` - delete |
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; `POST /branches` - create a new main-thread branch from a completed assistant turn checkpoint. Workspace files are not checkpointed, so the branch only best-effort copies the current workspace when branching from the **latest** turn (`workspace_clone_mode="current_thread_best_effort"`); branching from an older/historical turn skips the copy (`workspace_clone_mode="skipped_historical_turn"`) so the branch never inherits files that only exist in a later timeline; `GET /goal`, `PUT /goal`, `DELETE /goal` - read, set, and clear the active thread goal; `POST /compact` - manually summarize older active context into `summary_text` and retain the recent message window, blocked while a run is in flight; unexpected failures are logged server-side and return a generic 500 detail |
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; `POST /branches` - create a new main-thread branch from a completed assistant turn checkpoint and, when an addressable pre-user replay checkpoint exists, materialize it into the branch namespace so the inherited response remains regeneratable. Workspace files are not checkpointed, so the branch only best-effort copies the current workspace when branching from the **latest** turn (`workspace_clone_mode="current_thread_best_effort"`); branching from an older/historical turn skips the copy (`workspace_clone_mode="skipped_historical_turn"`) so the branch never inherits files that only exist in a later timeline. Branch creation also seeds the new thread's run-event feed from the branch checkpoint's visible messages (`history_seed_mode` in the response): the thread feed reads run_events, not checkpoints, so without the seed the inherited history disappears from the UI after the branch's first run (#4380); `GET /goal`, `PUT /goal`, `DELETE /goal` - read, set, and clear the active thread goal; `POST /compact` - manually summarize older active context into `summary_text` and retain the recent message window, blocked while a run is in flight; unexpected failures are logged server-side and return a generic 500 detail |
| **Artifacts** (`/api/threads/{id}/artifacts`) | `GET /{path}` - serve artifacts; active content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) are always forced as download attachments to reduce XSS risk; `?download=true` still forces download for other file types |
| **Suggestions** (`/api/suggestions`) | `GET /config` - returns global suggestions config boolean; `POST /threads/{id}/suggestions` - generate follow-up questions; rich list/block model content is normalized and inline reasoning (`<think>...</think>`, including unclosed/truncated blocks from reasoning models like MiniMax-M3) is stripped before JSON parsing |
| **Input Polish** (`/api/input-polish`) | `POST /` - rewrite a composer draft before it is sent. This is a short authenticated `runs:create` LLM request using `input_polish` config; it does not create a LangGraph run, persist a message, or modify thread state. Shares the non-graph one-shot LLM path (`deerflow.utils.oneshot_llm.run_oneshot_llm`) with the suggestions route so model build + Langfuse metadata + invoke stay in one place; validates the same stripped view of the draft it sends to the model, and preserves literal `<think>` substrings in the rewrite (`strip_think_blocks(truncate_unclosed=False)`) |
@ -440,13 +441,25 @@ metadata only.
- When a persistent `RunStore` is configured, `get()` and `list_by_thread()` hydrate historical runs from the store. In-memory records win for the same `run_id` so task, abort, and stream-control state stays attached to active local runs.
- `cancel()` returns a :class:`~deerflow.runtime.CancelOutcome` enum: `cancelled` (local cancel), `taken_over` (non-owning worker claimed the run because the owner's lease expired — marks it as `error`), `lease_valid_elsewhere` (owner's lease is still alive — caller should return 409 + `Retry-After`), `not_active_locally` (heartbeat disabled, preserving the old 409 path), `not_cancellable` (terminal state), or `unknown` (not found in memory or store). `create_or_reject(..., multitask_strategy="interrupt"|"rollback")` persists interrupted status through `RunStore.update_status()`, matching normal `set_status()` transitions.
- Store-only hydrated runs are readable history. In multi-worker mode with heartbeat enabled, cancel on a store-only run can take over (mark `error`) when the owner's lease has expired past the grace window; otherwise it fails with 409 + `Retry-After`. In single-worker mode (heartbeat off), store-only runs still return 409.
- Startup/orphan reconciliation must claim stale active rows with `RunStore.claim_for_takeover()`, not a plain `update_status()`. The final claim re-checks `status` and lease expiry atomically, so a heartbeat renewal between the candidate scan and the recovery write keeps the run active.
- `POST /wait` (both thread-scoped and `/api/runs/wait`) drains the stream bridge via `wait_for_run_completion()` instead of bare `await record.task`, so it honours the run's `on_disconnect` setting and cancels the background run on real client disconnect rather than returning a stale checkpoint (issue #3265).
- Redis `StreamBridge` keys use a rolling retained-buffer TTL (`stream_bridge.stream_ttl_seconds`, refreshed on `publish()` / `publish_end()`) as a leak safety net, not as a run timeout. Startup orphan recovery publishes `END_SENTINEL` and schedules stream cleanup for recovered runs; malformed `Last-Event-ID` reconnect values live-tail new Redis events rather than replaying the retained buffer. Do not broaden this into a shared-database multi-pod reaper without adding worker ownership/liveness first.
- Thread-scoped run creation accepts `checkpoint` / `checkpoint_id`; Gateway validates the checkpoint belongs to the request thread before writing `checkpoint_id` / `checkpoint_ns` into `config.configurable` for LangGraph branching.
- Thread-scoped Gateway runs evaluate an active `ThreadState.goal` after the visible turn completes. `runtime/goal.py` asks a non-thinking evaluator model to judge only visible conversation evidence and return a typed blocker; the evaluator model is created once per run and reused across hidden continuation checks. The evaluator runs after the graph root's tracing scope has already closed, so `create_goal_evaluator_model`/`evaluate_goal_completion` attach their own model-level tracing callbacks (`attach_tracing=True`) and inject Langfuse trace metadata (`thread_id`/`user_id`/`deerflow_trace_id`) directly onto the `ainvoke` call — the same standalone-caller pattern as `oneshot_llm.run_oneshot_llm` and `MemoryUpdater` (see Tracing System below). Satisfied goals are cleared; every non-satisfied evaluation — continuable or stand-down — is persisted with `last_evaluation` (the blocker, reason, and evidence summary; outcomes that stop the loop additionally record a `stand_down_reason` for observability), but only `goal_not_met_yet` evaluations are streamed as hidden `HumanMessage` continuations, and only when a durable assistant end-of-turn checkpoint exists, the run has not been aborted, the thread did not change during evaluation, and the no-progress breaker has not fired. The continuation cap is 8 — a hard maximum in the `0`–`8` range; callers requesting more are clamped (`set_goal`/TUI) or rejected with 422 (`PUT /goal`). The no-progress breaker keys on the latest visible assistant evidence (not the evaluator's free-text reason, which an LLM rewords every turn), so two consecutive continuations that add no new visible assistant output stop the loop after 2 attempts. Model-response cleanup helpers such as think-block stripping and code-fence stripping live in `deerflow.utils.llm_text` so `runtime/goal.py` and Gateway suggestion parsing share the same JSON-prep behavior.
- Run event stream changes must keep producer code, `deerflow/constants.py`, `runtime/events/catalog.py`, `contracts/run_event_stream_contract.json`, `backend/docs/RUN_EVENT_STREAM.md`, and `tests/test_run_event_stream_contract.py` in sync. The dependency-free constants module owns the persisted envelope limits (`event_type` 32 characters, `category` 16) and cross-layer workspace event identity; the catalog owns validated runtime definitions and categories. Dynamic middleware tags are limited to 21 characters after the `middleware:` prefix. The JSON contract owns payload schemas, backend-specific storage semantics, legacy aliases, and compatibility rules; conformance tests require both views and all producer groups to agree. `run.end.content` remains opaque and may retain nested Python values in memory while JSONL/database stores stringify non-JSON nested values, so consumers must not assume backend-identical nested output representations.
Proxied through nginx: `/api/langgraph/*` → Gateway LangGraph-compatible runtime, all other `/api/*` → Gateway REST APIs.
**Branch/regenerate checkpoint invariant**: `app/gateway/checkpoint_lineage.py`
walks `parent_config` rather than globally ordered checkpoint history so replay
anchors stay on the selected lineage after regenerations create sibling branches.
New conversation branches persist the pre-user replay anchor before their visible
head through the state mutation graph, which preserves materialized state in both
full and delta checkpoint modes. Only an explicitly absent legacy parent link may
use chronological compatibility lookup; cycles, dangling links, and depth-limit
exhaustion fail closed. Existing single-checkpoint branches are never repaired by
copying a raw checkpoint because delta state is not self-contained in one tuple.
### Sandbox System (`packages/harness/deerflow/sandbox/`)
**Interface**: Abstract `Sandbox` with `execute_command(command, env=None)`, `read_file`, `write_file`, `list_dir`. The optional `env` injects per-call environment variables (request-scoped secrets — see Request-Scoped Secrets below); `LocalSandbox` merges it via `subprocess.run(env=...)` and `AioSandbox` routes env-bearing commands through the `bash.exec(env=...)` API on a fresh session.
@ -509,7 +522,7 @@ Proxied through nginx: `/api/langgraph/*` → Gateway LangGraph-compatible runti
**Handled LLM failures**: `LLMErrorHandlingMiddleware` deliberately converts provider/model exceptions into an `AIMessage` so the graph can end cleanly, stamping `additional_kwargs.deerflow_error_fallback=true` plus error metadata. Clean graph termination does not imply subagent success: `SubagentExecutor` inspects the last assistant message at terminalization and maps a marked fallback to `SubagentStatus.FAILED`, which then emits `task_failed` and the existing structured `subagent_error`. Only the marker is authoritative — error-looking assistant prose without it remains a normal completed result, so neither the executor nor frontend parses display text as a status protocol.
**Guardrail caps & `stop_reason` (#3875 Phase 2)**: three independent axes can end a subagent run early, and all now surface *why* through one additive field rather than a new status enum. **Turn axis**: `recursion_limit` on the subagent `run_config` equals `max_turns`, so exhausting the turn budget raises `GraphRecursionError` from `agent.astream`; `executor.py::_aexecute` catches it specifically (before the generic `except Exception`). **Token axis**: `TokenBudgetMiddleware` is attached per-agent via `build_subagent_runtime_middlewares` from `subagents.token_budget` (default `max_tokens` **coupled to `summarization.enabled`** — 1,000,000 when subagent summarization is on, 2,000,000 when off, warn at 0.7, hard-stop at 1.0; a user-set budget always wins regardless of the switch — #3875 Phase 3; a backstop against a subagent that burns tokens on trivial work). It does *not* raise: at the hard-stop threshold it strips the in-flight turn's tool calls, forces `finish_reason="stop"`, and lets the run complete naturally with a final answer. **Loop axis**: `LoopDetectionMiddleware` (attached at the same point) catches repeated identical tool-call sets — or one tool *type* called many times with varying args — and its hard-stop likewise strips `tool_calls` and forces a final answer without raising, recording `loop_capped`. Each guard exposes its cap on a per-`run_id` `consume_stop_reason(run_id)` accessor; `_aexecute` collects **every** middleware with that method (duck-typed via `hasattr`, so the executor has no import coupling to the guard classes) and surfaces the first non-`None` reason — adding a future guard needs no executor change. **Surfacing**: whichever axis fired, `_aexecute` stamps a normal status plus an additive reason — `completed` + `stop_reason=token_capped|turn_capped|loop_capped` when a usable final answer (or partial recovered from the last streamed chunk via `_extract_final_result` → `utils/messages.py::message_content_to_text`, returning a `"No response Generated"` sentinel when no text survived) was produced; `failed` + `stop_reason=turn_capped` when nothing usable survived. `SubagentResult.stop_reason` flows through `task_tool.py::_task_result_command` → `format_subagent_result_message` (renders `Task Succeeded (capped: ...)` / `Task failed (capped: ...)`) and `make_subagent_additional_kwargs`, which stamps the additive `subagent_stop_reason` key alongside the normal `subagent_status`. **Why additive, not an enum**: a new status value would break v1 consumers; an optional field is ignored by older frontends and ledger readers, so the cross-language contract (`contracts/subagent_status_contract.json` v2 + `subagents/status_contract.py` + `frontend/.../subtask-result.ts`, pinned by `test_status_values_match_contract` / `test_stop_reason_values_match_contract`) stays backward-compatible. The durable delegation ledger captures `stop_reason` onto the entry and renders model-facing guidance ("hit a guardrail cap with a partial result; reuse it, retry tighter, or raise the per-agent budget (`max_turns` / `token_budget`)") so the lead reuses a capped completion knowingly instead of mistaking it for a clean one. (Phase 1 shipped this surfacing as a `MAX_TURNS_REACHED` status enum in #3949; Phase 2 replaced that enum with the additive `stop_reason` field per the agreed design — the `max_turns_reached` status value and `SubagentStatus.MAX_TURNS_REACHED` are gone.)
**Context compaction (#3875 Phase 3, #4039)**: subagents inherit `DeerFlowSummarizationMiddleware` via `build_subagent_runtime_middlewares`, gated on the **same** `summarization.enabled` switch the lead reads (one config covers both chains; trigger/keep/model/prompt come from the shared `summarization` config so they cannot drift). The subagent builder attaches `DurableContextMiddleware` immediately before summarization, using the same skills path/read-tool settings as the lead chain. Compaction stores the generated summary in `ThreadState.summary_text` rather than as a `messages` item; the durable-context wrapper therefore projects it into the next model request as guarded hidden human data. This is required when a message-count keep policy preserves only an assistant tool-call plus its tool results: without the injected summary the next request begins with assistant/tool history and strict OpenAI-compatible providers can reject it. Because `DurableContextMiddleware` inserts a second `SystemMessage(authority_contract)` after the subagent's leading system prompt, the builder also appends `SystemMessageCoalescingMiddleware` innermost (mirroring the lead chain, appended after the optional summarization middleware so it is unconditionally last) to merge every `SystemMessage` into one leading `system_message` — otherwise the durable fix would trade #4039's assistant-first HTTP 400 for a duplicate-system 400 on the same strict backends (#4040). The factory is called with `skip_memory_flush=True` on the subagent path: the lead's `memory_flush_hook` (attached when `memory.enabled`) flushes pre-compaction messages into durable memory keyed by `thread_id`, and subagents share the parent's `thread_id`, so without skipping the hook a subagent's internal turns would pollute the **parent** thread's durable memory. Placement differs from the lead chain (lead appends summarization *before* the guard trio; subagent appends it *after*) — benign because the middleware implements only `before_model` (compaction) with no `after_model`/`consume_stop_reason`, so it cannot disturb the Phase 2 guard-cap stop-reason channel. Compaction rewrites the messages channel via `RemoveMessage(id=REMOVE_ALL_MESSAGES)`, which shrinks `len(messages)` below the step-capture cursor mid-run; `capture_new_step_messages` (see Step capture below) resets the cursor to the new tail on contraction so steps appended after the compaction point are not silently dropped.
**Step capture & persistence (#3779)**: `executor.py` captures both assistant turns (`AIMessage`) **and** tool outputs (`ToolMessage`) via `subagents/step_events.py::capture_new_step_messages`, which walks the *newly-appended tail* of each `stream_mode="values"` chunk (not just `messages[-1]`) so a multi-tool-call turn — where LangGraph's `ToolNode` appends several `ToolMessage`s in one super-step — keeps every tool output instead of dropping all but the last. `runtime/runs/worker.py::_SubagentEventBuffer` additionally persists these `task_*` custom events to the `RunEventStore` as `subagent.start`/`subagent.step`/`subagent.end` (`category="subagent"`, `task_id` in `metadata`). It **batches** writes via `put_batch` (flushing on a terminal `subagent.end`, at `FLUSH_THRESHOLD` events, and in the worker's `finally`) rather than one `put()` per step, since `put()` is a documented low-frequency path (per-thread advisory lock per call) and a deep subagent (`max_turns=150`) emits hundreds of steps on the hot stream loop. `build_subagent_step` caps both the per-step `text` and each tool call's serialized `args` at `SUBAGENT_STEP_MAX_CHARS` (flagged `truncated` / `args_truncated`) so a large `write_file`/`bash` payload can't produce an unbounded row. The dedicated category keeps them out of `list_messages` (the thread feed) while `list_events` returns them for the frontend's fetch-on-expand backfill. `list_events` accepts `task_id` (filters on `metadata["task_id"]` — SQL-side in `DbRunEventStore` via `event_metadata["task_id"].as_string()`, in-memory in the JSONL/memory stores) plus an `after_seq` forward cursor, so the card pages through one subagent's steps without the run-wide `limit` truncating the tail (no schema migration: the filter rides the existing run-scoped index). `step_events.py` is a pure, unit-tested layer (`build_subagent_step` / `subagent_run_event`). **History contraction (#3875 Phase 3)**: `capture_new_step_messages` assumes append-only growth, but `DeerFlowSummarizationMiddleware` rewrites the messages channel via `RemoveMessage(id=REMOVE_ALL_MESSAGES)`, shrinking `len(messages)` below the cursor mid-run. On contraction (`total < processed_count`) the cursor resets to the new tail; `capture_step_message`'s id/content dedup prevents re-emitting pre-compaction steps, so steps appended after the compaction point are still captured instead of being dropped until `total` overtakes the stale cursor.
**Step capture & persistence (#3779)**: `executor.py` captures both assistant turns (`AIMessage`) **and** tool outputs (`ToolMessage`) via `subagents/step_events.py::capture_new_step_messages`, which walks the *newly-appended tail* of each `stream_mode="values"` chunk (not just `messages[-1]`) so a multi-tool-call turn — where LangGraph's `ToolNode` appends several `ToolMessage`s in one super-step — keeps every tool output instead of dropping all but the last. `runtime/runs/worker.py::_SubagentEventBuffer` additionally persists these `task_*` custom events to the `RunEventStore` as `subagent.start`/`subagent.step`/`subagent.end` (`category="subagent"`, `task_id` in `metadata`). It **batches** writes via `put_batch` (flushing on a terminal `subagent.end`, at `FLUSH_THRESHOLD` events, and in the worker's `finally`) rather than one `put()` per step, since `put()` is a documented low-frequency path (per-thread advisory lock per call) and a deep subagent (`max_turns=150`) emits hundreds of steps on the hot stream loop. `subagent_run_event` rejects malformed chunks that lack a non-empty `task_id`; running chunks additionally require a non-negative integer `message_index` and a message object, so persisted records always satisfy the required lifecycle envelope. `build_subagent_step` caps both the per-step `text` and each tool call's serialized `args` at `SUBAGENT_STEP_MAX_CHARS` (flagged `truncated` / `args_truncated`) so a large `write_file`/`bash` payload can't produce an unbounded row. The dedicated category keeps them out of `list_messages` (the thread feed) while `list_events` returns them for the frontend's fetch-on-expand backfill. `list_events` accepts `task_id` (filters on `metadata["task_id"]` — SQL-side in `DbRunEventStore` via `event_metadata["task_id"].as_string()`, in-memory in the JSONL/memory stores) plus an `after_seq` forward cursor, so the card pages through one subagent's steps without the run-wide `limit` truncating the tail (no schema migration: the filter rides the existing run-scoped index). `step_events.py` is a pure, unit-tested layer (`build_subagent_step` / `subagent_run_event`). **History contraction (#3875 Phase 3)**: `capture_new_step_messages` assumes append-only growth, but `DeerFlowSummarizationMiddleware` rewrites the messages channel via `RemoveMessage(id=REMOVE_ALL_MESSAGES)`, shrinking `len(messages)` below the cursor mid-run. On contraction (`total < processed_count`) the cursor resets to the new tail; `capture_step_message`'s id/content dedup prevents re-emitting pre-compaction steps, so steps appended after the compaction point are still captured instead of being dropped until `total` overtakes the stale cursor.
**Deferred MCP tools** (if `tool_search.enabled`): `SubagentExecutor._build_initial_state` assembles deferral after policy filtering via the shared `assemble_deferred_tools` (fail-closed), appends the `tool_search` tool, injects the `<available-deferred-tools>` section into the subagent's `SystemMessage`, and threads the setup to `_create_agent`, which attaches `McpRoutingMiddleware` (when PR1 routing metadata matches deferred tools) before `DeferredToolFilterMiddleware` through `build_subagent_runtime_middlewares(...)`. Subagents thus withhold full MCP schemas until promotion, same as the lead agent; each task run gets a fresh `ThreadState` so promotion is isolated per run
**Checkpointer isolation**: Subagent graphs are compiled with `checkpointer=False` to avoid inheriting the parent run's checkpointer, since subagents are one-shot and never resume.
**Checkpoint lineage / stream isolation**: `_aexecute` deliberately omits checkpoint-coordinate keys (`thread_id`, `checkpoint_ns`, `checkpoint_id`, `checkpoint_map`) from the child `RunnableConfig`. LangGraph must inherit those coordinates from the copied parent ContextVar so the delegated graph retains a non-root subgraph namespace; explicitly re-supplying even the same parent `thread_id` starts a new root lineage on LangGraph 1.2.6+ and can route child AI/tool frames into the parent `messages` stream. DeerFlow business components still receive the parent `thread_id` through `runtime.context`, which is the preferred lookup path for sandbox, middleware, and attribution code. Regression coverage in `tests/test_subagent_executor.py::TestSubagentCheckpointLineage` keeps the invocation-contract assertion active on every supported version and version-gates the production-shaped parent-stream test to LangGraph 1.2.6+, where the leak exists.
@ -809,6 +822,8 @@ Checkpointer storage runs in one of two channel modes, selected by `checkpoint_c
**Never bypass `CheckpointStateAccessor` (`runtime/checkpoint_state.py`) for thread-state access.** It is the single choke point binding graph + checkpointer + mode: it injects the mode marker into configs, runs the compatibility check before every `get`/`update`/`history`, and returns materialized state (delta checkpoints lack `channel_values.messages` — raw `get_tuple` reads see a sentinel). Gateway `services.py` builds and passes the accessor; thread-owned reads (state/history/regeneration) must use `build_thread_checkpoint_state_accessor` so the recorded assistant's middleware schema materializes every channel. `history(limit)` semantics: `0` means zero items (explicit empty), `None` means unlimited — do not pass `limit=0` through to `graph.get_state_history`. Assistant metadata lookup is fail-closed for mutation accessors so a store outage cannot silently select the default schema and discard extension channels. In `full` mode the read path degrades to a raw checkpointer read (`_RawCheckpointReadAccessor`) when the agent factory cannot build the graph (bad model config, MCP outage) — full checkpoints carry complete `channel_values`, so reads don't need the graph; degraded snapshots take `created_at` from the standard checkpoint `ts` field, falling back to metadata only for compatibility. The delta gate still applies on the degraded path; `next`/`tasks` degrade to empty and thread status falls back to the stored status because task presence is not derivable, while delta mode has no fallback (materialization needs the channel table).
**Replay checkpoint lookup prefers lineage and degrades only for an explicitly missing legacy parent link.** Branch and regenerate paths first walk `parent_config`, which prevents a global chronological scan from selecting a sibling created by regeneration. `CheckpointParentMissingError` alone enables the bounded newest-first history fallback in `app/gateway/checkpoint_lineage.py`; cycles, dangling/non-addressable parents, target mismatches, and depth exhaustion raise `CheckpointLineageIntegrityError` and fail closed instead of selecting a sibling. The compatibility scans request 400 raw checkpoints so up to 200 duration-only entries do not consume the effective branch-history budget; the fallback scans oldest-to-newest internally, skips duration-only checkpoints, and accepts only checkpoints with an addressable id as the replay base. A source history with no discoverable pre-user checkpoint preserves the historical single-checkpoint branch behavior instead of rejecting the branch; regeneration remains unavailable for that inherited response. Existing single-checkpoint branches are not mutated by regenerate preparation, and no raw checkpoint tuple is copied across threads because delta state depends on ancestry and pending writes. Regenerate source-run lookup uses the current thread's exact event, then the server-stamped `run_id` on the copied human message, then verified RunManager content matching; it does not read parent-thread events. Storage or checkpoint-mode failures are not treated as a missing base and still fail closed.
**Wholesale message replacement uses a state-only mutation graph + `Overwrite`.** `update_state` values pass through channel reducers (`add_messages` merge in full, append in delta), so replacing `messages` wholesale (run rollback, context compaction) requires `{"messages": Overwrite([...])}`. The write goes through `build_state_mutation_graph(as_node, mode, state_schema)` — when the write carries materialized state, `state_schema` MUST be the thread's effective schema (`graph_state_schema(assistant_graph)`), because the base-ThreadState fallback silently discards written channels contributed by custom `AgentMiddleware.state_schema`. Channels absent from the write are unaffected: forked checkpoints clone the parent's channel blobs, so middleware channels survive rollback/compaction regardless of schema (locked by `test_rollback_preserves_middleware_contributed_channels` and `test_compact_thread_context_preserves_middleware_contributed_channels`) — a compiled graph with one no-op node (entry = finish) whose checkpoint machinery (channels/versions/metadata) is identical to the agent graph's but schedules no pending tasks, so the restored/compacted head stays idle instead of re-triggering the agent. Never hand-write checkpoints via `checkpointer.aput` for this; raw writers elsewhere must preserve checkpoint parentage — severed ancestry breaks delta replay (see `runtime/runs/worker.py` writer parenting and `checkpoint_patches.py`).
**Run rollback flow** (`runtime/runs/worker.py`): `_capture_rollback_point` materializes pre-run state (messages via accessor, raw `pending_writes` via `aget_tuple`) into an immutable `RollbackPoint` before the run starts — capture failure disables rollback (fail-closed), never restores partial state. Cancel-with-rollback then forks from the pre-run checkpoint via the mutation graph and replays the captured pending writes.
@ -821,6 +836,44 @@ Checkpointer storage runs in one of two channel modes, selected by `checkpoint_c
- `runtime/context_compaction.py` — compaction via accessor + mutation graph (reference consumer)
- Tests: `tests/test_checkpoint_mode.py` (freeze/detect/gate), `tests/test_checkpoint_state.py` (accessor/mutation graph), `tests/test_delta_channel_checkpointers.py` (saver parity), `tests/test_threads_checkpoint_mode.py`, `tests/test_gateway_checkpoint_mode.py` (dual-mode e2e parity), `tests/test_context_compaction.py` (mutation-graph write, no scheduling), `tests/test_run_worker_rollback.py`
**Checkpoint channel benchmark**: `scripts/benchmark/bench_checkpoint_channels.py`
runs paired `full`/`delta` message-only StateGraphs in a fresh child process per
case, using sync `InMemorySaver` or `SqliteSaver` so reducer, serialization, and
saver costs stay separate from Gateway/async scheduling. It reports deterministic
correctness digests, write windows/percentiles, warm and graph-rebuilt cold reads,
logical checkpoint/write bytes, SQLite DB/WAL/SHM footprint, reducer replay time,
and peak RSS as versioned JSONL. The controller alternates mode order and rejects
performance data when paired modes materialize different state. Its default 1 GiB
estimated cumulative full-payload cap skips both modes of an oversized pair when
`full` is selected; intentional `--modes delta` diagnostics bypass this
full-payload cap, so size those runs explicitly. Use `--allow-large-cases` only
on a provisioned machine. Duplicate CSV matrix values are ignored with a warning;
use `--repetitions` for repeated samples. Summarize paired successful repetitions
with `scripts/benchmark/summarize_checkpoint_channels.py` (all ratios are
`delta/full`). `--profile-dir /tmp/checkpoint-profiles` writes one cProfile
artifact per case for attribution. Profiled rows carry `profiled: true`, and the
summarizer automatically excludes them from baseline summaries with a warning.
Storage-size collection relies on saver-specific diagnostic layouts; if those
layouts change, the timing/correctness row remains successful while storage
fields become `null` and `storage_stats_error` records the diagnostic failure.
Example:
```bash
cd backend
PYTHONPATH=. uv run python scripts/benchmark/bench_checkpoint_channels.py \
--backends sqlite --updates 100,500,999,1000,1001 --payload-bytes 128 \
--repetitions 7 --output /tmp/checkpoint-bench.jsonl
PYTHONPATH=. uv run python scripts/benchmark/summarize_checkpoint_channels.py \
/tmp/checkpoint-bench.jsonl
```
The sync storage benchmark is not an end-to-end Gateway benchmark. Complete
`ThreadState`/`DeltaThreadState`, async saver scheduling, history, mutation,
rollback, migration, and branch-heavy cases belong to the production-shaped
follow-up layer. Harness tests live in `tests/test_bench_checkpoint_channels.py`
and `tests/test_summarize_checkpoint_channels.py`; timing thresholds are not CI
gates.
### Terminal Workbench / TUI (`packages/harness/deerflow/tui/`)
A terminal-native UI over the embedded harness, exposed as the `deerflow` console script (`[project.scripts]` in `packages/harness/pyproject.toml`). It is a UI shell over `DeerFlowClient` and does **not** fork agent behavior. `textual` is an optional dependency (`deerflow-harness[tui]`; also in the backend dev group); the console script degrades to headless help when it is absent. Full guide: [docs/TUI.md](docs/TUI.md).
@ -924,8 +977,9 @@ Gateway API endpoints and `DeerFlowClient` methods can modify MCP servers and sk
- `stream(message, thread_id)` — subscribes to LangGraph `stream_mode=["values", "messages", "custom"]` and yields `StreamEvent`:
- `"values"` — full state snapshot (title, messages, artifacts); AI text already delivered via `messages` mode is **not** re-synthesized here to avoid duplicate deliveries
- `"messages-tuple"` — per-chunk update: for AI text this is a **delta** (concat per `id` to rebuild the full message); tool calls and tool results are emitted once each
- `"custom"` — forwarded from `StreamWriter`
- `"custom"` — forwarded from `StreamWriter`; DeerFlow-built-in custom events are dual-emitted through `deerflow.utils.custom_events`, so `astream_events(version="v2")` consumers also receive one `on_custom_event` with `name=payload["type"]` and the unchanged payload as `data`
- `"end"` — stream finished (carries cumulative `usage` counted once per message id)
- **Custom-event invariant** — production DeerFlow emitters must use `emit_custom_event` / `aemit_custom_event`, not call `StreamWriter` alone. Every built-in payload must carry a non-empty string `type`; typeless payloads remain writer-only and are intentionally absent from `astream_events`. The writer runs first and remains authoritative for Gateway, Web UI, and embedded-client compatibility; callback dispatch is best-effort and must not break that path. Async graph hooks must await the async helper rather than invoking synchronous dispatch on a running event loop.
- Agent created lazily via `create_agent()` + `build_middlewares()`, same as `make_lead_agent`
- Supports `checkpointer` parameter for state persistence across turns
- `reset_agent()` forces agent recreation (e.g. after memory or skill changes)

6
backend/Makefile
View File

@ -14,11 +14,11 @@ test-blocking-io:
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run pytest tests/blocking_io -q --tb=short
lint:
uvx ruff check .
uvx ruff format --check .
uv run ruff check .
uv run ruff format --check .
format:
uvx ruff check . --fix && uvx ruff format .
uv run ruff check . --fix && uv run ruff format .
detect-blocking-io:
@PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run python ../scripts/detect_blocking_io_static.py --output ../.deer-flow/blocking-io-findings.json

164
backend/app/gateway/checkpoint_lineage.py Normal file
View File

@ -0,0 +1,164 @@
"""Shared helpers for resolving replay checkpoints on one checkpoint lineage."""
from __future__ import annotations
from collections.abc import Sequence
from typing import Any
class CheckpointLineageError(RuntimeError):
"""Raised when a requested checkpoint ancestor cannot be resolved safely."""
class CheckpointParentMissingError(CheckpointLineageError):
"""Raised when a legacy checkpoint does not record its parent link."""
class CheckpointLineageIntegrityError(CheckpointLineageError):
"""Raised when recorded checkpoint lineage is present but unsafe to use."""
def checkpoint_messages(checkpoint_tuple: Any) -> list[Any]:
values = getattr(checkpoint_tuple, "values", None)
if isinstance(values, dict):
messages = values.get("messages", [])
return list(messages) if isinstance(messages, list) else []
checkpoint = getattr(checkpoint_tuple, "checkpoint", None) or {}
channel_values = checkpoint.get("channel_values", {}) if isinstance(checkpoint, dict) else {}
messages = channel_values.get("messages", []) if isinstance(channel_values, dict) else []
return list(messages) if isinstance(messages, list) else []
def checkpoint_configurable(checkpoint_tuple: Any) -> dict[str, Any]:
config = getattr(checkpoint_tuple, "config", None) or {}
configurable = config.get("configurable", {}) if isinstance(config, dict) else {}
return dict(configurable) if isinstance(configurable, dict) else {}
def checkpoint_metadata(checkpoint_tuple: Any) -> dict[str, Any]:
metadata = getattr(checkpoint_tuple, "metadata", None) or {}
return dict(metadata) if isinstance(metadata, dict) else {}
def is_duration_only_checkpoint(checkpoint_tuple: Any) -> bool:
writes = checkpoint_metadata(checkpoint_tuple).get("writes")
return isinstance(writes, dict) and "runtime_run_duration" in writes
def _message_id(message: Any) -> str | None:
value = getattr(message, "id", None)
if value is None and isinstance(message, dict):
value = message.get("id")
return str(value) if value else None
def _config_identity(config: dict[str, Any]) -> tuple[str, str, str] | None:
configurable = config.get("configurable", {})
thread_id = configurable.get("thread_id")
checkpoint_ns = configurable.get("checkpoint_ns", "")
checkpoint_id = configurable.get("checkpoint_id")
if not isinstance(thread_id, str) or not thread_id or not isinstance(checkpoint_id, str) or not checkpoint_id:
return None
return thread_id, str(checkpoint_ns or ""), checkpoint_id
def _checkpoint_identity(checkpoint_tuple: Any) -> tuple[str, str, str] | None:
return _config_identity(getattr(checkpoint_tuple, "config", {}) or {})
def _checkpoint_exists(checkpoint_tuple: Any) -> bool:
"""Distinguish a persisted empty checkpoint from an accessor miss.
LangGraph represents a missing explicit ``checkpoint_id`` as an empty
snapshot that echoes the requested config. Persisted snapshots always
carry metadata, a creation timestamp, or a raw checkpoint payload.
"""
explicit = getattr(checkpoint_tuple, "checkpoint_exists", None)
if isinstance(explicit, bool):
return explicit
if getattr(checkpoint_tuple, "metadata", None) is not None:
return True
if getattr(checkpoint_tuple, "created_at", None) is not None:
return True
return isinstance(getattr(checkpoint_tuple, "checkpoint", None), dict)
async def find_checkpoint_before_message(
accessor: Any,
head_checkpoint: Any,
message_id: str,
*,
max_depth: int,
) -> Any:
"""Walk one parent lineage and return the first checkpoint before ``message_id``.
Following ``parent_config`` is important after a regenerate: a thread can contain
sibling checkpoint branches, and a global time-ordered scan can otherwise select
a checkpoint from the wrong branch. Duration-only metadata checkpoints do not
represent an addressable conversation state and are skipped.
"""
if message_id not in {_message_id(message) for message in checkpoint_messages(head_checkpoint)}:
raise CheckpointLineageIntegrityError("Target message is not present in the checkpoint head")
current = head_checkpoint
visited: set[tuple[str, str, str]] = set()
current_identity = _checkpoint_identity(current)
if current_identity is not None:
visited.add(current_identity)
# Each step performs one ancestor read, but normal branch/regenerate
# histories cross the target boundary within 1–3 reads. Keep max_depth as
# a conservative safety cap for valid histories with many intermediate or
# duration-only checkpoints.
for _ in range(max_depth):
parent_config = getattr(current, "parent_config", None)
if not isinstance(parent_config, dict):
raise CheckpointParentMissingError("Checkpoint lineage ended before the target message")
parent = await accessor.aget(parent_config)
parent_identity = _checkpoint_identity(parent)
requested_parent_identity = _config_identity(parent_config)
if parent_identity is None or not _checkpoint_exists(parent) or (requested_parent_identity is not None and parent_identity != requested_parent_identity):
raise CheckpointLineageIntegrityError("Checkpoint parent link is not addressable")
if parent_identity is not None:
if parent_identity in visited:
raise CheckpointLineageIntegrityError("Checkpoint lineage contains a cycle")
visited.add(parent_identity)
if is_duration_only_checkpoint(parent):
current = parent
continue
parent_message_ids = {_message_id(message) for message in checkpoint_messages(parent)}
if message_id not in parent_message_ids:
return parent
current = parent
raise CheckpointLineageIntegrityError(f"Checkpoint lineage exceeded the scan limit ({max_depth})")
def find_checkpoint_before_message_chronologically(
checkpoints: Sequence[Any],
message_id: str,
) -> tuple[Any | None, bool]:
"""Return ``(replay_base, target_found)`` from newest-first history.
This is a compatibility fallback for imported or legacy checkpoints that do
not carry ``parent_config`` links. Callers must prefer the lineage walk when
links are available because a chronological scan cannot distinguish sibling
checkpoint branches. Duration-only checkpoints are ignored, and only
checkpoints with an addressable id can become the replay base.
"""
previous_checkpoint = None
for checkpoint_tuple in reversed(checkpoints):
if is_duration_only_checkpoint(checkpoint_tuple):
continue
message_ids = {_message_id(message) for message in checkpoint_messages(checkpoint_tuple)}
if message_id in message_ids:
return previous_checkpoint, True
if checkpoint_configurable(checkpoint_tuple).get("checkpoint_id"):
previous_checkpoint = checkpoint_tuple
return None, False

54
backend/app/gateway/routers/github_webhooks.py
View File

@ -187,19 +187,27 @@ async def receive_github_webhook(
automatically retry a failed delivery of any kind — 5xx, timeout, or
connection error are all simply recorded as failed; see
https://docs.github.com/en/webhooks/using-webhooks/handling-failed-webhook-deliveries.
A 200 response marks the delivery permanently successful, so
swallowing a transient registry filesystem error or bus publish
failure into a 200 would silently drop a real webhook forever with no
way to recover it. Returning 503 instead keeps the delivery correctly
recorded as failed in GitHub's Recent Deliveries / Deliveries API, so
an operator's manual "Redeliver" click, a REST API call, or a
self-hosted recovery script (the pattern GitHub's own docs recommend)
can still recover it — by the time that redelivery lands the
underlying outage is usually gone. The `is_route_enabled()` startup
check still handles *configuration* errors fail-closed (route absent
→ 404); 503 is reserved for runtime failures worth making recoverable
this way. Permanent / non-retryable conditions (unknown event, missing
channel service) keep returning 200.
Swallowing a transient registry filesystem error or bus publish
failure into a 200 would mark the delivery successful, so it never
surfaces as failed in GitHub's Recent Deliveries / Deliveries API and
a scheduled recovery script filtering on non-OK status (the pattern
GitHub's own docs recommend) will never flag it for redelivery. That
is a discoverability problem, not literal unrecoverability — GitHub's
manual "Redeliver" button and its REST/App redelivery endpoints place
no failed-status precondition on the delivery id (see
https://docs.github.com/en/webhooks/testing-and-troubleshooting-webhooks/redelivering-webhooks),
so an operator who independently identifies this exact delivery can
still redeliver it by hand within GitHub's ~3-day redelivery window —
they just get no automated signal telling them to. Returning 503
keeps the delivery correctly recorded as failed instead, so that same
manual click, REST call, or recovery script actually finds it rather
than merely being capable of redelivering it if asked, well inside
the window the underlying outage usually clears in. The
`is_route_enabled()` startup check still handles *configuration*
errors fail-closed (route absent → 404); 503 is reserved for runtime
failures worth making discoverable and recoverable this way.
Permanent / non-retryable conditions (unknown event, missing channel
service) keep returning 200.
The route is fail-closed: :func:`is_route_enabled` should have already
prevented this handler from being mounted when no secret is configured.
@ -326,14 +334,20 @@ async def receive_github_webhook(
# ``fanout_event`` calls the registry (filesystem) and the
# message bus; both can fail transiently (disk hiccup, bus
# queue full, asyncio cancellation). Swallowing those into a
# 200 would permanently drop a real webhook, since GitHub
# treats 200 as final success and does not automatically
# retry any failure — including 5xx (see
# 200 would hide a real failure: GitHub treats 200 as final
# success and never automatically retries any failure,
# including 5xx (see
# https://docs.github.com/en/webhooks/using-webhooks/handling-failed-webhook-deliveries).
# 503 keeps the delivery recorded as failed so a manual
# "Redeliver", the REST API, or a recovery script can recover
# it. The startup-time ``is_route_enabled`` check still
# covers fail-closed *configuration* errors.
# A 200 delivery can still be redelivered by hand — GitHub's
# manual "Redeliver" button and REST redelivery endpoint have
# no failed-status precondition (see
# https://docs.github.com/en/webhooks/testing-and-troubleshooting-webhooks/redelivering-webhooks)
# — but nothing flags it for an operator or recovery script
# to do so. 503 keeps the delivery correctly recorded as
# failed so that same manual click, REST call, or recovery
# script actually finds and recovers it. The startup-time
# ``is_route_enabled`` check still covers fail-closed
# *configuration* errors.
try:
dispatch_result = await fanout_event(
service.bus,

104
backend/app/gateway/routers/thread_runs.py
View File

@ -23,9 +23,19 @@ from langchain_core.messages import BaseMessage
from pydantic import BaseModel, Field
from app.gateway.authz import require_permission
from app.gateway.checkpoint_lineage import (
CheckpointLineageError,
CheckpointParentMissingError,
checkpoint_configurable,
checkpoint_messages,
find_checkpoint_before_message,
find_checkpoint_before_message_chronologically,
is_duration_only_checkpoint,
)
from app.gateway.deps import get_current_user, get_feedback_service, get_run_event_store, get_run_manager, get_run_store, get_stream_bridge
from app.gateway.pagination import trim_run_message_page
from app.gateway.services import build_checkpoint_state_accessor, build_thread_checkpoint_state_accessor, sse_consumer, start_run, wait_for_run_completion
from app.gateway.utils import sanitize_log_param
from deerflow.runtime import CancelOutcome, RunRecord, RunStatus, serialize_channel_values_for_api
from deerflow.utils.messages import ORIGINAL_USER_CONTENT_KEY, get_original_user_content_text, message_to_text
from deerflow.workspace_changes import get_workspace_changes_response
@ -37,12 +47,12 @@ REGENERATE_HISTORY_SCAN_LIMIT = 200
# (one per successful run in steady state) consume roughly half of history.
REGENERATE_HISTORY_RAW_SCAN_LIMIT = REGENERATE_HISTORY_SCAN_LIMIT * 2
THREAD_MESSAGE_PAGE_SCAN_BATCH = 201
_MISSING_REGENERATE_BASE_DETAIL = "Could not find an addressable checkpoint before the target user message"
_UNSAFE_REGENERATE_LINEAGE_DETAIL = "Could not safely resolve the checkpoint before the target user message"
def _is_duration_only_checkpoint(checkpoint_tuple: Any) -> bool:
metadata = getattr(checkpoint_tuple, "metadata", None)
writes = metadata.get("writes") if isinstance(metadata, dict) else None
return isinstance(writes, dict) and "runtime_run_duration" in writes
return is_duration_only_checkpoint(checkpoint_tuple)
def compute_run_durations(runs) -> dict[str, int]:
@ -294,15 +304,11 @@ def _is_middleware_message_row(row: dict[str, Any]) -> bool:
def _checkpoint_messages(snapshot: Any) -> list[Any]:
values = getattr(snapshot, "values", None) or {}
messages = values.get("messages", []) if isinstance(values, dict) else []
return messages if isinstance(messages, list) else []
return checkpoint_messages(snapshot)
def _checkpoint_configurable(checkpoint_tuple: Any) -> dict[str, Any]:
config = getattr(checkpoint_tuple, "config", None) or {}
configurable = config.get("configurable", {}) if isinstance(config, dict) else {}
return dict(configurable) if isinstance(configurable, dict) else {}
return checkpoint_configurable(checkpoint_tuple)
def _checkpoint_response(checkpoint_tuple: Any) -> dict[str, Any]:
@ -356,7 +362,13 @@ def _run_last_ai_matches_message(record: RunRecord, message: Any) -> bool:
return last_ai_message == target_text[: len(last_ai_message)]
async def _find_target_run_id(thread_id: str, message_id: str, target_message: Any, request: Request) -> str:
async def _find_target_run_id(
thread_id: str,
message_id: str,
target_message: Any,
source_human: Any,
request: Request,
) -> str:
event_store = get_run_event_store(request)
rows = await event_store.list_messages(thread_id, limit=REGENERATE_HISTORY_SCAN_LIMIT)
for row in reversed(rows):
@ -366,6 +378,11 @@ async def _find_target_run_id(thread_id: str, message_id: str, target_message: A
run_id = row.get("run_id")
if isinstance(run_id, str) and run_id:
return run_id
source_run_id = _message_additional_kwargs(source_human).get("run_id")
if isinstance(source_run_id, str) and source_run_id:
return source_run_id
run_mgr = get_run_manager(request)
user_id = await get_current_user(request)
records = await run_mgr.list_by_thread(thread_id, user_id=user_id, limit=10)
@ -385,8 +402,37 @@ async def _find_target_run_id(thread_id: str, message_id: str, target_message: A
raise HTTPException(status_code=409, detail="Could not find source run for assistant message")
async def _find_base_checkpoint_before_human(thread_id: str, human_message_id: str, request: Request) -> Any:
async def _find_base_checkpoint_before_human(
thread_id: str,
human_message_id: str,
request: Request,
*,
head_checkpoint: Any | None = None,
) -> Any:
accessor, base_config = await build_thread_checkpoint_state_accessor(request, thread_id=thread_id)
if head_checkpoint is not None:
try:
return await find_checkpoint_before_message(
accessor,
head_checkpoint,
human_message_id,
max_depth=REGENERATE_HISTORY_RAW_SCAN_LIMIT,
)
except CheckpointParentMissingError:
# Old checkpoints and imported histories may not have parent links.
# Preserve the bounded chronological fallback for those records.
logger.debug(
"Could not resolve parent lineage for regenerate thread %s; falling back to history scan",
sanitize_log_param(thread_id),
exc_info=True,
)
except CheckpointLineageError as exc:
logger.warning(
"Rejected unsafe checkpoint lineage for regenerate thread %s",
sanitize_log_param(thread_id),
exc_info=True,
)
raise HTTPException(status_code=409, detail=_UNSAFE_REGENERATE_LINEAGE_DETAIL) from exc
try:
raw_checkpoints = await accessor.ahistory(base_config, limit=REGENERATE_HISTORY_RAW_SCAN_LIMIT)
checkpoints = [item for item in raw_checkpoints if not _is_duration_only_checkpoint(item)]
@ -394,19 +440,14 @@ async def _find_base_checkpoint_before_human(thread_id: str, human_message_id: s
logger.exception("Failed to list checkpoints for regenerate thread %s", thread_id)
raise HTTPException(status_code=500, detail="Failed to inspect checkpoint history") from exc
previous_checkpoint = None
for checkpoint_tuple in reversed(checkpoints):
messages = _checkpoint_messages(checkpoint_tuple)
message_ids = {_message_id(message) for message in messages}
if human_message_id in message_ids:
if previous_checkpoint is None:
raise HTTPException(
status_code=409,
detail="Could not find an addressable checkpoint before the target user message",
)
return previous_checkpoint
if _checkpoint_configurable(checkpoint_tuple).get("checkpoint_id"):
previous_checkpoint = checkpoint_tuple
previous_checkpoint, target_found = find_checkpoint_before_message_chronologically(raw_checkpoints, human_message_id)
if target_found:
if previous_checkpoint is None:
raise HTTPException(
status_code=409,
detail=_MISSING_REGENERATE_BASE_DETAIL,
)
return previous_checkpoint
if len(checkpoints) >= REGENERATE_HISTORY_SCAN_LIMIT:
logger.warning(
@ -451,8 +492,19 @@ async def _prepare_regenerate_payload(thread_id: str, message_id: str, request:
if not previous_human_id:
raise HTTPException(status_code=409, detail="The source user message is missing an id")
base_checkpoint_tuple = await _find_base_checkpoint_before_human(thread_id, previous_human_id, request)
target_run_id = await _find_target_run_id(thread_id, message_id, target_message, request)
base_checkpoint_tuple = await _find_base_checkpoint_before_human(
thread_id,
previous_human_id,
request,
head_checkpoint=latest_checkpoint,
)
target_run_id = await _find_target_run_id(
thread_id,
message_id,
target_message,
previous_human,
request,
)
checkpoint = _checkpoint_response(base_checkpoint_tuple)
metadata = {
"regenerate_from_message_id": message_id,

163
backend/app/gateway/routers/threads.py
View File

@ -25,7 +25,14 @@ from pydantic import BaseModel, Field, field_validator
from sqlalchemy.exc import IntegrityError
from app.gateway.authz import require_permission
from app.gateway.deps import get_checkpointer, get_run_manager
from app.gateway.checkpoint_lineage import (
CheckpointLineageError,
CheckpointParentMissingError,
find_checkpoint_before_message,
find_checkpoint_before_message_chronologically,
is_duration_only_checkpoint,
)
from app.gateway.deps import get_checkpointer, get_run_event_store, get_run_manager
from app.gateway.internal_auth import get_trusted_internal_owner_user_id
from app.gateway.services import (
build_checkpoint_state_accessor,
@ -54,6 +61,7 @@ from deerflow.runtime.goal import (
read_thread_goal,
write_thread_goal,
)
from deerflow.runtime.journal import build_branch_history_seed_events
from deerflow.runtime.runs.worker import valid_duration_entry
from deerflow.runtime.user_context import get_effective_user_id
from deerflow.utils.file_io import run_file_io
@ -89,6 +97,7 @@ _SERVER_RESERVED_METADATA_KEYS: frozenset[str] = frozenset({"owner_id", "user_id
_SIDECAR_METADATA_KEY = "deerflow_sidecar"
_BRANCH_METADATA_KEY = "deerflow_branch"
_BRANCH_HISTORY_SCAN_LIMIT = 200
_BRANCH_HISTORY_RAW_SCAN_LIMIT = _BRANCH_HISTORY_SCAN_LIMIT * 2
def _strip_reserved_metadata(metadata: dict[str, Any] | None) -> dict[str, Any]:
@ -158,13 +167,26 @@ def _matches_branch_target(messages: list[Any], target_message_ids: set[str]) ->
return not any(_is_branch_visible_message(message) for message in messages[target_end_index + 1 :])
def _branch_target_human_message(messages: list[Any], target_message_ids: set[str]) -> Any | None:
index_by_id = {_message_id(message): index for index, message in enumerate(messages) if _message_id(message)}
if not target_message_ids.issubset(index_by_id.keys()):
return None
target_start_index = min(index_by_id[message_id] for message_id in target_message_ids)
return next(
(message for message in reversed(messages[:target_start_index]) if _message_type(message) == "human" and _is_branch_visible_message(message)),
None,
)
async def _find_branch_checkpoint(
accessor: Any,
config: dict[str, Any],
target_message_ids: set[str],
) -> Any:
try:
for snapshot in await accessor.ahistory(config, limit=_BRANCH_HISTORY_SCAN_LIMIT):
for snapshot in await accessor.ahistory(config, limit=_BRANCH_HISTORY_RAW_SCAN_LIMIT):
if is_duration_only_checkpoint(snapshot):
continue
if _matches_branch_target(_checkpoint_messages(snapshot), target_message_ids):
return snapshot
except _CHECKPOINT_MODE_ERRORS as exc:
@ -183,7 +205,9 @@ async def _branch_targets_latest_turn(
) -> bool:
"""Return whether the target turn is the final visible turn."""
try:
for snapshot in await accessor.ahistory(config, limit=_BRANCH_HISTORY_SCAN_LIMIT):
for snapshot in await accessor.ahistory(config, limit=_BRANCH_HISTORY_RAW_SCAN_LIMIT):
if is_duration_only_checkpoint(snapshot):
continue
messages = _checkpoint_messages(snapshot)
if not messages:
continue
@ -198,6 +222,57 @@ async def _branch_targets_latest_turn(
return False
async def _find_branch_replay_base(
accessor: Any,
config: dict[str, Any],
snapshot: Any,
target_human_id: str,
) -> Any | None:
"""Resolve a replay base while preserving unlinked legacy histories."""
try:
return await find_checkpoint_before_message(
accessor,
snapshot,
target_human_id,
max_depth=_BRANCH_HISTORY_RAW_SCAN_LIMIT,
)
except CheckpointParentMissingError:
thread_id = config.get("configurable", {}).get("thread_id", "")
logger.debug(
"Could not resolve parent lineage for branch thread %s; falling back to history scan",
sanitize_log_param(thread_id),
exc_info=True,
)
except CheckpointLineageError as exc:
thread_id = config.get("configurable", {}).get("thread_id", "")
logger.warning(
"Rejected unsafe checkpoint lineage for branch thread %s",
sanitize_log_param(thread_id),
exc_info=True,
)
raise HTTPException(status_code=409, detail="This turn can no longer be branched from.") from exc
try:
history = await accessor.ahistory(config, limit=_BRANCH_HISTORY_RAW_SCAN_LIMIT)
except _CHECKPOINT_MODE_ERRORS as exc:
thread_id = config.get("configurable", {}).get("thread_id", "")
raise _checkpoint_mode_http_error(exc, thread_id) from exc
except Exception as exc:
thread_id = config.get("configurable", {}).get("thread_id", "")
logger.exception("Failed to scan replay checkpoint history for thread %s", sanitize_log_param(thread_id))
raise HTTPException(status_code=500, detail="Failed to inspect checkpoint history") from exc
replay_base, target_found = find_checkpoint_before_message_chronologically(history, target_human_id)
if not target_found:
logger.warning(
"Could not locate branch user message %s in chronological history for thread %s",
sanitize_log_param(target_human_id),
sanitize_log_param(config.get("configurable", {}).get("thread_id", "")),
)
return replay_base
def _ignore_branch_user_data(directory: str, names: list[str]) -> set[str]:
ignored: set[str] = set()
base = Path(directory)
@ -415,6 +490,9 @@ class ThreadBranchResponse(BaseModel):
parent_checkpoint_id: str
branched_from_message_id: str
workspace_clone_mode: str
# "seeded" | "skipped_empty" | "failed" — whether the parent history was
# copied into the branch's run-event feed (see branch_thread).
history_seed_mode: str
# ---------------------------------------------------------------------------
@ -696,6 +774,16 @@ async def branch_thread(thread_id: str, body: ThreadBranchRequest, request: Requ
parent_checkpoint_id = _checkpoint_id(snapshot)
if not parent_checkpoint_id:
raise HTTPException(status_code=409, detail="This turn can no longer be branched from.")
target_human = _branch_target_human_message(_checkpoint_messages(snapshot), target_message_ids)
target_human_id = _message_id(target_human)
if not target_human_id:
raise HTTPException(status_code=409, detail="This turn can no longer be branched from.")
replay_base_tuple = await _find_branch_replay_base(
source_accessor,
source_config,
snapshot,
target_human_id,
)
# Workspace files are not checkpointed, so they only reflect the *current* thread
# state. Cloning them onto a branch from an older turn would leak files created
@ -736,20 +824,39 @@ async def branch_thread(thread_id: str, body: ThreadBranchRequest, request: Requ
branch_reducer_fields = graph_reducer_channels(getattr(branch_accessor, "graph", None))
if branch_reducer_fields is None:
branch_reducer_fields = THREAD_STATE_REDUCER_FIELDS
branch_values = {}
for key, value in dict(snapshot.values).items():
if key in branch_reducer_fields:
branch_values[key] = Overwrite(list(value) if key == "messages" and isinstance(value, list) else value)
else:
branch_values[key] = value
new_config.setdefault("metadata", {}).update(
{
**branch_metadata,
"source": "branch",
}
)
def branch_values(source_snapshot: Any) -> dict[str, Any]:
values: dict[str, Any] = {}
for key, value in dict(source_snapshot.values).items():
if key in branch_reducer_fields:
values[key] = Overwrite(list(value) if key == "messages" and isinstance(value, list) else value)
else:
values[key] = value
return values
# Stamp both synthetic checkpoints with the branch-creation time because
# serializers fall back to metadata when snapshot.created_at is absent.
checkpoint_metadata_updates = {
**branch_metadata,
"source": "branch",
"updated_at": now,
"created_at": now,
}
new_config.setdefault("metadata", {}).update(checkpoint_metadata_updates)
try:
await branch_accessor.aupdate(new_config, branch_values, as_node="branch")
head_config = new_config
if replay_base_tuple is not None:
head_config = await branch_accessor.aupdate(
new_config,
branch_values(replay_base_tuple),
as_node="branch",
)
head_config.setdefault("metadata", {}).update(checkpoint_metadata_updates)
await branch_accessor.aupdate(
head_config,
branch_values(snapshot),
as_node="branch",
)
except _CHECKPOINT_MODE_ERRORS as exc:
raise _checkpoint_mode_http_error(exc, new_thread_id) from exc
except Exception:
@ -768,6 +875,29 @@ async def branch_thread(thread_id: str, body: ThreadBranchRequest, request: Requ
logger.exception("Failed to write branch thread_meta for %s", sanitize_log_param(new_thread_id))
raise HTTPException(status_code=500, detail="Failed to create branch") from None
# The thread feed (GET /messages, /messages/page) reads the run-event
# store, not checkpoints, and a fresh branch has no run_events — so the
# inherited history would vanish from the UI as soon as the branch's
# first run refreshes the feed (#4380 problem 2). Seed the branch's
# run_events from the same checkpoint snapshot the branch was created
# from. Best-effort: on failure the branch stays usable, with history
# visible only through the checkpoint overlay until it is re-branched.
try:
seed_events = build_branch_history_seed_events(
_checkpoint_messages(snapshot),
thread_id=new_thread_id,
run_id=f"branch-seed-{new_thread_id}",
parent_thread_id=thread_id,
)
if seed_events:
await get_run_event_store(request).put_batch(seed_events)
history_seed_mode = "seeded"
else:
history_seed_mode = "skipped_empty"
except Exception:
logger.exception("Failed to seed branch history run-events for thread %s", sanitize_log_param(new_thread_id))
history_seed_mode = "failed"
if branch_from_latest_turn:
workspace_clone_mode = await _copy_branch_user_data(thread_id, new_thread_id)
else:
@ -778,6 +908,7 @@ async def branch_thread(thread_id: str, body: ThreadBranchRequest, request: Requ
parent_checkpoint_id=parent_checkpoint_id,
branched_from_message_id=body.message_id,
workspace_clone_mode=workspace_clone_mode,
history_seed_mode=history_seed_mode,
)

3
backend/app/gateway/services.py
View File

@ -629,9 +629,10 @@ class _RawCheckpointSnapshot:
metadata, config ancestry, created_at) comes straight from the tuple.
"""
__slots__ = ("config", "values", "metadata", "parent_config", "created_at", "tasks", "tasks_known", "next")
__slots__ = ("checkpoint_exists", "config", "values", "metadata", "parent_config", "created_at", "tasks", "tasks_known", "next")
def __init__(self, config: dict[str, Any], tup: Any | None) -> None:
self.checkpoint_exists = tup is not None
self.config = getattr(tup, "config", None) or config
checkpoint = getattr(tup, "checkpoint", None) or {}
self.values = dict(checkpoint.get("channel_values") or {})

2
backend/docs/README.md
View File

@ -19,6 +19,7 @@ This directory contains detailed documentation for the DeerFlow backend.
| Document | Description |
|----------|-------------|
| [STREAMING.md](STREAMING.md) | Token-level streaming design: Gateway vs DeerFlowClient paths, `stream_mode` semantics, per-id dedup |
| [RUN_EVENT_STREAM.md](RUN_EVENT_STREAM.md) | Persisted run event stream contract: envelope, producers, consumers, and known gaps |
| [FILE_UPLOAD.md](FILE_UPLOAD.md) | File upload functionality |
| [PATH_EXAMPLES.md](PATH_EXAMPLES.md) | Path types and usage examples |
| [SANDBOX_MEMORY_PROFILING.md](SANDBOX_MEMORY_PROFILING.md) | Sandbox memory baseline and runtime comparison guide |
@ -54,6 +55,7 @@ docs/
├── summarization.md # Summarization feature
├── plan_mode_usage.md # Plan mode feature
├── STREAMING.md # Token-level streaming design
├── RUN_EVENT_STREAM.md # Persisted run event stream contract
├── AUTO_TITLE_GENERATION.md # Title generation
├── TITLE_GENERATION_IMPLEMENTATION.md # Title implementation details
└── TODO.md # Roadmap and issues

179
backend/docs/RUN_EVENT_STREAM.md Normal file
View File

@ -0,0 +1,179 @@
# Run Event Stream
The run event stream is DeerFlow's append-only record of what happened during
an agent run. Producers write through `RunEventStore`; history, debug, subtask,
memory-audit, and workspace-review consumers read projections of the same rows.
The machine-readable contract is
`contracts/run_event_stream_contract.json`. Canonical event names and
categories live in `deerflow.runtime.events.catalog`; conformance tests require
the runtime catalog and JSON contract to match exactly.
## Record Envelope
Every persisted event has these required fields:
| Field | Meaning |
| --- | --- |
| `thread_id` | Thread that owns the event. |
| `run_id` | Run that produced the event. |
| `seq` | Store-assigned sequence, strictly increasing within a thread. |
| `event_type` | Fixed event name or documented dynamic pattern. |
| `category` | Consumer-routing bucket. |
| `content` | Event payload, normally a string or JSON object. |
| `metadata` | Filterable or audit metadata. |
| `created_at` | Timezone-aware ISO-8601 timestamp. |
Backends may return additional fields. `DbRunEventStore`, for example, returns
`user_id` and may add serialization markers such as `content_is_json` to
metadata. Consumers must ignore unknown envelope and metadata fields.
`event_type` is limited to 32 characters and `category` to 16 characters by the
database schema. Catalog-backed definitions enforce the same limits before
writing so they cannot emit values that only the memory or JSONL store accepts.
`seq` is thread-global, not run-local. Memory and database stores assign it
monotonically for their supported deployment modes. JSONL only provides this
guarantee within one process; shared multi-process deployments must use the
database store.
## Categories
`category="message"` means an event is eligible for a message projection; it
does not guarantee that the row is visible in the UI. Thread-history APIs also
filter middleware model calls and superseded regenerate runs, and the frontend
honors message-level visibility markers such as `hide_from_ui`.
All other categories are excluded from message projections and are available
through run-event or specialized APIs:
| Category | Purpose |
| --- | --- |
| `trace` | Execution evidence. |
| `outputs` | Root graph completion output. |
| `error` | Callback-observed failure evidence. |
| `middleware` | Middleware state-change audit evidence. |
| `context` | Effective hidden-context identity. |
| `subagent` | Subagent lifecycle and step history. |
| `workspace` | Workspace/output file-change evidence. |
## Producers
`RunJournal` emits callback-derived events:
| Event type | Category | Producer |
| --- | --- | --- |
| `run.start` | `trace` | Root `on_chain_start()` |
| `run.end` | `outputs` | Root `on_chain_end()` |
| `run.error` | `error` | `on_chain_error()` |
| `llm.human.input` | `message` | First persisted lead-agent human input |
| `llm.ai.response` | `message` | `on_llm_end()` |
| `llm.tool.result` | `message` | `on_tool_end()` |
| `llm.error` | `trace` | `on_llm_error()` |
| `context:memory` | `context` | `record_memory_context()` |
| `middleware:{tag}` | `middleware` | `record_middleware()` |
Current middleware tags are `guardrail`, `safety_termination`,
`skill_activation`, and `skill_secrets`. The pattern is intentionally open so
new middleware tags are additive. Because the full event type is limited to 32
characters and `middleware:` uses 11, a tag must contain 1-21 characters.
### Opaque Run Outputs
`run.end.content` is the root graph output and is intentionally opaque. Its
nested representation is not currently identical across storage backends:
- `MemoryRunEventStore` retains the original Python container and nested
values.
- `JsonlRunEventStore` and `DbRunEventStore` serialize through
`json.dumps(default=str)`, so nested values that are not directly JSON
serializable are read back as strings.
Consumers may use `run.end` as completion evidence, but must not depend on
backend-identical nested output values. Normalizing those values would be a
separate runtime compatibility change rather than part of this current-state
contract.
`subagents/step_events.py::subagent_run_event()` maps streamed `task_*` chunks
to persisted events. The worker batches them through `put_batch()`:
| Event type | Source chunk | Required content |
| --- | --- | --- |
| `subagent.start` | `task_started` | `task_id`, `description` |
| `subagent.step` | `task_running` | `task_id`, `message_index`, `kind`, `text`, `truncated`; AI steps add `tool_calls`, tool steps add `tool_name` |
| `subagent.end` | terminal `task_*` | `task_id`, `status`; optional model, usage, result/error, and truncation fields |
Terminal subagent status is one of `completed`, `failed`, `cancelled`, or
`timed_out`.
Malformed lifecycle chunks are not persisted. Every chunk requires a non-empty
string `task_id`; `task_running` additionally requires a non-negative integer
`message_index` and a message object.
`workspace_changes.record_workspace_changes()` writes `workspace_changes` in
category `workspace` when a run changed files. Its string content is a summary;
the structured versioned summary, file list, and limits live in
`metadata.workspace_changes`.
The JSON contract defines required and optional payload fields using JSON
Schema. It is the authoritative field-level reference.
## Consumers
| Consumer | Read path and behavior |
| --- | --- |
| Frontend thread history | `GET /api/threads/{thread_id}/messages/page` scans `list_messages()`, removes middleware rows and superseded regenerate runs, then applies frontend message visibility rules. |
| Per-run message clients | Thread-scoped and stateless run message endpoints call `list_messages_by_run()`. |
| Run debug/audit | `GET /api/threads/{thread_id}/runs/{run_id}/events` calls `list_events()` and supports `event_types`, `task_id`, `limit`, and `after_seq`. |
| Historical subtask cards | Fetch `subagent.step` through the run-events endpoint, filtered and paginated by `task_id`. |
| Memory audit | Filters run events to `context:memory` and compares `content_sha256`; full memory text is not duplicated into the event store. |
| Workspace review | `GET /api/threads/{thread_id}/runs/{run_id}/workspace-changes` projects the latest `workspace_changes` payload. |
Token and cost summaries are not reconstructed by reading event rows.
`RunJournal` accumulates usage while callbacks fire, and the worker writes the
aggregates to `RunRow`.
External Langfuse/LangSmith tracing is a parallel callback pipeline, not a
`RunEventStore` consumer. It is correlated through trace metadata rather than
being derived from these rows.
Evaluation consumers discussed in #4243 are planned rather than present in
this tree. They should read evidence through `list_events()` and treat the
compatibility and terminal-state limits below as part of that integration.
## Compatibility
The existing mixture of dot-separated, colon-separated, and bare-word names is
frozen. This contract documents current behavior; it does not normalize names.
A rename, removal, category change, required-field removal, or required-field
type change is breaking and needs an explicit versioned migration or dual-write
period.
Adding a new event type or optional field is additive. Consumers must ignore
unknown event types and unknown optional fields. Producers must add a catalog
entry, update the JSON contract and this document, and extend the conformance
tests in the same change.
`ai_message` is a read-only legacy alias for `llm.ai.response`. Current
producers never emit it. Category-based message projections and store queries
for the last visible AI message recognize previously persisted alias rows, so
the `/messages/page` endpoint also attaches feedback correctly. The legacy
`/messages` endpoint still returns those rows but only enriches feedback for the
canonical name. Legacy aliases live outside the canonical catalog and must not
be used by new producers.
## Known Gaps
- Tool-call intent is embedded in `llm.ai.response.content.tool_calls`; it is
not a first-class event. A missing or timed-out result may have no dedicated
outcome event.
- `run.end.metadata.status` is only a root graph completion marker and is
always `success`. `RunRow.status` remains authoritative for lifecycle state,
and worker loss may leave no terminal event.
- Nested non-JSON values in `run.end.content` have backend-dependent
representations: memory retains Python values, while JSONL and database
stores read them back as strings.
- Loop detection and deferred-tool promotion do not currently emit middleware
events.
- Journal attribution, token accounting, and external tracing metadata still
depend on manual instrumentation at several LLM call sites.

12
backend/docs/STREAMING.md
View File

@ -7,7 +7,7 @@
## TL;DR
- DeerFlow 有**两条并行**的流式路径：**Gateway 路径**（async / HTTP SSE / JSON 序列化）服务浏览器和 IM 渠道；**DeerFlowClient 路径**（sync / in-process / 原生 LangChain 对象）服务 Jupyter、脚本、测试。它们**无法合并**——消费者模型不同。
- 两条路径都从 `create_agent()` 工厂出发，核心都是订阅 LangGraph 的 `stream_mode=["values", "messages", "custom"]`。`values` 是节点级 state 快照，`messages` 是 LLM token 级 delta，`custom` 是显式 `StreamWriter` 事件。**这三种模式不是详细程度的梯度，是三个独立的事件源**，要 token 流就必须显式订阅 `messages`。
- 两条路径都从 `create_agent()` 工厂出发，核心都是订阅 LangGraph 的 `stream_mode=["values", "messages", "custom"]`。`values` 是节点级 state 快照，`messages` 是 LLM token 级 delta，`custom` 是显式 `StreamWriter` 事件。DeerFlow 内置 custom 事件同时通过 callback dispatch 暴露为 `astream_events(version="v2")` 的 `on_custom_event`，供 AG-UI 等 callback 型消费者使用。**这些接口不是详细程度的梯度，而是独立事件源**，消费者必须订阅自己需要的接口。
- 嵌入式 client 为每个 `stream()` 调用维护三个 `set[str]`：`seen_ids` / `streamed_ids` / `counted_usage_ids`。三者看起来相似但管理**三个独立的不变式**，不能合并。
---
@ -65,11 +65,14 @@ flowchart LR
LG -->|"每个节点完成后"| V["values: 完整 state 快照"]
Node1 -->|"LLM 每产生一个 token"| M["messages: (AIMessageChunk, meta)"]
Node1 -->|"StreamWriter.write()"| C["custom: 任意 dict"]
Node1 -->|"emit_custom_event()"| E["DeerFlow custom event helper"]
E -->|"StreamWriter.write()"| C["custom: 任意 dict"]
E -->|"dispatch_custom_event()"| A["astream_events(v2): on_custom_event"]
class V values
class M messages
class C custom
class A custom
```
| Mode | 发射时机 | Payload | 粒度 |
@ -77,6 +80,9 @@ flowchart LR
| `values` | 每个 graph 节点完成后 | 完整 state dict（title、messages、artifacts）| 节点级 |
| `messages` | LLM 每次 yield 一个 chunk；tool 节点完成时 | `(AIMessageChunk \| ToolMessage, metadata_dict)` | token 级 |
| `custom` | 用户代码显式调用 `StreamWriter.write()` | 任意 dict | 应用定义 |
| `on_custom_event` | 用户代码调用 `dispatch_custom_event()`；通过 `astream_events(version="v2")` 消费 | `name` + 任意 `data` | 应用定义 |
DeerFlow 自身产生的事件必须通过 `deerflow.utils.custom_events` 的同步或异步 helper 发送，并且每个内置 payload 必须携带非空字符串 `type`；缺少合法 `type` 的 payload 只进入 `custom` stream，不会出现在 `astream_events`。helper 先写入 `custom` stream，再 best-effort dispatch callback；callback 名称取 payload 的 `type`，`data` 保留完整 payload。这样原生 Gateway / Web UI / `DeerFlowClient` 的 custom 事件不变，`astream_events` 消费者也能观察同一事件。callback dispatch 的普通异常只记 debug 日志，不允许打断原有 writer 链路；writer 的异常语义保持不变。
### 两套命名的由来
@ -293,6 +299,8 @@ sequenceDiagram
本文档的直接起因是 bytedance/deer-flow#1969：`DeerFlowClient.stream()` 原本只订阅 `["values", "custom"]`，**漏了 `"messages"`**。结果 `client.stream("hello")` 等价于一次性返回，视觉上和 `chat()` 没区别。
Custom 事件还有一条独立回归边界：`get_stream_writer()` 产生的 chunk 不会自动成为 `astream_events(version="v2")` 的 `on_custom_event`，而 callback dispatch 也不会自动进入 `stream_mode="custom"`。测试必须使用真实最小 LangGraph 同时锁定两种 API，断言每个消费者各收到一次且 payload 相同；仅 mock 任一函数无法证明协议互操作。
这类 bug 有三个结构性原因：
1. **多协议层命名**：`messages` / `messages-tuple` / HTTP SSE `messages` 是同一概念的三个名字。在其中一层出错不会在另外两层报错。

54
backend/packages/harness/deerflow/agents/lead_agent/agent.py
View File

@ -46,6 +46,7 @@ from deerflow.agents.middlewares.token_usage_middleware import TokenUsageMiddlew
from deerflow.agents.middlewares.tool_error_handling_middleware import build_lead_runtime_middlewares
from deerflow.agents.middlewares.view_image_middleware import ViewImageMiddleware
from deerflow.agents.thread_state import get_thread_state_schema, normalize_middleware_state_schemas
from deerflow.authz.tool_filter import apply_tool_authorization
from deerflow.config.agents_config import load_agent_config, validate_agent_name
from deerflow.config.app_config import AppConfig, get_app_config
from deerflow.config.memory_config import should_use_memory_tools
@ -273,6 +274,7 @@ def build_middlewares(
deferred_setup=None,
mcp_routing_middleware: AgentMiddleware | None = None,
user_id: str | None = None,
authorization_provider=None,
):
"""Build the lead-agent middleware chain based on runtime configuration.
@ -293,12 +295,22 @@ def build_middlewares(
deferred MCP schemas before the deferred filter runs.
user_id: Effective user ID for user-scoped skill loading. Passed through
to ``SkillActivationMiddleware`` so it can resolve per-user custom skills.
authorization_provider: Provider already resolved for assembly-time
filtering. Reused by the execution-time authorization middleware.
Returns:
List of middleware instances.
"""
resolved_app_config = app_config or get_app_config()
middlewares = build_lead_runtime_middlewares(app_config=resolved_app_config, lazy_init=True)
runtime_middleware_kwargs = {
"app_config": resolved_app_config,
"lazy_init": True,
}
if authorization_provider is not None:
runtime_middleware_kwargs["authorization_provider"] = authorization_provider
if authorization_provider is not None and deferred_setup is not None:
runtime_middleware_kwargs["deferred_setup"] = deferred_setup
middlewares = build_lead_runtime_middlewares(**runtime_middleware_kwargs)
# Always inject current date (and optionally memory) as <system-reminder> into the
# first HumanMessage to keep the system prompt fully static for prefix-cache reuse.
@ -622,16 +634,26 @@ def _make_lead_agent(config: RunnableConfig, *, app_config: AppConfig):
configured_tools = raw_tools
if non_interactive:
configured_tools = [tool for tool in configured_tools if tool.name not in _NON_INTERACTIVE_DISABLED_TOOL_NAMES]
authorization_candidates = [*configured_tools]
if skill_setup.describe_skill_tool:
authorization_candidates.append(skill_setup.describe_skill_tool)
if should_use_memory_tools(resolved_app_config.memory):
_append_memory_tools_without_name_conflicts(authorization_candidates)
configured_tool_ids = {id(tool) for tool in configured_tools}
authorized_tools, _authz_provider = apply_tool_authorization(
authorization_candidates,
context=cfg,
app_config=resolved_app_config,
)
configured_tools = [tool for tool in authorized_tools if id(tool) in configured_tool_ids]
late_tools = [tool for tool in authorized_tools if id(tool) not in configured_tool_ids]
final_tools, setup = assemble_deferred_tools(configured_tools, enabled=resolved_app_config.tool_search.enabled)
final_tools.extend(late_tools)
mcp_routing_middleware = build_mcp_routing_middleware(
final_tools,
setup,
top_k=resolved_app_config.tool_search.auto_promote_top_k,
)
if skill_setup.describe_skill_tool:
final_tools.append(skill_setup.describe_skill_tool)
if should_use_memory_tools(resolved_app_config.memory):
_append_memory_tools_without_name_conflicts(final_tools)
return create_agent(
model=create_chat_model(name=model_name, thinking_enabled=thinking_enabled, app_config=resolved_app_config, attach_tracing=False),
tools=final_tools,
@ -644,6 +666,7 @@ def _make_lead_agent(config: RunnableConfig, *, app_config: AppConfig):
deferred_setup=setup,
mcp_routing_middleware=mcp_routing_middleware,
user_id=resolved_user_id,
authorization_provider=_authz_provider,
),
mode,
),
@ -691,17 +714,27 @@ def _make_lead_agent(config: RunnableConfig, *, app_config: AppConfig):
configured_tools = raw_tools + extra_tools
if non_interactive:
configured_tools = [tool for tool in configured_tools if tool.name not in _NON_INTERACTIVE_DISABLED_TOOL_NAMES]
authorization_candidates = [*configured_tools]
if skill_setup.describe_skill_tool:
authorization_candidates.append(skill_setup.describe_skill_tool)
if should_use_memory_tools(resolved_app_config.memory):
_append_memory_tools_without_name_conflicts(authorization_candidates)
configured_tool_ids = {id(tool) for tool in configured_tools}
authorized_tools, _authz_provider = apply_tool_authorization(
authorization_candidates,
context=cfg,
app_config=resolved_app_config,
)
configured_tools = [tool for tool in authorized_tools if id(tool) in configured_tool_ids]
late_tools = [tool for tool in authorized_tools if id(tool) not in configured_tool_ids]
final_tools, setup = assemble_deferred_tools(configured_tools, enabled=resolved_app_config.tool_search.enabled)
final_tools.extend(late_tools)
mcp_routing_middleware = build_mcp_routing_middleware(
final_tools,
setup,
top_k=resolved_app_config.tool_search.auto_promote_top_k,
)
mcp_routing_hints_section = get_mcp_routing_hints_prompt_section(configured_tools, deferred_names=setup.deferred_names)
if skill_setup.describe_skill_tool:
final_tools.append(skill_setup.describe_skill_tool)
if should_use_memory_tools(resolved_app_config.memory):
_append_memory_tools_without_name_conflicts(final_tools)
mcp_routing_hints_section = get_mcp_routing_hints_prompt_section(authorized_tools, deferred_names=setup.deferred_names)
return create_agent(
model=create_chat_model(name=model_name, thinking_enabled=thinking_enabled, reasoning_effort=reasoning_effort, app_config=resolved_app_config, attach_tracing=False, model_overrides=agent_model_overrides),
tools=final_tools,
@ -715,6 +748,7 @@ def _make_lead_agent(config: RunnableConfig, *, app_config: AppConfig):
deferred_setup=setup,
mcp_routing_middleware=mcp_routing_middleware,
user_id=resolved_user_id,
authorization_provider=_authz_provider,
),
mode,
),

61
backend/packages/harness/deerflow/agents/middlewares/llm_error_handling_middleware.py
View File

@ -23,6 +23,7 @@ from langchain_core.messages import AIMessage
from langgraph.errors import GraphBubbleUp
from deerflow.config.app_config import AppConfig
from deerflow.utils.custom_events import aemit_custom_event, emit_custom_event
logger = logging.getLogger(__name__)
@ -709,6 +710,26 @@ class LLMErrorHandlingMiddleware(AgentMiddleware[AgentState]):
detail=_extract_error_detail(exc),
)
def _build_retry_event(
self,
attempt: int,
wait_ms: int,
reason: str,
*,
max_attempts: int,
) -> dict[str, Any]:
return {
"type": "llm_retry",
"attempt": attempt,
# Effective budget for this call (burst-rate == 2), not the
# configured ceiling - the frontend renders this and the
# ``message`` below, so both must describe the loop that runs.
"max_attempts": max_attempts,
"wait_ms": wait_ms,
"reason": reason,
"message": self._build_retry_message(attempt, wait_ms, reason, max_attempts=max_attempts),
}
def _emit_retry_event(
self,
attempt: int,
@ -721,22 +742,36 @@ class LLMErrorHandlingMiddleware(AgentMiddleware[AgentState]):
from langgraph.config import get_stream_writer
writer = get_stream_writer()
writer(
{
"type": "llm_retry",
"attempt": attempt,
# Effective budget for this call (burst-rate == 2), not the
# configured ceiling - the frontend renders this and the
# ``message`` below, so both must describe the loop that runs.
"max_attempts": max_attempts,
"wait_ms": wait_ms,
"reason": reason,
"message": self._build_retry_message(attempt, wait_ms, reason, max_attempts=max_attempts),
}
emit_custom_event(
self._build_retry_event(attempt, wait_ms, reason, max_attempts=max_attempts),
writer=writer,
)
except GraphBubbleUp:
raise
except Exception:
logger.debug("Failed to emit llm_retry event", exc_info=True)
async def _aemit_retry_event(
self,
attempt: int,
wait_ms: int,
reason: str,
*,
max_attempts: int,
) -> None:
try:
from langgraph.config import get_stream_writer
writer = get_stream_writer()
await aemit_custom_event(
self._build_retry_event(attempt, wait_ms, reason, max_attempts=max_attempts),
writer=writer,
)
except GraphBubbleUp:
raise
except Exception:
logger.debug("Failed to emit async llm_retry event", exc_info=True)
@override
def wrap_model_call(
self,
@ -834,7 +869,7 @@ class LLMErrorHandlingMiddleware(AgentMiddleware[AgentState]):
wait_ms,
_extract_error_detail(exc),
)
self._emit_retry_event(attempt, wait_ms, reason, max_attempts=max_attempts)
await self._aemit_retry_event(attempt, wait_ms, reason, max_attempts=max_attempts)
await asyncio.sleep(wait_ms / 1000)
attempt += 1
continue

181
backend/packages/harness/deerflow/agents/middlewares/safety_finish_reason_middleware.py
View File

@ -1,6 +1,8 @@
"""Suppress tool execution when the provider safety-terminated the response.
"""Repair AIMessages the provider safety-terminated so they are neither
executed nor persisted empty.
Background — see issue bytedance/deer-flow#3028.
Background — see issues bytedance/deer-flow#3028 (truncated tool calls) and
#4393 (empty response poisons the thread).
Some providers (OpenAI ``finish_reason='content_filter'``, Anthropic
``stop_reason='refusal'``, Gemini ``finish_reason='SAFETY'`` ...) can stop
@ -12,11 +14,22 @@ they were complete. The agent then sees the truncated file, tries to fix it,
gets filtered again, and loops.
This middleware sits at ``after_model`` and gates that behaviour: when a
configured ``SafetyTerminationDetector`` fires *and* the AIMessage carries
tool calls, we strip the tool calls (both structured and raw provider
payloads), append a user-facing explanation, and stash observability fields
in ``additional_kwargs.safety_termination`` so logs, traces, and SSE
consumers can see what happened.
configured ``SafetyTerminationDetector`` fires it either
* strips the AIMessage's tool calls (both structured and raw provider
payloads) when it carries any — the truncated-tool-call case (#3028), or
* backfills a user-facing explanation when the message is otherwise blank
(no tool calls, no visible content) — the empty-response case (#4393),
where the empty assistant message would otherwise be persisted and then
rejected by strict OpenAI-compatible providers on every following request
("message ... with role 'assistant' must not be empty"), stranding the
whole thread until a new chat is started.
A safety-terminated message that carries visible text but no tool calls is
left untouched so its partial answer still reaches the user. Either way we
append the explanation and stash observability fields in
``additional_kwargs.safety_termination`` so logs, traces, and SSE consumers
can see what happened.
Hook choice: ``after_model`` (not ``wrap_model_call``) because the response
is a *normal* return — not an exception — and we want to participate in the
@ -35,11 +48,13 @@ and Loop then accounts against the cleaned message.
from __future__ import annotations
import logging
from dataclasses import dataclass
from typing import TYPE_CHECKING, override
from langchain.agents import AgentState
from langchain.agents.middleware import AgentMiddleware
from langchain_core.messages import AIMessage
from langgraph.errors import GraphBubbleUp
from langgraph.runtime import Runtime
from deerflow.agents.middlewares.safety_termination_detectors import (
@ -48,6 +63,9 @@ from deerflow.agents.middlewares.safety_termination_detectors import (
default_detectors,
)
from deerflow.agents.middlewares.tool_call_metadata import clone_ai_message_with_tool_calls
from deerflow.runtime.events.catalog import MIDDLEWARE_SAFETY_TERMINATION_TAG
from deerflow.utils.custom_events import aemit_custom_event, emit_custom_event
from deerflow.utils.messages import message_content_to_text
if TYPE_CHECKING:
from deerflow.config.safety_finish_reason_config import SafetyFinishReasonConfig
@ -63,9 +81,24 @@ _USER_FACING_MESSAGE = (
"or ask for a narrower output."
)
# Used when the safety termination produced no tool calls *and* no content:
# the message is rewritten only so it is not persisted empty (see #4393), so
# it must not claim any tool calls were suppressed.
_USER_FACING_EMPTY_MESSAGE = "The model provider stopped this response with a safety-related signal ({reason_field}={reason_value!r}, detector={detector!r}) and returned no content. Please rephrase your request or start a new conversation."
@dataclass(frozen=True)
class _SafetyIntervention:
update: dict
termination: SafetyTermination
suppressed_names: list[str]
message: AIMessage
tool_calls: list[dict]
class SafetyFinishReasonMiddleware(AgentMiddleware[AgentState]):
"""Strip tool_calls from AIMessages flagged by a SafetyTerminationDetector."""
"""Repair AIMessages flagged by a SafetyTerminationDetector: strip tool
calls, or backfill an explanation when the message is otherwise empty."""
def __init__(self, detectors: list[SafetyTerminationDetector] | None = None) -> None:
super().__init__()
@ -135,8 +168,10 @@ class SafetyFinishReasonMiddleware(AgentMiddleware[AgentState]):
message: AIMessage,
termination: SafetyTermination,
) -> AIMessage:
suppressed_names = [tc.get("name") or "unknown" for tc in (message.tool_calls or [])]
explanation = _USER_FACING_MESSAGE.format(
tool_calls = message.tool_calls or []
suppressed_names = [tc.get("name") or "unknown" for tc in tool_calls]
template = _USER_FACING_MESSAGE if tool_calls else _USER_FACING_EMPTY_MESSAGE
explanation = template.format(
reason_field=termination.reason_field,
reason_value=termination.reason_value,
detector=termination.detector,
@ -167,6 +202,25 @@ class SafetyFinishReasonMiddleware(AgentMiddleware[AgentState]):
# ----- observability ---------------------------------------------------
@staticmethod
def _build_event_payload(
termination: SafetyTermination,
suppressed_names: list[str],
runtime: Runtime,
) -> dict:
thread_id = None
if runtime is not None and getattr(runtime, "context", None):
thread_id = runtime.context.get("thread_id") if isinstance(runtime.context, dict) else None
return {
"type": "safety_termination",
"detector": termination.detector,
"reason_field": termination.reason_field,
"reason_value": termination.reason_value,
"suppressed_tool_call_count": len(suppressed_names),
"suppressed_tool_call_names": suppressed_names,
"thread_id": thread_id,
}
def _emit_event(
self,
termination: SafetyTermination,
@ -181,29 +235,42 @@ class SafetyFinishReasonMiddleware(AgentMiddleware[AgentState]):
from langgraph.config import get_stream_writer
writer = get_stream_writer()
except GraphBubbleUp:
raise
except Exception: # noqa: BLE001
logger.debug("get_stream_writer unavailable; skipping safety_termination event", exc_info=True)
return
thread_id = None
if runtime is not None and getattr(runtime, "context", None):
thread_id = runtime.context.get("thread_id") if isinstance(runtime.context, dict) else None
try:
writer(
{
"type": "safety_termination",
"detector": termination.detector,
"reason_field": termination.reason_field,
"reason_value": termination.reason_value,
"suppressed_tool_call_count": len(suppressed_names),
"suppressed_tool_call_names": suppressed_names,
"thread_id": thread_id,
}
)
emit_custom_event(self._build_event_payload(termination, suppressed_names, runtime), writer=writer)
except GraphBubbleUp:
raise
except Exception: # noqa: BLE001
logger.debug("Failed to emit safety_termination stream event", exc_info=True)
async def _aemit_event(
self,
termination: SafetyTermination,
suppressed_names: list[str],
runtime: Runtime,
) -> None:
try:
from langgraph.config import get_stream_writer
writer = get_stream_writer()
except GraphBubbleUp:
raise
except Exception: # noqa: BLE001
logger.debug("get_stream_writer unavailable; skipping async safety_termination event", exc_info=True)
return
try:
await aemit_custom_event(self._build_event_payload(termination, suppressed_names, runtime), writer=writer)
except GraphBubbleUp:
raise
except Exception: # noqa: BLE001
logger.debug("Failed to emit async safety_termination stream event", exc_info=True)
def _record_audit_event(
self,
termination: SafetyTermination,
@ -251,7 +318,7 @@ class SafetyFinishReasonMiddleware(AgentMiddleware[AgentState]):
try:
journal.record_middleware(
tag="safety_termination",
tag=MIDDLEWARE_SAFETY_TERMINATION_TAG,
name=type(self).__name__,
hook="after_model",
action="suppress_tool_calls",
@ -259,11 +326,11 @@ class SafetyFinishReasonMiddleware(AgentMiddleware[AgentState]):
)
except Exception: # noqa: BLE001
# Audit-event persistence must never break agent execution.
logger.debug("Failed to record middleware:safety_termination event", exc_info=True)
logger.warning("Failed to record middleware:safety_termination event", exc_info=True)
# ----- main apply ------------------------------------------------------
def _apply(self, state: AgentState, runtime: Runtime) -> dict | None:
def _prepare_intervention(self, state: AgentState, runtime: Runtime) -> _SafetyIntervention | None:
messages = state.get("messages", [])
if not messages:
return None
@ -272,17 +339,34 @@ class SafetyFinishReasonMiddleware(AgentMiddleware[AgentState]):
if not isinstance(last, AIMessage):
return None
# Issue scope: only intervene when there's something to suppress.
# ``content_filter`` without tool_calls is allowed through unchanged
# so the partial text response (if any) reaches the user naturally.
tool_calls = last.tool_calls
if not tool_calls:
# Two provider-safety failure modes are worth rewriting; a safety
# termination that produced visible text with no tool calls is left
# untouched so the partial answer still reaches the user naturally.
# 1. tool_calls present: they may be truncated/unsafe (#3028), so
# suppress them.
# 2. blank content and no tool_calls: an empty assistant message
# that strict OpenAI-compatible providers (Moonshot/Kimi, ...)
# reject on the *next* request ("message ... with role
# 'assistant' must not be empty", #4393), which poisons the whole
# thread until a new chat is started. Backfill an explanation so
# the persisted message is non-empty.
tool_calls = list(last.tool_calls or [])
# ``or ""`` normalizes every "no visible content" shape to blank:
# None, "", [] and whitespace all count. None is reachable via
# ``model_copy(update={"content": None})`` (a rewrite path that skips
# validation); without the guard message_content_to_text stringifies
# it to "None" and the backfill would be skipped, re-poisoning the
# thread this fix is meant to protect.
content_is_blank = not message_content_to_text(last.content or "").strip()
if not tool_calls and not content_is_blank:
return None
termination = self._detect(last)
if termination is None:
return None
backfilled_empty = content_is_blank and not tool_calls
# Stamp stop_reason so the worker can surface this capped completion
# alongside loop_capped / token_capped (#4176).
ctx = getattr(runtime, "context", None)
@ -295,21 +379,36 @@ class SafetyFinishReasonMiddleware(AgentMiddleware[AgentState]):
thread_id = runtime.context.get("thread_id") if isinstance(runtime.context, dict) else None
logger.warning(
"Provider safety termination detected — suppressed %d tool call(s)",
"Provider safety termination detected — suppressed %d tool call(s), backfilled_empty_content=%s",
len(tool_calls),
backfilled_empty,
extra={
"thread_id": thread_id,
"detector": termination.detector,
"reason_field": termination.reason_field,
"reason_value": termination.reason_value,
"suppressed_tool_call_names": [tc.get("name") for tc in tool_calls],
"backfilled_empty_content": backfilled_empty,
},
)
self._emit_event(termination, [tc.get("name") or "unknown" for tc in tool_calls], runtime)
self._record_audit_event(termination, last, list(tool_calls), runtime)
tool_calls = list(tool_calls)
return _SafetyIntervention(
update={"messages": [patched]},
termination=termination,
suppressed_names=[tc.get("name") or "unknown" for tc in tool_calls],
message=last,
tool_calls=tool_calls,
)
return {"messages": [patched]}
def _apply(self, state: AgentState, runtime: Runtime) -> dict | None:
intervention = self._prepare_intervention(state, runtime)
if intervention is None:
return None
self._emit_event(intervention.termination, intervention.suppressed_names, runtime)
self._record_audit_event(intervention.termination, intervention.message, intervention.tool_calls, runtime)
return intervention.update
# ----- hooks -----------------------------------------------------------
@ -319,4 +418,10 @@ class SafetyFinishReasonMiddleware(AgentMiddleware[AgentState]):
@override
async def aafter_model(self, state: AgentState, runtime: Runtime) -> dict | None:
return self._apply(state, runtime)
intervention = self._prepare_intervention(state, runtime)
if intervention is None:
return None
await self._aemit_event(intervention.termination, intervention.suppressed_names, runtime)
self._record_audit_event(intervention.termination, intervention.message, intervention.tool_calls, runtime)
return intervention.update

12
backend/packages/harness/deerflow/agents/middlewares/skill_activation_middleware.py
View File

@ -17,6 +17,10 @@ from langchain.agents.middleware import AgentMiddleware
from langchain.agents.middleware.types import ModelRequest, ModelResponse
from langchain_core.messages import AIMessage, HumanMessage
from deerflow.runtime.events.catalog import (
MIDDLEWARE_SKILL_ACTIVATION_TAG,
MIDDLEWARE_SKILL_SECRETS_TAG,
)
from deerflow.runtime.secret_context import (
_SECRETS_BINDING_AUDIT_KEY,
_SLASH_SKILL_ACTIVATION_RUN_KEY,
@ -294,7 +298,7 @@ Follow this skill before choosing a general workflow. Load supporting resources
return
try:
journal.record_middleware(
"skill_activation",
MIDDLEWARE_SKILL_ACTIVATION_TAG,
name="SkillActivationMiddleware",
hook=hook,
action="activate",
@ -306,7 +310,7 @@ Follow this skill before choosing a general workflow. Load supporting resources
},
)
except Exception:
logger.debug("Failed to record slash skill activation audit event", exc_info=True)
logger.warning("Failed to record slash skill activation audit event", exc_info=True)
def _prepare_model_request(self, request: ModelRequest, *, hook: str) -> tuple[ModelRequest | AIMessage | None, _Activation | None]:
run_context = self._run_context(request)
@ -533,14 +537,14 @@ Follow this skill before choosing a general workflow. Load supporting resources
return
try:
journal.record_middleware(
"skill_secrets",
MIDDLEWARE_SKILL_SECRETS_TAG,
name="SkillActivationMiddleware",
hook=hook,
action="bind_secrets",
changes=audit_state,
)
except Exception:
logger.debug("Failed to record skill secret binding audit event", exc_info=True)
logger.warning("Failed to record skill secret binding audit event", exc_info=True)
@staticmethod
def _make_activation_message(target: HumanMessage, activation_content: str) -> HumanMessage:

43
backend/packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py
View File

@ -157,6 +157,8 @@ def _build_runtime_middlewares(
include_uploads: bool,
include_dangling_tool_call_patch: bool,
lazy_init: bool = True,
authorization_provider=None,
authorization_infrastructure_tool_names: frozenset[str] = frozenset(),
) -> list[AgentMiddleware]:
"""Build shared base middlewares for agent execution."""
from deerflow.agents.middlewares.input_sanitization_middleware import InputSanitizationMiddleware
@ -199,7 +201,33 @@ def _build_runtime_middlewares(
tail.append(DanglingToolCallMiddleware())
tail.append(LLMErrorHandlingMiddleware(app_config=app_config))
# Guardrail middleware (if configured)
# Authorization uses the existing GuardrailMiddleware so execution-time
# deny, audit, and fail-closed handling stay in one proven implementation.
# It is appended before an explicit guardrail provider, making authorization
# the outer guard and avoiding an unnecessary external policy call for an
# already-denied tool.
authorization_config = app_config.authorization
if authorization_config.enabled is True:
if authorization_provider is None:
from deerflow.authz.runtime import resolve_authorization_provider
authorization_provider = resolve_authorization_provider(authorization_config)
if authorization_provider is not None:
from deerflow.authz.adapter import GuardrailAuthorizationAdapter
from deerflow.guardrails.middleware import GuardrailMiddleware
tail.append(
GuardrailMiddleware(
GuardrailAuthorizationAdapter(
authorization_provider,
default_role=authorization_config.default_role,
infrastructure_tool_names=authorization_infrastructure_tool_names,
),
fail_closed=authorization_config.fail_closed,
)
)
# Explicit guardrail middleware remains independently active when configured.
guardrails_config = app_config.guardrails
if guardrails_config.enabled and guardrails_config.provider:
import inspect
@ -264,13 +292,21 @@ def _build_runtime_middlewares(
return middlewares
def build_lead_runtime_middlewares(*, app_config: AppConfig, lazy_init: bool = True) -> list[AgentMiddleware]:
def build_lead_runtime_middlewares(
*,
app_config: AppConfig,
lazy_init: bool = True,
authorization_provider=None,
deferred_setup: "DeferredToolSetup | None" = None,
) -> list[AgentMiddleware]:
"""Middlewares shared by lead agent runtime before lead-only middlewares."""
return _build_runtime_middlewares(
app_config=app_config,
include_uploads=True,
include_dangling_tool_call_patch=True,
lazy_init=lazy_init,
authorization_provider=authorization_provider,
authorization_infrastructure_tool_names=(frozenset({deferred_setup.tool_search_tool.name}) if authorization_provider is not None and deferred_setup is not None and deferred_setup.tool_search_tool is not None else frozenset()),
)
@ -282,6 +318,7 @@ def build_subagent_runtime_middlewares(
deferred_setup: "DeferredToolSetup | None" = None,
mcp_routing_middleware: AgentMiddleware | None = None,
agent_name: str | None = None,
authorization_provider=None,
) -> list[AgentMiddleware]:
"""Middlewares shared by subagent runtime before subagent-only middlewares."""
if app_config is None:
@ -294,6 +331,8 @@ def build_subagent_runtime_middlewares(
include_uploads=False,
include_dangling_tool_call_patch=True,
lazy_init=lazy_init,
authorization_provider=authorization_provider,
authorization_infrastructure_tool_names=(frozenset({deferred_setup.tool_search_tool.name}) if authorization_provider is not None and deferred_setup is not None and deferred_setup.tool_search_tool is not None else frozenset()),
)
if model_name is None and app_config.models:

4
backend/packages/harness/deerflow/authz/__init__.py
View File

@ -1,10 +1,12 @@
"""Pluggable fine-grained authorization (resource-level RBAC and beyond)."""
from deerflow.authz.adapter import GuardrailAuthorizationAdapter
from deerflow.authz.enforcement import filter_tools_by_authorization
from deerflow.authz.principal import build_principal_from_context, normalize_authz_attributes
from deerflow.authz.provider import AuthorizationProvider, AuthzDecision, AuthzReason, AuthzRequest, Principal
from deerflow.authz.rbac import RbacAuthorizationProvider
from deerflow.authz.runtime import resolve_authorization_provider
from deerflow.authz.tool_filter import apply_tool_authorization
__all__ = [
"AuthzDecision",
@ -14,7 +16,9 @@ __all__ = [
"GuardrailAuthorizationAdapter",
"Principal",
"RbacAuthorizationProvider",
"apply_tool_authorization",
"build_principal_from_context",
"filter_tools_by_authorization",
"normalize_authz_attributes",
"resolve_authorization_provider",
]

22
backend/packages/harness/deerflow/authz/adapter.py
View File

@ -17,6 +17,8 @@ with consistent ``default_role`` and ``attributes`` semantics.
from __future__ import annotations
from collections.abc import Iterable
from deerflow.authz.principal import build_principal_from_context
from deerflow.authz.provider import AuthorizationProvider, AuthzDecision, AuthzRequest
from deerflow.guardrails.provider import GuardrailDecision, GuardrailReason, GuardrailRequest
@ -36,6 +38,10 @@ class GuardrailAuthorizationAdapter:
``AuthorizationConfig.default_role``.
resource_type: Resource type for all ``AuthzRequest`` instances.
action: Action for all ``AuthzRequest`` instances.
infrastructure_tool_names: Framework tools created from an already
authorized capability set. These may execute without a second
provider decision; callers must derive the names from the current
build's concrete deferred setup rather than from static config.
"""
name = "authorization"
@ -47,11 +53,23 @@ class GuardrailAuthorizationAdapter:
default_role: str = "user",
resource_type: str = "tool",
action: str = "call",
infrastructure_tool_names: Iterable[str] = (),
) -> None:
self._provider = provider
self._default_role = default_role
self._resource_type = resource_type
self._action = action
self._infrastructure_tool_names = frozenset(infrastructure_tool_names)
def _infrastructure_decision(self, request: GuardrailRequest) -> GuardrailDecision | None:
"""Allow framework tools created from an already-filtered capability set."""
if request.tool_name not in self._infrastructure_tool_names:
return None
return GuardrailDecision(
allow=True,
reasons=[GuardrailReason(code="authz.infrastructure_tool")],
policy_id="authz:infrastructure",
)
def _to_authz(self, gr: GuardrailRequest) -> AuthzRequest:
"""Map a guardrail request to an authorization request."""
@ -104,6 +122,8 @@ class GuardrailAuthorizationAdapter:
here would duplicate that logic and risk divergent behavior between
the two layers.
"""
if infrastructure_decision := self._infrastructure_decision(request):
return infrastructure_decision
decision = self._provider.authorize(self._to_authz(request))
return self._to_guardrail(decision)
@ -112,5 +132,7 @@ class GuardrailAuthorizationAdapter:
See :meth:`evaluate` for exception-propagation rationale.
"""
if infrastructure_decision := self._infrastructure_decision(request):
return infrastructure_decision
decision = await self._provider.aauthorize(self._to_authz(request))
return self._to_guardrail(decision)

42
backend/packages/harness/deerflow/authz/enforcement.py Normal file
View File

@ -0,0 +1,42 @@
"""Shared Phase 1B authorization enforcement helpers."""
from __future__ import annotations
import logging
from collections.abc import Sequence
from langchain_core.tools import BaseTool
from deerflow.authz.provider import AuthorizationProvider, Principal
logger = logging.getLogger(__name__)
def filter_tools_by_authorization(
tools: Sequence[BaseTool],
*,
provider: AuthorizationProvider | None,
principal: Principal,
fail_closed: bool,
) -> list[BaseTool]:
"""Return the policy-visible subset of *tools* without changing its order.
The caller must invoke this before deferred-tool assembly. Provider errors
and malformed filter results deny every tool when ``fail_closed`` is true;
an explicitly configured fail-open policy preserves the original set.
"""
original_tools = list(tools)
if provider is None:
return original_tools
candidates = [tool.name for tool in original_tools]
try:
allowed = provider.filter_resources(principal, "tool", candidates)
if not isinstance(allowed, list) or any(not isinstance(name, str) for name in allowed):
raise TypeError("AuthorizationProvider.filter_resources must return list[str]")
except Exception:
logger.exception("Authorization provider failed while filtering tools")
return [] if fail_closed else original_tools
allowed_names = set(allowed)
return [tool for tool in original_tools if tool.name in allowed_names]

11
backend/packages/harness/deerflow/authz/rbac.py
View File

@ -121,6 +121,14 @@ class RbacAuthorizationProvider:
compiled = self._compile_resource_policy(role_name, resource_key, resource_policy)
self._policies[(role_name, resource_key)] = compiled
def validate_role(self, role: str, *, field: str = "role") -> None:
"""Fail fast when an operator-configured role is not defined."""
role = _require_non_empty_string(role, field=field)
if role not in self._known_roles:
if field == "role":
raise ValueError(f"Unknown role '{role}'; known roles: {sorted(self._known_roles)}")
raise ValueError(f"{field} '{role}' is not defined; known roles: {sorted(self._known_roles)}")
@staticmethod
def _compile_resource_policy(
role_name: str,
@ -194,8 +202,7 @@ class RbacAuthorizationProvider:
if role is None or role == "":
raise ValueError("Principal has no role; cannot evaluate RBAC policy")
if role not in self._known_roles:
raise ValueError(f"Unknown role '{role}'; known roles: {sorted(self._known_roles)}")
self.validate_role(role)
resource = _require_non_empty_string(resource, field=resource_field)
resource_key = _RESOURCE_POLICY_KEYS.get(resource, resource)

8
backend/packages/harness/deerflow/authz/runtime.py
View File

@ -47,4 +47,12 @@ def resolve_authorization_provider(
if not isinstance(instance, AuthorizationProvider):
raise ValueError(f"Authorization provider '{class_path}' does not satisfy the AuthorizationProvider Protocol")
from deerflow.authz.rbac import RbacAuthorizationProvider
if isinstance(instance, RbacAuthorizationProvider):
try:
instance.validate_role(config.default_role, field="authorization.default_role")
except ValueError as err:
raise ValueError(f"Invalid authorization default_role for provider '{class_path}': {err}") from err
return instance

72
backend/packages/harness/deerflow/authz/tool_filter.py Normal file
View File

@ -0,0 +1,72 @@
"""Convenience wrapper for Layer 1 tool authorization filtering.
Combines provider resolution, Principal construction, and tool filtering into
a single call so the three assembly paths (lead agent, subagent, embedded
client) stay one-liners.
"""
from __future__ import annotations
from collections.abc import Mapping
from typing import Any
from langchain_core.tools import BaseTool
from deerflow.authz.enforcement import filter_tools_by_authorization
from deerflow.authz.principal import build_principal_from_context
from deerflow.authz.provider import AuthorizationProvider
from deerflow.authz.runtime import resolve_authorization_provider
from deerflow.config.app_config import AppConfig
def apply_tool_authorization(
tools: list[BaseTool],
*,
context: Mapping[str, Any],
app_config: AppConfig,
authorization_provider: AuthorizationProvider | None = None,
) -> tuple[list[BaseTool], AuthorizationProvider | None]:
"""Apply Layer 1 tool authorization filtering.
Resolves the provider (or reuses a caller-provided one so Layer 1 and
Layer 2 share a single instance), builds a Principal from *context*, and
filters *tools* in place by the provider's policy.
When ``authorization.enabled`` is false, this is a no-op: returns the
original tools and ``None``.
Args:
tools: Candidate tools (already skill-filtered etc.).
context: Runtime context mapping (the merged ``cfg`` dict or an
equivalent dict assembled from ``self.*`` fields).
app_config: The resolved AppConfig (used for authorization settings).
authorization_provider: An already-resolved provider, or ``None`` to
resolve from ``app_config.authorization`` here.
Returns:
``(filtered_tools, provider)`` — the filtered tool list and the
provider instance (for passing to Layer 2 middleware wiring, or
``None`` when authorization is disabled).
"""
authz_config = app_config.authorization
# Guard against Mock objects in tests: MagicMock attribute access returns
# a truthy child mock for ``enabled``, which would trigger provider
# resolution on a non-string ``provider.use``. Real AuthorizationConfig
# has ``enabled: bool``; if it's not actually ``True``, skip.
if authz_config.enabled is not True:
return tools, None
if authorization_provider is None:
authorization_provider = resolve_authorization_provider(authz_config)
if authorization_provider is None:
return tools, None
principal = build_principal_from_context(context, default_role=authz_config.default_role)
filtered = filter_tools_by_authorization(
tools,
provider=authorization_provider,
principal=principal,
fail_closed=authz_config.fail_closed,
)
return filtered, authorization_provider

102
backend/packages/harness/deerflow/client.py
View File

@ -17,6 +17,7 @@ Usage:
import asyncio
import concurrent.futures
import copy
import json
import logging
import mimetypes
@ -24,7 +25,7 @@ import os
import shutil
import tempfile
import uuid
from collections.abc import Generator, Sequence
from collections.abc import Generator, Mapping, Sequence
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Literal
@ -37,6 +38,7 @@ from langchain_core.runnables import RunnableConfig
from deerflow.agents.lead_agent.agent import build_middlewares
from deerflow.agents.lead_agent.prompt import apply_prompt_template, get_enabled_skills_for_config
from deerflow.agents.thread_state import get_thread_state_schema, normalize_middleware_state_schemas
from deerflow.authz.principal import build_principal_from_context
from deerflow.config.agents_config import AGENT_NAME_PATTERN
from deerflow.config.app_config import get_app_config, is_trace_correlation_enabled, reload_app_config
from deerflow.config.extensions_config import ExtensionsConfig, SkillStateConfig, get_extensions_config, reload_extensions_config
@ -68,6 +70,18 @@ from deerflow.uploads.manager import (
logger = logging.getLogger(__name__)
_EMBEDDED_AUTHORIZATION_CONTEXT_KEYS = frozenset(
{
"user_id",
"user_role",
"oauth_provider",
"oauth_id",
"channel_user_id",
"is_internal",
"authz_attributes",
}
)
def _run_async_from_sync(coro):
"""Run an async helper from this synchronous client API."""
@ -242,9 +256,27 @@ class DeerFlowClient:
recursion_limit=overrides.get("recursion_limit", 100),
)
def _ensure_agent(self, config: RunnableConfig):
def _ensure_agent(self, config: RunnableConfig, *, context: Mapping[str, Any] | None = None):
"""Create (or recreate) the agent when config-dependent params change."""
cfg = config.get("configurable", {})
cfg = dict(config.get("configurable", {}) or {})
if context is not None:
cfg.update(context)
authorization_identity = None
if self._app_config.authorization.enabled:
principal = build_principal_from_context(
cfg,
default_role=self._app_config.authorization.default_role,
)
authorization_identity = (
principal.user_id,
principal.role,
principal.oauth_provider,
principal.oauth_id,
principal.channel_user_id,
principal.is_internal,
copy.deepcopy(principal.attributes),
)
key = (
cfg.get("model_name"),
cfg.get("thinking_enabled"),
@ -255,6 +287,7 @@ class DeerFlowClient:
self._agent_name,
frozenset(self._available_skills) if self._available_skills is not None else None,
self._checkpoint_channel_mode,
authorization_identity,
)
if self._agent is not None and self._agent_config_key == key:
@ -267,15 +300,9 @@ class DeerFlowClient:
max_total_subagents = cfg.get("max_total_subagents", self._app_config.subagents.max_total_per_run)
tools = self._get_tools(model_name=model_name, subagent_enabled=subagent_enabled)
final_tools, deferred_setup = assemble_deferred_tools(tools, enabled=self._app_config.tool_search.enabled)
mcp_routing_middleware = build_mcp_routing_middleware(
final_tools,
deferred_setup,
top_k=self._app_config.tool_search.auto_promote_top_k,
)
mcp_routing_hints_section = get_mcp_routing_hints_prompt_section(tools, deferred_names=deferred_setup.deferred_names)
# Wire deferred skill discovery — mirrors agent.py so config flag works on both paths.
# Add framework-provided tools before authorization so Layer 1 sees
# every capability that can become model-visible.
skills_list = get_enabled_skills_for_config(self._app_config)
if self._available_skills is not None:
skills_list = [s for s in skills_list if s.name in self._available_skills]
@ -284,8 +311,31 @@ class DeerFlowClient:
enabled=self._app_config.skills.deferred_discovery,
container_base_path=self._app_config.skills.container_path,
)
late_tools = []
if skill_setup.describe_skill_tool:
final_tools.append(skill_setup.describe_skill_tool)
late_tools.append(skill_setup.describe_skill_tool)
# Apply authorization Layer 1 before deferred assembly.
from deerflow.authz.tool_filter import apply_tool_authorization
configured_tool_ids = {id(tool) for tool in tools}
authorized_tools, _authz_provider = apply_tool_authorization(
[*tools, *late_tools],
context=cfg,
app_config=self._app_config,
)
tools = [tool for tool in authorized_tools if id(tool) in configured_tool_ids]
late_tools = [tool for tool in authorized_tools if id(tool) not in configured_tool_ids]
final_tools, deferred_setup = assemble_deferred_tools(tools, enabled=self._app_config.tool_search.enabled)
final_tools.extend(late_tools)
mcp_routing_middleware = build_mcp_routing_middleware(
final_tools,
deferred_setup,
top_k=self._app_config.tool_search.auto_promote_top_k,
)
mcp_routing_hints_section = get_mcp_routing_hints_prompt_section(authorized_tools, deferred_names=deferred_setup.deferred_names)
effective_user_id = cfg.get("user_id") or get_effective_user_id()
kwargs: dict[str, Any] = {
# attach_tracing=False because ``stream()`` injects tracing
@ -304,7 +354,8 @@ class DeerFlowClient:
app_config=self._app_config,
deferred_setup=deferred_setup,
mcp_routing_middleware=mcp_routing_middleware,
user_id=get_effective_user_id(),
user_id=effective_user_id,
authorization_provider=_authz_provider,
),
self._checkpoint_channel_mode,
),
@ -317,7 +368,7 @@ class DeerFlowClient:
app_config=self._app_config,
deferred_names=deferred_setup.deferred_names,
mcp_routing_hints_section=mcp_routing_hints_section,
user_id=get_effective_user_id(),
user_id=effective_user_id,
skill_names=skill_setup.skill_names or None,
),
"state_schema": get_thread_state_schema(self._checkpoint_channel_mode),
@ -739,7 +790,9 @@ class DeerFlowClient:
message: User message text.
thread_id: Thread ID for conversation context. Auto-generated if None.
**kwargs: Override client defaults (model_name, thinking_enabled,
plan_mode, subagent_enabled, recursion_limit).
plan_mode, subagent_enabled, recursion_limit). Trusted embedded
callers may also provide user_id, user_role, oauth_provider,
oauth_id, channel_user_id, is_internal, and authz_attributes.
Yields:
StreamEvent with one of:
@ -786,23 +839,34 @@ class DeerFlowClient:
existing_callbacks = list(config.get("callbacks") or [])
config["callbacks"] = [*existing_callbacks, *tracing_callbacks]
run_id = str(uuid.uuid4())
context: dict[str, Any] = {"thread_id": thread_id, "run_id": run_id}
for key in _EMBEDDED_AUTHORIZATION_CONTEXT_KEYS:
if key in kwargs:
context[key] = kwargs[key]
configurable = config.get("configurable") or {}
deerflow_trace_id = get_current_trace_id()
effective_user_id = context.get("user_id") or get_effective_user_id()
if self._app_config.authorization.enabled:
# Match the existing user-scoped storage/tracing identity when an
# embedded caller relies on CurrentUser instead of an explicit
# user_id override. Layer 1, Layer 2, and the agent cache must see
# the same actor.
context["user_id"] = effective_user_id
inject_langfuse_metadata(
config,
thread_id=thread_id,
user_id=get_effective_user_id(),
user_id=effective_user_id,
assistant_id=self._agent_name or "lead-agent",
model_name=configurable.get("model_name") or self._model_name,
environment=self._environment or os.environ.get("DEER_FLOW_ENV") or os.environ.get("ENVIRONMENT"),
deerflow_trace_id=deerflow_trace_id,
)
self._ensure_agent(config)
self._ensure_agent(config, context=context)
run_id = str(uuid.uuid4())
state: dict[str, Any] = {"messages": [HumanMessage(content=message, additional_kwargs={"run_id": run_id})]}
context = {"thread_id": thread_id, "run_id": run_id}
if deerflow_trace_id:
context[DEERFLOW_TRACE_METADATA_KEY] = deerflow_trace_id
if self._agent_name:

11
backend/packages/harness/deerflow/constants.py
View File

@ -8,3 +8,14 @@ DEFAULT_SKILLS_CONTAINER_PATH = "/mnt/skills"
# the browser tools (which write here) and the scanner (which ignores it) import
# this single source of truth so the name cannot drift between them.
BROWSER_FRAMES_DIRNAME = ".browser-frames"
# Persisted run-event envelope limits. Runtime definitions and the ORM both
# import these from this dependency-free module so lower layers never need to
# initialize deerflow.runtime just to validate storage constraints.
RUN_EVENT_TYPE_MAX_LENGTH = 32
RUN_EVENT_CATEGORY_MAX_LENGTH = 16
# Workspace changes are produced below the runtime layer, so their persisted
# event identity also lives here rather than in the runtime event catalog.
WORKSPACE_CHANGES_EVENT_TYPE = "workspace_changes"
WORKSPACE_CHANGES_EVENT_CATEGORY = "workspace"

5
backend/packages/harness/deerflow/guardrails/middleware.py
View File

@ -14,6 +14,7 @@ from langgraph.types import Command
from deerflow.authz.principal import normalize_authz_attributes
from deerflow.guardrails.provider import GuardrailDecision, GuardrailProvider, GuardrailReason, GuardrailRequest
from deerflow.runtime.events.catalog import MIDDLEWARE_GUARDRAIL_TAG
logger = logging.getLogger(__name__)
@ -112,14 +113,14 @@ class GuardrailMiddleware(AgentMiddleware[AgentState]):
try:
journal.record_middleware(
tag="guardrail",
tag=MIDDLEWARE_GUARDRAIL_TAG,
name=type(self).__name__,
hook="wrap_tool_call",
action=action,
changes=changes,
)
except Exception: # noqa: BLE001
logger.debug("Failed to record middleware:guardrail event", exc_info=True)
logger.warning("Failed to record middleware:guardrail event", exc_info=True)
@override
def wrap_tool_call(

7
backend/packages/harness/deerflow/persistence/models/run_event.py
View File

@ -7,6 +7,7 @@ from datetime import UTC, datetime
from sqlalchemy import JSON, DateTime, Index, String, Text, UniqueConstraint
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.constants import RUN_EVENT_CATEGORY_MAX_LENGTH, RUN_EVENT_TYPE_MAX_LENGTH
from deerflow.persistence.base import Base
@ -20,9 +21,9 @@ class RunEventRow(Base):
# created before auth was introduced; populated by auth middleware on
# new writes and by the boot-time orphan migration on existing rows.
user_id: Mapped[str | None] = mapped_column(String(64), nullable=True, index=True)
event_type: Mapped[str] = mapped_column(String(32), nullable=False)
category: Mapped[str] = mapped_column(String(16), nullable=False)
# "message" | "trace" | "lifecycle"
event_type: Mapped[str] = mapped_column(String(RUN_EVENT_TYPE_MAX_LENGTH), nullable=False)
category: Mapped[str] = mapped_column(String(RUN_EVENT_CATEGORY_MAX_LENGTH), nullable=False)
# Category values and semantics are defined by runtime/events/catalog.py
content: Mapped[str] = mapped_column(Text, default="")
event_metadata: Mapped[dict] = mapped_column(JSON, default=dict)
seq: Mapped[int] = mapped_column(nullable=False)

113
backend/packages/harness/deerflow/runtime/events/catalog.py Normal file
View File

@ -0,0 +1,113 @@
"""Canonical names and categories for persisted run events.
Producers import these definitions instead of repeating event-name/category
pairs. The public JSON contract is checked against this catalog in backend
tests, so either side changing without the other fails CI.
"""
from __future__ import annotations
from dataclasses import dataclass
from deerflow.constants import (
RUN_EVENT_CATEGORY_MAX_LENGTH,
RUN_EVENT_TYPE_MAX_LENGTH,
WORKSPACE_CHANGES_EVENT_CATEGORY,
WORKSPACE_CHANGES_EVENT_TYPE,
)
def _validate_category(category: str) -> None:
if not category:
raise ValueError("Run event category must not be empty")
if len(category) > RUN_EVENT_CATEGORY_MAX_LENGTH:
raise ValueError(f"Run event category must not exceed {RUN_EVENT_CATEGORY_MAX_LENGTH} characters: {category!r}")
@dataclass(frozen=True, slots=True)
class RunEventDefinition:
event_type: str
category: str
def __post_init__(self) -> None:
if not self.event_type:
raise ValueError("Run event type must not be empty")
if len(self.event_type) > RUN_EVENT_TYPE_MAX_LENGTH:
raise ValueError(f"Run event type must not exceed {RUN_EVENT_TYPE_MAX_LENGTH} characters: {self.event_type!r}")
_validate_category(self.category)
@dataclass(frozen=True, slots=True)
class RunEventPattern:
pattern: str
prefix: str
category: str
def __post_init__(self) -> None:
_validate_category(self.category)
def event_type(self, suffix: str) -> str:
if not suffix:
raise ValueError("Run event suffix must not be empty")
max_suffix_length = RUN_EVENT_TYPE_MAX_LENGTH - len(self.prefix)
if len(suffix) > max_suffix_length:
raise ValueError(f"Run event suffix for {self.pattern!r} must not exceed {max_suffix_length} characters: {suffix!r}")
return f"{self.prefix}{suffix}"
RUN_START_EVENT = RunEventDefinition("run.start", "trace")
RUN_END_EVENT = RunEventDefinition("run.end", "outputs")
RUN_ERROR_EVENT = RunEventDefinition("run.error", "error")
LLM_HUMAN_INPUT_EVENT = RunEventDefinition("llm.human.input", "message")
LLM_AI_RESPONSE_EVENT = RunEventDefinition("llm.ai.response", "message")
LLM_TOOL_RESULT_EVENT = RunEventDefinition("llm.tool.result", "message")
LLM_ERROR_EVENT = RunEventDefinition("llm.error", "trace")
MEMORY_CONTEXT_EVENT = RunEventDefinition("context:memory", "context")
SUBAGENT_START_EVENT = RunEventDefinition("subagent.start", "subagent")
SUBAGENT_STEP_EVENT = RunEventDefinition("subagent.step", "subagent")
SUBAGENT_END_EVENT = RunEventDefinition("subagent.end", "subagent")
WORKSPACE_CHANGES_EVENT = RunEventDefinition(WORKSPACE_CHANGES_EVENT_TYPE, WORKSPACE_CHANGES_EVENT_CATEGORY)
MIDDLEWARE_EVENT_PATTERN = RunEventPattern(
pattern="middleware:{tag}",
prefix="middleware:",
category="middleware",
)
MIDDLEWARE_EVENT_TAG_MAX_LENGTH = RUN_EVENT_TYPE_MAX_LENGTH - len(MIDDLEWARE_EVENT_PATTERN.prefix)
MIDDLEWARE_GUARDRAIL_TAG = "guardrail"
MIDDLEWARE_SAFETY_TERMINATION_TAG = "safety_termination"
MIDDLEWARE_SKILL_ACTIVATION_TAG = "skill_activation"
MIDDLEWARE_SKILL_SECRETS_TAG = "skill_secrets"
MIDDLEWARE_EVENT_TAGS = (
MIDDLEWARE_GUARDRAIL_TAG,
MIDDLEWARE_SAFETY_TERMINATION_TAG,
MIDDLEWARE_SKILL_ACTIVATION_TAG,
MIDDLEWARE_SKILL_SECRETS_TAG,
)
JOURNAL_RUN_EVENT_DEFINITIONS = (
RUN_START_EVENT,
RUN_END_EVENT,
RUN_ERROR_EVENT,
LLM_HUMAN_INPUT_EVENT,
LLM_AI_RESPONSE_EVENT,
LLM_TOOL_RESULT_EVENT,
LLM_ERROR_EVENT,
MEMORY_CONTEXT_EVENT,
)
SUBAGENT_RUN_EVENT_DEFINITIONS = (
SUBAGENT_START_EVENT,
SUBAGENT_STEP_EVENT,
SUBAGENT_END_EVENT,
)
WORKSPACE_RUN_EVENT_DEFINITIONS = (WORKSPACE_CHANGES_EVENT,)
FIXED_RUN_EVENT_DEFINITIONS = (
*JOURNAL_RUN_EVENT_DEFINITIONS,
*SUBAGENT_RUN_EVENT_DEFINITIONS,
*WORKSPACE_RUN_EVENT_DEFINITIONS,
)

6
backend/packages/harness/deerflow/runtime/events/store/base.py
View File

@ -6,7 +6,8 @@ through the same interface, distinguished by the ``category`` field.
Implementations:
- MemoryRunEventStore: in-memory dict (development, tests)
- Future: DB-backed store (SQLAlchemy ORM), JSONL file store
- DbRunEventStore: SQLAlchemy ORM-backed persistence
- JsonlRunEventStore: JSONL file persistence for local/debug use
"""
from __future__ import annotations
@ -24,7 +25,8 @@ class RunEventStore(abc.ABC):
2. seq is strictly increasing within the same thread
3. list_messages() only returns category="message" events
4. list_events() returns all events for the specified run
5. Returned dicts match the RunEvent field structure
5. Returned dicts contain the required RunEvent envelope fields; backends
may add documented fields such as DbRunEventStore.user_id
"""
@abc.abstractmethod

165
backend/packages/harness/deerflow/runtime/journal.py
View File

@ -6,11 +6,11 @@ handles token usage accumulation.
Key design decisions:
- on_llm_new_token is NOT implemented -- only complete messages via on_llm_end
- on_chat_model_start captures structured prompts as llm_request (OpenAI format) and
- on_chat_model_start captures the first user-visible prompt as llm.human.input and
extracts the first human message for run.input, because it is more reliable than
on_chain_start (fires on every node) — messages here are fully structured.
- on_chain_start with parent_run_id=None emits a run.start trace marking root invocation.
- on_llm_end emits llm_response in OpenAI Chat Completions format
- on_llm_end emits llm.ai.response in checkpoint-aligned AIMessage.model_dump() format
- Token usage accumulated in memory, written to RunRow on run completion
- Caller identification via tags injection (lead_agent / subagent:{name} / middleware:{name})
"""
@ -20,16 +20,27 @@ from __future__ import annotations
import asyncio
import logging
import time
from collections.abc import Awaitable, Callable, Mapping
from collections.abc import Awaitable, Callable, Mapping, Sequence
from datetime import UTC, datetime
from typing import TYPE_CHECKING, Any, cast
from uuid import UUID
from langchain_core.callbacks import BaseCallbackHandler
from langchain_core.messages import AIMessage, AnyMessage, BaseMessage, HumanMessage, ToolMessage
from langchain_core.messages import AIMessage, AnyMessage, BaseMessage, HumanMessage, ToolMessage, messages_from_dict
from langgraph.types import Command
from deerflow.agents.human_input import read_human_input_response
from deerflow.runtime.events.catalog import (
LLM_AI_RESPONSE_EVENT,
LLM_ERROR_EVENT,
LLM_HUMAN_INPUT_EVENT,
LLM_TOOL_RESULT_EVENT,
MEMORY_CONTEXT_EVENT,
MIDDLEWARE_EVENT_PATTERN,
RUN_END_EVENT,
RUN_ERROR_EVENT,
RUN_START_EVENT,
)
from deerflow.utils.messages import message_to_text, restore_original_human_message
if TYPE_CHECKING:
@ -53,6 +64,102 @@ def _should_persist_human_input_message(message: BaseMessage) -> bool:
return response is not None and response["source"] in _PERSISTED_HIDDEN_HUMAN_INPUT_RESPONSE_SOURCES
def _coerce_seed_message(message: Any) -> Any:
"""Return ``message`` as a ``BaseMessage``, deserializing dict form if needed.
``_checkpoint_messages`` (threads.py) returns whatever the snapshot holds,
and its sibling branch-matching helpers all handle a message being either a
``BaseMessage`` or a ``model_dump()``-shaped dict (serde differences across
checkpoint backends/modes). The seed path must handle both too — otherwise a
dict-backed checkpoint seeds nothing and the branch silently reports
``skipped_empty`` while history exists. Unparseable dicts fall through
unchanged and are dropped by the ``isinstance(BaseMessage)`` guard.
"""
if isinstance(message, BaseMessage):
return message
if isinstance(message, Mapping):
msg_type = message.get("type")
if isinstance(msg_type, str) and msg_type:
try:
return messages_from_dict([{"type": msg_type, "data": dict(message)}])[0]
except Exception:
logger.warning("branch seed: could not deserialize checkpoint message dict (type=%s)", msg_type)
return message
def build_branch_history_seed_events(
messages: Sequence[Any],
*,
thread_id: str,
run_id: str,
parent_thread_id: str,
) -> list[dict]:
"""Serialize a branch checkpoint's messages into run-event message rows.
Thread branching copies checkpoint state, but the thread feed
(``list_messages`` / ``GET /threads/{id}/messages/page``) reads the
run-event store — which a fresh branch has no rows in, so the inherited
history vanishes from the UI as soon as the branch's first run refreshes
the feed (#4380). Seeding the branch's run_events from the same
checkpoint snapshot the branch was created from keeps the feed
consistent with what the branch actually contains.
Mirrors RunJournal's message-event contract so seeded rows are
indistinguishable from journaled ones except by the ``branch_seed``
marker: same event types, ``category="message"``, ``content=
message.model_dump()``, the human-input persistence rule
(``_should_persist_human_input_message``), the original-user-text
restoration, and the same treatment of ``hide_from_ui`` AI/tool rows —
RunJournal persists them (``on_llm_end`` / ``_persist_tool_result_message``
do not filter) and the frontend hides them client-side, so the seed writes
them too rather than dropping them.
The one deliberate divergence, because a checkpoint message carries no run
scope: AI rows omit RunJournal's run-scoped enrichment (``usage`` /
``latency_ms`` / ``llm_call_index``), and ``caller`` is stamped
``lead_agent`` rather than the message's original caller (unrecoverable
here). Neither is observable today — no consumer indexes those metadata
keys, and per-message ``caller`` drives no attribution (the ``by_caller``
usage panel is run-scoped, not fed from the message feed).
"""
events: list[dict] = []
created_at = datetime.now(UTC).isoformat()
seed_metadata = {"branch_seed": True, "branch_parent_thread_id": parent_thread_id}
for raw_message in messages:
message = _coerce_seed_message(raw_message)
if not isinstance(message, BaseMessage):
continue
if isinstance(message, HumanMessage):
if not _should_persist_human_input_message(message):
continue
event_type = "llm.human.input"
content = restore_original_human_message(message).model_dump()
metadata: dict[str, Any] = {"caller": "lead_agent", **seed_metadata}
elif isinstance(message, AIMessage):
event_type = "llm.ai.response"
content = message.model_dump()
metadata = {"caller": "lead_agent", **seed_metadata}
elif isinstance(message, ToolMessage):
event_type = "llm.tool.result"
content = message.model_dump()
metadata = dict(seed_metadata)
else:
# System / remove / summary artifacts never enter the thread feed.
continue
events.append(
{
"thread_id": thread_id,
"run_id": run_id,
"event_type": event_type,
"category": "message",
"content": content,
"metadata": metadata,
"created_at": created_at,
}
)
return events
class RunJournal(BaseCallbackHandler):
"""LangChain callback handler that captures events to RunEventStore."""
@ -156,8 +263,8 @@ class RunJournal(BaseCallbackHandler):
# Root graph invocation — emit a single trace event for the run start.
chain_name = (serialized or {}).get("name", "unknown")
self._put(
event_type="run.start",
category="trace",
event_type=RUN_START_EVENT.event_type,
category=RUN_START_EVENT.category,
content={"chain": chain_name},
metadata={"caller": caller, **(metadata or {})},
)
@ -175,13 +282,18 @@ class RunJournal(BaseCallbackHandler):
if parent_run_id is not None:
return
self._reconcile_final_tool_messages(outputs)
self._put(event_type="run.end", category="outputs", content=outputs, metadata={"status": "success"})
self._put(
event_type=RUN_END_EVENT.event_type,
category=RUN_END_EVENT.category,
content=outputs,
metadata={"status": "success"},
)
self._flush_sync()
def on_chain_error(self, error: BaseException, *, run_id: UUID, **kwargs: Any) -> None:
self._put(
event_type="run.error",
category="error",
event_type=RUN_ERROR_EVENT.event_type,
category=RUN_ERROR_EVENT.category,
content=str(error),
metadata={"error_type": type(error).__name__},
)
@ -198,7 +310,7 @@ class RunJournal(BaseCallbackHandler):
tags: list[str] | None = None,
**kwargs: Any,
) -> None:
"""Capture structured prompt messages for llm_request event.
"""Capture the first user-visible prompt as llm.human.input.
This is also the canonical place to extract the first human message:
messages are fully structured here, it fires only on real LLM calls,
@ -226,8 +338,8 @@ class RunJournal(BaseCallbackHandler):
persisted_message = restore_original_human_message(m)
self.set_first_human_message(self._message_text(persisted_message))
self._put(
event_type="llm.human.input",
category="message",
event_type=LLM_HUMAN_INPUT_EVENT.event_type,
category=LLM_HUMAN_INPUT_EVENT.category,
content=persisted_message.model_dump(),
metadata={"caller": caller},
)
@ -291,10 +403,10 @@ class RunJournal(BaseCallbackHandler):
call_index = self._llm_call_index
self._seen_llm_starts.add(rid)
# Trace event: llm_response (OpenAI completion format)
# Message event: checkpoint-aligned llm.ai.response payload.
self._put(
event_type="llm.ai.response",
category="message",
event_type=LLM_AI_RESPONSE_EVENT.event_type,
category=LLM_AI_RESPONSE_EVENT.category,
content=message.model_dump(),
metadata={
"caller": caller,
@ -342,7 +454,11 @@ class RunJournal(BaseCallbackHandler):
def on_llm_error(self, error: BaseException, *, run_id: UUID, **kwargs: Any) -> None:
self._llm_start_times.pop(str(run_id), None)
self._put(event_type="llm.error", category="trace", content=str(error))
self._put(
event_type=LLM_ERROR_EVENT.event_type,
category=LLM_ERROR_EVENT.category,
content=str(error),
)
def on_tool_start(self, serialized, input_str, *, run_id, parent_run_id=None, tags=None, metadata=None, inputs=None, **kwargs):
"""Handle tool start event, cache tool call ID for later correlation"""
@ -403,7 +519,11 @@ class RunJournal(BaseCallbackHandler):
self._current_run_tool_call_names[tool_call_id] = str(name or "")
def _persist_tool_result_message(self, message: BaseMessage) -> None:
self._put(event_type="llm.tool.result", category="message", content=message.model_dump())
self._put(
event_type=LLM_TOOL_RESULT_EVENT.event_type,
category=LLM_TOOL_RESULT_EVENT.category,
content=message.model_dump(),
)
identity = self._message_identity(message)
if identity:
self._persisted_tool_message_identities.add(identity)
@ -619,15 +739,16 @@ class RunJournal(BaseCallbackHandler):
Args:
tag: Short identifier for the middleware (e.g., "title", "summarize",
"guardrail"). Used to form event_type="middleware:{tag}".
"guardrail"). Used to form event_type="middleware:{tag}" and
limited by the persisted event-type column width.
name: Full middleware class name.
hook: Lifecycle hook that triggered the action (e.g., "after_model").
action: Specific action performed (e.g., "generate_title").
changes: Dict describing the state changes made.
"""
self._put(
event_type=f"middleware:{tag}",
category="middleware",
event_type=MIDDLEWARE_EVENT_PATTERN.event_type(tag),
category=MIDDLEWARE_EVENT_PATTERN.category,
content={"name": name, "hook": hook, "action": action, "changes": changes},
)
@ -642,8 +763,8 @@ class RunJournal(BaseCallbackHandler):
if self._memory_context_recorded:
return
self._put(
event_type="context:memory",
category="context",
event_type=MEMORY_CONTEXT_EVENT.event_type,
category=MEMORY_CONTEXT_EVENT.category,
content={"content_sha256": content_sha256},
)
self._memory_context_recorded = True

23
backend/packages/harness/deerflow/runtime/runs/manager.py
View File

@ -1130,13 +1130,28 @@ class RunManager:
# Still owned by a local task — skip
continue
try:
claimed = await self._call_store_with_retry(
"claim_for_takeover",
record.run_id,
lambda: self._store.claim_for_takeover(
record.run_id,
grace_seconds=grace_seconds,
error=error,
),
)
except Exception:
logger.warning("Failed to claim orphaned run %s for reconciliation", record.run_id, exc_info=True)
continue
if not claimed:
logger.info(
"Skipped orphaned run %s recovery because the takeover claim no longer matched",
record.run_id,
)
continue
record.status = RunStatus.error
record.error = error
record.updated_at = now
persisted = await self._persist_status(record, RunStatus.error, error=error)
if not persisted:
logger.warning("Skipped orphaned run %s recovery because error status was not persisted", record.run_id)
continue
recovered.append(record)
if recovered:

99
backend/packages/harness/deerflow/runtime/runs/worker.py
View File

@ -687,21 +687,24 @@ async def run_agent(
logger.info("Run %s abort requested — stopping", run_id)
break
mode, chunk = _unpack_stream_item(item, lg_modes, stream_subgraphs)
mode, chunk, namespace = _unpack_stream_item(item, lg_modes, stream_subgraphs)
if mode is None:
continue
llm_error_fallback_message = llm_error_fallback_message or _extract_llm_error_fallback_message(chunk, pre_existing_message_ids)
sse_event = _lg_mode_to_sse_event(mode)
if file_tool_chunk_batcher is not None and mode != "messages":
pending_chunks = file_tool_chunk_batcher.finish() if mode == "values" else file_tool_chunk_batcher.flush()
for publish_chunk in pending_chunks:
await bridge.publish(run_id, "messages", serialize(publish_chunk, mode="messages"))
chunks_to_publish = file_tool_chunk_batcher.push(chunk) if mode == "messages" and file_tool_chunk_batcher is not None else [chunk]
for publish_chunk in chunks_to_publish:
await bridge.publish(run_id, sse_event, serialize(publish_chunk, mode=mode))
if mode == "custom":
await subagent_events.add(chunk)
if not namespace:
# Only root-graph frames may decide the parent run's error
# fallback: a delegated subagent's marked fallback is the
# executor's to map (task_failed), not this run's.
llm_error_fallback_message = llm_error_fallback_message or _extract_llm_error_fallback_message(chunk, pre_existing_message_ids)
await _publish_stream_item(
bridge=bridge,
run_id=run_id,
mode=mode,
chunk=chunk,
namespace=namespace,
file_tool_chunk_batcher=file_tool_chunk_batcher,
subagent_events=subagent_events,
)
finally:
stream_error = sys.exception()
if file_tool_chunk_batcher is not None:
@ -1776,23 +1779,77 @@ def _unpack_stream_item(
item: Any,
lg_modes: list[str],
stream_subgraphs: bool,
) -> tuple[str | None, Any]:
"""Unpack a multi-mode or subgraph stream item into (mode, chunk).
) -> tuple[str | None, Any, tuple[str, ...]]:
"""Unpack a multi-mode or subgraph stream item into (mode, chunk, namespace).
Returns ``(None, None)`` if the item cannot be parsed.
``namespace`` is the subgraph namespace tuple LangGraph prefixes onto each
frame when ``subgraphs=True``; it is empty for root-graph frames. Delegated
subagent graphs inherit the parent's checkpoint namespace (see
``subagents/executor.py``), so their frames arrive here with a non-empty
namespace and must not be mistaken for root frames.
Returns ``(None, None, ())`` if the item cannot be parsed.
"""
if stream_subgraphs:
if isinstance(item, tuple) and len(item) == 3:
_ns, mode, chunk = item
return str(mode), chunk
ns, mode, chunk = item
namespace = tuple(str(part) for part in ns) if isinstance(ns, (list, tuple)) else (str(ns),)
return str(mode), chunk, namespace
if isinstance(item, tuple) and len(item) == 2:
mode, chunk = item
return str(mode), chunk
return None, None
return str(mode), chunk, ()
return None, None, ()
if isinstance(item, tuple) and len(item) == 2:
mode, chunk = item
return str(mode), chunk
return str(mode), chunk, ()
# Fallback: single-element output from first mode
return lg_modes[0] if lg_modes else None, item
return lg_modes[0] if lg_modes else None, item, ()
def _compose_sse_event(sse_event: str, namespace: tuple[str, ...]) -> str:
"""Namespace-qualified SSE event name, LangGraph Platform style.
Root frames keep the bare event name; subgraph frames become
``mode|ns1|ns2`` so clients can tell them apart. The LangGraph SDK parses
exactly this shape (``event.split("|").slice(1)``) and routes
subagent-namespaced values away from the thread view.
"""
if not namespace:
return sse_event
return "|".join((sse_event, *namespace))
async def _publish_stream_item(
*,
bridge: Any,
run_id: str,
mode: str,
chunk: Any,
namespace: tuple[str, ...],
file_tool_chunk_batcher: Any,
subagent_events: Any,
) -> None:
"""Publish one stream frame, preserving the subgraph namespace.
A subgraph frame published under a bare event name impersonates the root
graph: a delegated subagent's ``values`` snapshot then replaces the whole
thread view in SDK clients and its token chunks flood the parent message
stream (#4399). Subgraph frames therefore keep their namespace in the event
name and bypass the root-only consumers (file-tool chunk batcher, subagent
event persistence — task_* lifecycle events are root frames already).
"""
sse_event = _compose_sse_event(_lg_mode_to_sse_event(mode), namespace)
if namespace:
await bridge.publish(run_id, sse_event, serialize(chunk, mode=mode))
return
if file_tool_chunk_batcher is not None and mode != "messages":
pending_chunks = file_tool_chunk_batcher.finish() if mode == "values" else file_tool_chunk_batcher.flush()
for publish_chunk in pending_chunks:
await bridge.publish(run_id, "messages", serialize(publish_chunk, mode="messages"))
chunks_to_publish = file_tool_chunk_batcher.push(chunk) if mode == "messages" and file_tool_chunk_batcher is not None else [chunk]
for publish_chunk in chunks_to_publish:
await bridge.publish(run_id, sse_event, serialize(publish_chunk, mode=mode))
if mode == "custom":
await subagent_events.add(chunk)

24
backend/packages/harness/deerflow/subagents/executor.py
View File

@ -520,6 +520,9 @@ class SubagentExecutor:
"deferred_setup": deferred_setup,
"agent_name": self.config.name,
}
authz_provider = getattr(self, "_authz_provider", None)
if authz_provider is not None:
middleware_kwargs["authorization_provider"] = authz_provider
if mcp_routing_middleware is not None:
middleware_kwargs["mcp_routing_middleware"] = mcp_routing_middleware
middlewares = build_subagent_runtime_middlewares(**middleware_kwargs)
@ -650,6 +653,27 @@ class SubagentExecutor:
# Load skills as conversation items (Codex pattern)
skills = await self._load_skills()
filtered_tools = self._apply_skill_allowed_tools(skills)
# Apply authorization Layer 1: filter tools before deferred assembly
# so denied tools can never enter the DeferredToolCatalog.
from deerflow.authz.tool_filter import apply_tool_authorization
resolved_app_config = self.app_config or get_app_config()
authz_context = {
"user_id": self.user_id,
"user_role": self.user_role,
"oauth_provider": self.oauth_provider,
"oauth_id": self.oauth_id,
"channel_user_id": self.channel_user_id,
"is_internal": self.is_internal,
"authz_attributes": self.authz_attributes,
}
filtered_tools, self._authz_provider = apply_tool_authorization(
filtered_tools,
context=authz_context,
app_config=resolved_app_config,
)
# Assemble deferred tool_search AFTER policy filtering (fail-closed),
# mirroring the lead path so subagents stop binding full MCP schemas.
# The generated tool_search helper is intentionally not subject to the

37
backend/packages/harness/deerflow/subagents/step_events.py
View File

@ -23,6 +23,11 @@ from typing import Any
from langchain_core.messages import AIMessage, BaseMessage, ToolMessage
from deerflow.runtime.events.catalog import (
SUBAGENT_END_EVENT,
SUBAGENT_START_EVENT,
SUBAGENT_STEP_EVENT,
)
from deerflow.utils.messages import message_content_to_text
from .status_contract import normalize_token_usage
@ -36,7 +41,7 @@ SUBAGENT_STEP_MAX_CHARS = 8192
#: ``RunEvent.category`` for persisted subagent steps. A dedicated category (not
#: ``"message"``) keeps these events out of ``list_messages`` (the thread message
#: feed) while still being returned by ``list_events`` for fetch-on-expand (#3779).
SUBAGENT_EVENT_CATEGORY = "subagent"
SUBAGENT_EVENT_CATEGORY = SUBAGENT_START_EVENT.category
#: Map of ``task_*`` terminal custom-event types to their persisted status.
_TERMINAL_EVENT_STATUS: dict[str, str] = {
@ -193,8 +198,8 @@ def subagent_run_event(chunk: Any) -> dict[str, Any] | None:
Returns the ``event_type`` / ``category`` / ``content`` / ``metadata`` for a
persistable subagent lifecycle event, or ``None`` for any chunk that is not a
subagent event (so the worker only persists what it recognizes). ``thread_id``
/ ``run_id`` are filled in by the caller.
valid subagent event (so the worker only persists what it recognizes).
``thread_id`` / ``run_id`` are filled in by the caller.
"""
if not isinstance(chunk, dict):
return None
@ -204,21 +209,31 @@ def subagent_run_event(chunk: Any) -> dict[str, Any] | None:
return None
task_id = chunk.get("task_id")
if not isinstance(task_id, str) or not task_id:
return None
if event == "task_started":
description = chunk.get("description")
if description is not None and not isinstance(description, str):
return None
return {
"event_type": "subagent.start",
"category": SUBAGENT_EVENT_CATEGORY,
"content": {"task_id": task_id, "description": chunk.get("description")},
"event_type": SUBAGENT_START_EVENT.event_type,
"category": SUBAGENT_START_EVENT.category,
"content": {"task_id": task_id, "description": description},
"metadata": {"task_id": task_id},
}
if event == "task_running":
message_index = chunk.get("message_index")
message = chunk.get("message")
if isinstance(message_index, bool) or not isinstance(message_index, int) or message_index < 0:
return None
if not isinstance(message, dict):
return None
return {
"event_type": "subagent.step",
"category": SUBAGENT_EVENT_CATEGORY,
"content": build_subagent_step(chunk.get("message") or {}, task_id=task_id, message_index=message_index),
"event_type": SUBAGENT_STEP_EVENT.event_type,
"category": SUBAGENT_STEP_EVENT.category,
"content": build_subagent_step(message, task_id=task_id, message_index=message_index),
"metadata": {"task_id": task_id, "message_index": message_index},
}
@ -245,8 +260,8 @@ def subagent_run_event(chunk: Any) -> dict[str, Any] | None:
if error_truncated:
content["error_truncated"] = True
return {
"event_type": "subagent.end",
"category": SUBAGENT_EVENT_CATEGORY,
"event_type": SUBAGENT_END_EVENT.event_type,
"category": SUBAGENT_END_EVENT.category,
"content": content,
"metadata": {"task_id": task_id},
}

41
backend/packages/harness/deerflow/tools/builtins/task_tool.py
View File

@ -32,6 +32,7 @@ from deerflow.subagents.status_contract import (
)
from deerflow.tools.types import Runtime
from deerflow.trace_context import DEERFLOW_TRACE_METADATA_KEY, get_current_trace_id, normalize_trace_id
from deerflow.utils.custom_events import aemit_custom_event
if TYPE_CHECKING:
from deerflow.config.app_config import AppConfig
@ -419,13 +420,14 @@ async def task_tool(
writer = get_stream_writer()
# Send Task Started message'
writer(
await aemit_custom_event(
{
"type": "task_started",
"task_id": task_id,
"description": description,
"model_name": effective_model,
}
},
writer=writer,
)
try:
@ -434,7 +436,10 @@ async def task_tool(
if result is None:
logger.error(f"[trace={trace_id}] Task {task_id} not found in background tasks")
writer({"type": "task_failed", "task_id": task_id, "error": "Task disappeared from background tasks"})
await aemit_custom_event(
{"type": "task_failed", "task_id": task_id, "error": "Task disappeared from background tasks"},
writer=writer,
)
cleanup_background_task(task_id)
error = f"Task {task_id} disappeared from background tasks"
return _task_result_command(
@ -460,7 +465,7 @@ async def task_tool(
# Send task_running event for each new message
for i in range(last_message_count, current_message_count):
message = ai_messages[i]
writer(
await aemit_custom_event(
{
"type": "task_running",
"task_id": task_id,
@ -469,7 +474,8 @@ async def task_tool(
"total_messages": current_message_count,
"usage": usage,
"model_name": effective_model,
}
},
writer=writer,
)
logger.info(f"[trace={trace_id}] Task {task_id} sent message #{i + 1}/{current_message_count}")
last_message_count = current_message_count
@ -478,14 +484,15 @@ async def task_tool(
if result.status == SubagentStatus.COMPLETED:
_cache_subagent_usage(tool_call_id, usage, enabled=cache_token_usage)
_report_subagent_usage(runtime, result)
writer(
await aemit_custom_event(
{
"type": "task_completed",
"task_id": task_id,
"result": result.result,
"usage": usage,
"model_name": effective_model,
}
},
writer=writer,
)
logger.info(f"[trace={trace_id}] Task {task_id} completed after {poll_count} polls")
cleanup_background_task(task_id)
@ -503,14 +510,15 @@ async def task_tool(
elif result.status == SubagentStatus.FAILED:
_cache_subagent_usage(tool_call_id, usage, enabled=cache_token_usage)
_report_subagent_usage(runtime, result)
writer(
await aemit_custom_event(
{
"type": "task_failed",
"task_id": task_id,
"error": result.error,
"usage": usage,
"model_name": effective_model,
}
},
writer=writer,
)
logger.error(f"[trace={trace_id}] Task {task_id} failed: {result.error}")
cleanup_background_task(task_id)
@ -528,14 +536,15 @@ async def task_tool(
elif result.status == SubagentStatus.CANCELLED:
_cache_subagent_usage(tool_call_id, usage, enabled=cache_token_usage)
_report_subagent_usage(runtime, result)
writer(
await aemit_custom_event(
{
"type": "task_cancelled",
"task_id": task_id,
"error": result.error,
"usage": usage,
"model_name": effective_model,
}
},
writer=writer,
)
logger.info(f"[trace={trace_id}] Task {task_id} cancelled: {result.error}")
cleanup_background_task(task_id)
@ -549,14 +558,15 @@ async def task_tool(
elif result.status == SubagentStatus.TIMED_OUT:
_cache_subagent_usage(tool_call_id, usage, enabled=cache_token_usage)
_report_subagent_usage(runtime, result)
writer(
await aemit_custom_event(
{
"type": "task_timed_out",
"task_id": task_id,
"error": result.error,
"usage": usage,
"model_name": effective_model,
}
},
writer=writer,
)
logger.warning(f"[trace={trace_id}] Task {task_id} timed out: {result.error}")
cleanup_background_task(task_id)
@ -581,13 +591,14 @@ async def task_tool(
_report_subagent_usage(runtime, result)
usage = _summarize_usage(getattr(result, "token_usage_records", None))
_cache_subagent_usage(tool_call_id, usage, enabled=cache_token_usage)
writer(
await aemit_custom_event(
{
"type": "task_timed_out",
"task_id": task_id,
"usage": usage,
"model_name": effective_model,
}
},
writer=writer,
)
# The task may still be running in the background. Signal cooperative
# cancellation and schedule deferred cleanup to remove the entry from

57
backend/packages/harness/deerflow/utils/custom_events.py Normal file
View File

@ -0,0 +1,57 @@
"""Compatibility helpers for DeerFlow custom stream events."""
from __future__ import annotations
import logging
from collections.abc import Callable
from typing import Any
from langchain_core.callbacks import adispatch_custom_event, dispatch_custom_event
from langgraph.errors import GraphBubbleUp
logger = logging.getLogger(__name__)
StreamWriter = Callable[[Any], None]
def _event_name(payload: dict[str, Any]) -> str | None:
event_type = payload.get("type")
if isinstance(event_type, str) and event_type:
return event_type
logger.debug("Custom stream payload has no non-empty string 'type'; skipping callback dispatch")
return None
def emit_custom_event(payload: dict[str, Any], *, writer: StreamWriter) -> None:
"""Emit one event to LangGraph's custom stream and callback APIs.
The writer remains the primary compatibility path. Callback dispatch is
best-effort so an optional ``astream_events`` consumer cannot break an
existing DeerFlow run.
"""
writer(payload)
event_name = _event_name(payload)
if event_name is None:
return
try:
dispatch_custom_event(event_name, payload)
except GraphBubbleUp:
raise
except Exception:
logger.debug("Failed to dispatch custom callback event %s", event_name, exc_info=True)
async def aemit_custom_event(payload: dict[str, Any], *, writer: StreamWriter) -> None:
"""Async counterpart to :func:`emit_custom_event`."""
writer(payload)
event_name = _event_name(payload)
if event_name is None:
return
try:
await adispatch_custom_event(event_name, payload)
except GraphBubbleUp:
raise
except Exception:
logger.debug("Failed to dispatch async custom callback event %s", event_name, exc_info=True)

3
backend/packages/harness/deerflow/workspace_changes/recorder.py
View File

@ -12,6 +12,7 @@ from deerflow.config import get_paths
from .diff import compare_snapshots, get_changed_paths
from .scanner import scan_workspace_roots
from .types import (
WORKSPACE_CHANGES_EVENT_CATEGORY,
WORKSPACE_CHANGES_EVENT_TYPE,
WORKSPACE_CHANGES_METADATA_KEY,
WorkspaceChangeLimits,
@ -153,7 +154,7 @@ async def record_workspace_changes(
thread_id=thread_id,
run_id=run_id,
event_type=WORKSPACE_CHANGES_EVENT_TYPE,
category="workspace",
category=WORKSPACE_CHANGES_EVENT_CATEGORY,
content=content,
metadata={WORKSPACE_CHANGES_METADATA_KEY: payload},
)

6
backend/packages/harness/deerflow/workspace_changes/types.py
View File

@ -4,7 +4,11 @@ from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Literal
WORKSPACE_CHANGES_EVENT_TYPE = "workspace_changes"
from deerflow.constants import (
WORKSPACE_CHANGES_EVENT_CATEGORY as WORKSPACE_CHANGES_EVENT_CATEGORY,
)
from deerflow.constants import WORKSPACE_CHANGES_EVENT_TYPE as WORKSPACE_CHANGES_EVENT_TYPE
WORKSPACE_CHANGES_METADATA_KEY = "workspace_changes"
WorkspaceChangeStatus = Literal["created", "modified", "deleted", "symlink_created"]

717
backend/scripts/benchmark/bench_checkpoint_channels.py Normal file
View File

@ -0,0 +1,717 @@
#!/usr/bin/env python3
"""Benchmark DeerFlow's full and DeltaChannel checkpoint message storage.
The public CLI is a controller. Every benchmark case runs in a fresh child
process and, for SQLite, a fresh database. This mirrors the restart-required
checkpoint mode boundary and prevents one mode's graph/channel caches from
warming the other.
Examples::
PYTHONPATH=. uv run python scripts/benchmark/bench_checkpoint_channels.py \
--updates 10,100,500,999,1000,1001,2000 \
--payload-bytes 128,4096 \
--output checkpoint-bench.jsonl
PYTHONPATH=. uv run python scripts/benchmark/bench_checkpoint_channels.py \
--backends sqlite --updates 1000 --payload-bytes 128 \
--repetitions 7 --output snapshot-boundary.jsonl
The controller suppresses matrix pairs whose estimated cumulative full-mode
message payload exceeds ``--max-estimated-full-bytes``. Both modes are skipped
as a pair so every emitted full/delta result remains comparable. Use
``--allow-large-cases`` only on a machine provisioned for the resulting disk
and memory use.
"""
from __future__ import annotations
import argparse
import cProfile
import gc
import hashlib
import importlib.metadata
import json
import os
import platform
import statistics
import subprocess
import sys
import tempfile
import time
from collections.abc import Callable
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Annotated, Any, Literal, TypedDict
from langchain_core.messages import AIMessage, AnyMessage, BaseMessage, HumanMessage
from langgraph.channels import DeltaChannel
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.checkpoint.sqlite import SqliteSaver
from langgraph.graph import StateGraph
from langgraph.graph.message import add_messages
from deerflow.agents.thread_state import merge_message_writes
from deerflow.runtime.checkpoint_mode import inject_checkpoint_mode
from deerflow.runtime.checkpoint_state import CheckpointStateAccessor
try:
import resource
except ImportError: # pragma: no cover - Windows only
resource = None # type: ignore[assignment]
Mode = Literal["full", "delta"]
Backend = Literal["memory", "sqlite"]
SCHEMA_VERSION = 1
BENCHMARK_VERSION = 1
PRODUCTION_SNAPSHOT_FREQUENCY = 1000
DEFAULT_MAX_ESTIMATED_FULL_BYTES = 1024**3
_GIT_SHA_ENV = "DEERFLOW_CHECKPOINT_BENCH_GIT_SHA"
_MODES: tuple[Mode, ...] = ("full", "delta")
_BACKENDS: tuple[Backend, ...] = ("memory", "sqlite")
_STORAGE_STAT_FIELDS = (
"logical_checkpoint_bytes",
"logical_write_bytes",
"checkpoint_rows",
"write_rows",
)
class _FullBenchmarkState(TypedDict):
messages: Annotated[list[AnyMessage], add_messages]
class _DeltaBenchmarkState(TypedDict):
messages: Annotated[
list[AnyMessage],
DeltaChannel(merge_message_writes, snapshot_frequency=PRODUCTION_SNAPSHOT_FREQUENCY),
]
@dataclass(frozen=True)
class BenchmarkCase:
mode: Mode
backend: Backend
update_count: int
payload_bytes: int
repetition: int
seed: int
scenario: str = "append"
snapshot_frequency: int = PRODUCTION_SNAPSHOT_FREQUENCY
def __post_init__(self) -> None:
if self.mode not in _MODES:
raise ValueError(f"unsupported mode: {self.mode!r}")
if self.backend not in _BACKENDS:
raise ValueError(f"unsupported backend: {self.backend!r}")
if self.update_count <= 0:
raise ValueError("update_count must be positive")
if self.payload_bytes <= 0:
raise ValueError("payload_bytes must be positive")
if self.repetition < 0:
raise ValueError("repetition must be non-negative")
if self.snapshot_frequency != PRODUCTION_SNAPSHOT_FREQUENCY:
raise ValueError(f"Phase 1 supports only production snapshot_frequency={PRODUCTION_SNAPSHOT_FREQUENCY}")
if self.scenario != "append":
raise ValueError(f"unsupported scenario: {self.scenario!r}")
def _parse_positive_int_csv(value: str, *, option: str) -> list[int]:
if not value or value.startswith(",") or value.endswith(",") or ",," in value:
raise ValueError(f"{option} must be a comma-separated list of positive integers")
result: list[int] = []
seen: set[int] = set()
duplicates: list[int] = []
try:
parsed = [int(part.strip()) for part in value.split(",")]
except ValueError as exc:
raise ValueError(f"{option} must be a comma-separated list of positive integers") from exc
if any(item <= 0 for item in parsed):
raise ValueError(f"{option} values must be positive integers")
for item in parsed:
if item not in seen:
result.append(item)
seen.add(item)
elif item not in duplicates:
duplicates.append(item)
if duplicates:
print(
f"{option}: ignored duplicate value(s): {', '.join(str(item) for item in duplicates)}; use --repetitions for repeated samples.",
file=sys.stderr,
)
return result
def _parse_choice_csv(value: str, *, option: str, choices: tuple[str, ...]) -> list[str]:
if not value or value.startswith(",") or value.endswith(",") or ",," in value:
raise ValueError(f"{option} must contain one or more of: {', '.join(choices)}")
result: list[str] = []
duplicates: list[str] = []
for raw in value.split(","):
item = raw.strip()
if item not in choices:
raise ValueError(f"{option} contains unsupported value {item!r}; expected: {', '.join(choices)}")
if item not in result:
result.append(item)
elif item not in duplicates:
duplicates.append(item)
if duplicates:
print(
f"{option}: ignored duplicate value(s): {', '.join(duplicates)}; use --repetitions for repeated samples.",
file=sys.stderr,
)
return result
def _expand_cases(
*,
modes: list[str],
backends: list[str],
update_counts: list[int],
payload_bytes: list[int],
repetitions: int,
seed: int,
) -> list[BenchmarkCase]:
"""Build a matrix with alternating mode order to reduce order bias."""
cases: list[BenchmarkCase] = []
for repetition in range(repetitions):
for backend in backends:
for payload in payload_bytes:
for update_index, update_count in enumerate(update_counts):
ordered_modes = list(modes)
if (repetition + update_index) % 2 == 1:
ordered_modes.reverse()
for mode in ordered_modes:
cases.append(
BenchmarkCase(
mode=mode, # type: ignore[arg-type]
backend=backend, # type: ignore[arg-type]
update_count=update_count,
payload_bytes=payload,
repetition=repetition,
seed=seed,
)
)
return cases
def _estimated_full_payload_bytes(case: BenchmarkCase) -> int:
"""Lower-bound cumulative message content serialized by full mode."""
return case.payload_bytes * case.update_count * (case.update_count + 1) // 2
def _filter_oversized_pairs(cases: list[BenchmarkCase], *, max_bytes: int | None) -> tuple[list[BenchmarkCase], list[BenchmarkCase]]:
if max_bytes is None:
return cases, []
group_fields = ("backend", "scenario", "snapshot_frequency", "update_count", "payload_bytes", "repetition")
oversized_keys = {tuple(getattr(case, field) for field in group_fields) for case in cases if case.mode == "full" and _estimated_full_payload_bytes(case) > max_bytes}
kept = [case for case in cases if tuple(getattr(case, field) for field in group_fields) not in oversized_keys]
skipped = [case for case in cases if tuple(getattr(case, field) for field in group_fields) in oversized_keys]
return kept, skipped
def _message_for_update(index: int, payload_bytes: int) -> BaseMessage:
content = "x" * payload_bytes
message_id = f"bench-message-{index:08d}"
if index % 2 == 0:
return HumanMessage(id=message_id, content=content)
return AIMessage(id=message_id, content=content)
def _canonical_messages_digest(messages: list[AnyMessage]) -> str:
canonical = [
{
"id": message.id,
"type": message.type,
"content": message.content,
}
for message in messages
]
payload = json.dumps(canonical, ensure_ascii=False, separators=(",", ":"), sort_keys=True).encode("utf-8")
return hashlib.sha256(payload).hexdigest()
def _percentile(values: list[float], percentile: float) -> float:
if not values:
return 0.0
ordered = sorted(values)
if len(ordered) == 1:
return ordered[0]
rank = (len(ordered) - 1) * percentile / 100
lower = int(rank)
upper = min(lower + 1, len(ordered) - 1)
fraction = rank - lower
return ordered[lower] * (1 - fraction) + ordered[upper] * fraction
def _window_median(values: list[float], window: Literal["first", "middle", "last"]) -> float:
if not values:
return 0.0
width = max(1, len(values) // 10)
if window == "first":
selected = values[:width]
elif window == "last":
selected = values[-width:]
else:
center = len(values) // 2
start = max(0, center - width // 2)
selected = values[start : start + width]
return statistics.median(selected)
def _noop(_state: dict[str, Any]) -> dict[str, Any]:
return {}
def _build_graph(mode: Mode, saver: Any) -> Any:
schema = _DeltaBenchmarkState if mode == "delta" else _FullBenchmarkState
builder = StateGraph(schema)
builder.add_node("noop", _noop)
builder.set_entry_point("noop")
builder.set_finish_point("noop")
return builder.compile(checkpointer=saver)
def _config(case: BenchmarkCase) -> dict[str, Any]:
config: dict[str, Any] = {
"configurable": {
"thread_id": f"checkpoint-bench-{case.seed}-{case.repetition}",
}
}
inject_checkpoint_mode(config, case.mode)
return config
def _resolve_git_sha() -> str:
try:
return subprocess.run(
["git", "rev-parse", "HEAD"],
cwd=Path(__file__).resolve().parents[3],
check=True,
capture_output=True,
text=True,
timeout=5,
).stdout.strip()
except (OSError, subprocess.SubprocessError):
return "unknown"
def _base_row(case: BenchmarkCase) -> dict[str, Any]:
git_sha = os.environ.get(_GIT_SHA_ENV) or _resolve_git_sha()
try:
langgraph_version = importlib.metadata.version("langgraph")
except importlib.metadata.PackageNotFoundError:
langgraph_version = "unknown"
return {
"schema_version": SCHEMA_VERSION,
"benchmark_version": BENCHMARK_VERSION,
"success": True,
"error": None,
"profiled": False,
"git_sha": git_sha,
"python_version": platform.python_version(),
"langgraph_version": langgraph_version,
"platform": platform.platform(),
"mode": case.mode,
"backend": case.backend,
"scenario": case.scenario,
"snapshot_frequency": case.snapshot_frequency,
"update_count": case.update_count,
"payload_bytes": case.payload_bytes,
"repetition": case.repetition,
"seed": case.seed,
}
def _safe_error(error: BaseException | str, *, work_dir: Path | None = None) -> str:
message = str(error).replace(str(Path.home()), "<home>")
if work_dir is not None:
message = message.replace(str(work_dir), "<work-dir>")
return message[:2000]
def _collect_storage_stats(collector: Callable[[], dict[str, int]]) -> dict[str, Any]:
"""Keep timing data usable when a saver's diagnostic layout changes."""
try:
return {**collector(), "storage_stats_error": None}
except Exception as exc:
return {
**dict.fromkeys(_STORAGE_STAT_FIELDS),
"storage_stats_error": _safe_error(exc),
}
def _memory_storage_stats(saver: InMemorySaver, thread_id: str) -> dict[str, int]:
checkpoint_rows = 0
checkpoint_bytes = 0
for namespace in saver.storage.get(thread_id, {}).values():
for checkpoint, metadata, _parent_id in namespace.values():
checkpoint_rows += 1
checkpoint_bytes += len(checkpoint[1]) + len(metadata[1])
for (stored_thread_id, _namespace, _channel, _version), (_type_tag, blob) in saver.blobs.items():
if stored_thread_id == thread_id:
checkpoint_bytes += len(blob)
write_rows = 0
write_bytes = 0
for (stored_thread_id, _namespace, _checkpoint_id), writes in saver.writes.items():
if stored_thread_id != thread_id:
continue
for _task_id, _channel, (_type_tag, blob), _task_path in writes.values():
write_rows += 1
write_bytes += len(blob)
return {
"logical_checkpoint_bytes": checkpoint_bytes,
"logical_write_bytes": write_bytes,
"checkpoint_rows": checkpoint_rows,
"write_rows": write_rows,
}
def _sqlite_storage_stats(saver: SqliteSaver, thread_id: str) -> dict[str, int]:
with saver.cursor(transaction=False) as cursor:
checkpoint_rows, checkpoint_bytes = cursor.execute(
"SELECT COUNT(*), COALESCE(SUM(length(checkpoint) + length(metadata)), 0) FROM checkpoints WHERE thread_id = ?",
(thread_id,),
).fetchone()
write_rows, write_bytes = cursor.execute(
"SELECT COUNT(*), COALESCE(SUM(length(value)), 0) FROM writes WHERE thread_id = ?",
(thread_id,),
).fetchone()
return {
"logical_checkpoint_bytes": int(checkpoint_bytes),
"logical_write_bytes": int(write_bytes),
"checkpoint_rows": int(checkpoint_rows),
"write_rows": int(write_rows),
}
def _file_size(path: Path) -> int:
try:
return path.stat().st_size
except FileNotFoundError:
return 0
def _peak_rss_bytes() -> int | None:
if resource is None:
return None
peak_rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
return int(peak_rss) if sys.platform == "darwin" else int(peak_rss * 1024)
def _write_and_read(case: BenchmarkCase, saver: Any, messages: list[BaseMessage]) -> tuple[dict[str, Any], list[AnyMessage]]:
graph = _build_graph(case.mode, saver)
accessor = CheckpointStateAccessor.bind(graph, saver, mode=case.mode)
config = _config(case)
update_latencies: list[float] = []
write_start = time.perf_counter()
for message in messages:
update_start = time.perf_counter()
graph.invoke({"messages": [message]}, config)
update_latencies.append((time.perf_counter() - update_start) * 1000)
write_total_ms = (time.perf_counter() - write_start) * 1000
warm_start = time.perf_counter()
snapshot = accessor.get(config)
warm_read_ms = (time.perf_counter() - warm_start) * 1000
materialized = list(snapshot.values.get("messages", []))
metrics = {
"write_total_ms": write_total_ms,
"write_p50_ms": _percentile(update_latencies, 50),
"write_p95_ms": _percentile(update_latencies, 95),
"write_p99_ms": _percentile(update_latencies, 99),
"write_first_window_ms": _window_median(update_latencies, "first"),
"write_middle_window_ms": _window_median(update_latencies, "middle"),
"write_last_window_ms": _window_median(update_latencies, "last"),
"warm_read_ms": warm_read_ms,
}
return metrics, materialized
def _cold_read(case: BenchmarkCase, saver: Any) -> tuple[float, list[AnyMessage]]:
graph = _build_graph(case.mode, saver)
accessor = CheckpointStateAccessor.bind(graph, saver, mode=case.mode)
gc.collect()
start = time.perf_counter()
snapshot = accessor.get(_config(case))
elapsed_ms = (time.perf_counter() - start) * 1000
return elapsed_ms, list(snapshot.values.get("messages", []))
def _validate_materialized(case: BenchmarkCase, expected: list[BaseMessage], warm: list[AnyMessage], cold: list[AnyMessage]) -> tuple[int, str]:
expected_digest = _canonical_messages_digest(expected)
warm_digest = _canonical_messages_digest(warm)
cold_digest = _canonical_messages_digest(cold)
if len(warm) != case.update_count or len(cold) != case.update_count:
raise AssertionError(f"expected {case.update_count} messages, materialized warm={len(warm)} cold={len(cold)}")
if warm_digest != expected_digest or cold_digest != expected_digest:
raise AssertionError("materialized message content or ordering differs from deterministic input")
return len(cold), cold_digest
def _run_memory_case(case: BenchmarkCase, messages: list[BaseMessage]) -> dict[str, Any]:
saver = InMemorySaver()
metrics, warm = _write_and_read(case, saver, messages)
stats = _collect_storage_stats(lambda: _memory_storage_stats(saver, _config(case)["configurable"]["thread_id"]))
cold_read_ms, cold = _cold_read(case, saver)
actual_count, digest = _validate_materialized(case, messages, warm, cold)
return {
**metrics,
**stats,
"cold_read_ms": cold_read_ms,
# InMemorySaver has no durable external storage to reopen. The cold
# sample rebuilds the graph/channel table over the same saver only.
"saver_reopen_ms": 0.0,
"db_bytes": None,
"wal_bytes": None,
"shm_bytes": None,
"durable_db_bytes": None,
"expected_message_count": case.update_count,
"actual_message_count": actual_count,
"content_sha256": digest,
}
def _run_sqlite_case(case: BenchmarkCase, messages: list[BaseMessage], db_path: Path) -> dict[str, Any]:
with SqliteSaver.from_conn_string(str(db_path)) as saver:
saver.setup()
metrics, warm = _write_and_read(case, saver, messages)
stats = _collect_storage_stats(lambda: _sqlite_storage_stats(saver, _config(case)["configurable"]["thread_id"]))
db_bytes = _file_size(db_path)
wal_bytes = _file_size(Path(f"{db_path}-wal"))
shm_bytes = _file_size(Path(f"{db_path}-shm"))
durable_db_bytes = _file_size(db_path)
reopen_start = time.perf_counter()
with SqliteSaver.from_conn_string(str(db_path)) as reopened:
reopened.setup()
saver_reopen_ms = (time.perf_counter() - reopen_start) * 1000
cold_read_ms, cold = _cold_read(case, reopened)
actual_count, digest = _validate_materialized(case, messages, warm, cold)
return {
**metrics,
**stats,
"cold_read_ms": cold_read_ms,
"saver_reopen_ms": saver_reopen_ms,
"db_bytes": db_bytes,
"wal_bytes": wal_bytes,
"shm_bytes": shm_bytes,
"durable_db_bytes": durable_db_bytes,
"expected_message_count": case.update_count,
"actual_message_count": actual_count,
"content_sha256": digest,
}
def _run_case(case: BenchmarkCase, *, work_dir: Path) -> dict[str, Any]:
row = _base_row(case)
messages = [_message_for_update(index, case.payload_bytes) for index in range(case.update_count)]
try:
if case.backend == "memory":
measured = _run_memory_case(case, messages)
else:
measured = _run_sqlite_case(case, messages, work_dir / "checkpoint-benchmark.sqlite")
reducer_writes = [[message] for message in messages]
reducer_start = time.perf_counter()
reduced = merge_message_writes([], reducer_writes)
reducer_replay_ms = (time.perf_counter() - reducer_start) * 1000
if _canonical_messages_digest(reduced) != _canonical_messages_digest(messages):
raise AssertionError("standalone reducer diagnostic produced incorrect state")
row.update(measured)
row["reducer_replay_ms"] = reducer_replay_ms
row["peak_rss_bytes"] = _peak_rss_bytes()
except Exception as exc:
row["success"] = False
row["error"] = _safe_error(exc, work_dir=work_dir)
return row
def _run_profiled_case(case: BenchmarkCase, *, work_dir: Path, profile_path: Path) -> dict[str, Any]:
profile_path.parent.mkdir(parents=True, exist_ok=True)
profiler = cProfile.Profile()
row = profiler.runcall(_run_case, case, work_dir=work_dir)
row["profiled"] = True
profiler.dump_stats(profile_path)
return row
def _comparison_key(row: dict[str, Any]) -> tuple[Any, ...]:
return tuple(
row.get(field)
for field in (
"backend",
"scenario",
"snapshot_frequency",
"update_count",
"payload_bytes",
"repetition",
)
)
def _validate_cross_mode_rows(rows: list[dict[str, Any]]) -> None:
grouped: dict[tuple[Any, ...], list[dict[str, Any]]] = {}
for row in rows:
grouped.setdefault(_comparison_key(row), []).append(row)
for group in grouped.values():
successful = [row for row in group if row.get("success")]
modes = {row.get("mode") for row in successful}
if not {"full", "delta"}.issubset(modes):
continue
signatures = {(row.get("actual_message_count"), row.get("content_sha256")) for row in successful if row.get("mode") in {"full", "delta"}}
if len(signatures) == 1:
continue
for row in successful:
row["success"] = False
row["error"] = "cross-mode materialized state mismatch"
def _failure_row(case: BenchmarkCase, error: str) -> dict[str, Any]:
row = _base_row(case)
row["success"] = False
row["error"] = _safe_error(error)
return row
def _profile_filename(case: BenchmarkCase) -> str:
return f"{case.backend}-{case.mode}-updates-{case.update_count}-payload-{case.payload_bytes}-rep-{case.repetition}.prof"
def _run_child_case(case: BenchmarkCase, *, timeout_seconds: float, git_sha: str, profile_dir: Path | None = None) -> dict[str, Any]:
encoded_case = json.dumps(asdict(case), separators=(",", ":"))
command = [sys.executable, str(Path(__file__).resolve()), "--worker-case", encoded_case]
if profile_dir is not None:
command.extend(["--worker-profile", str(profile_dir / _profile_filename(case))])
started = time.perf_counter()
child_env = os.environ.copy()
child_env[_GIT_SHA_ENV] = git_sha
try:
completed = subprocess.run(
command,
check=False,
capture_output=True,
text=True,
timeout=timeout_seconds,
env=child_env,
)
except subprocess.TimeoutExpired:
return _failure_row(case, f"child process timed out after {timeout_seconds:g} seconds")
child_process_ms = (time.perf_counter() - started) * 1000
output_lines = [line for line in completed.stdout.splitlines() if line.strip()]
if not output_lines:
return _failure_row(case, f"child process returned {completed.returncode} without a result")
try:
row = json.loads(output_lines[-1])
except json.JSONDecodeError:
return _failure_row(case, f"child process returned {completed.returncode} with malformed JSON")
row["child_process_ms"] = child_process_ms
if completed.returncode != 0 and row.get("success"):
row["success"] = False
row["error"] = f"child process exited with status {completed.returncode}"
return row
def _build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description="Benchmark full and delta checkpoint message channels")
parser.add_argument("--modes", default="full,delta", help="Comma-separated modes (default: full,delta)")
parser.add_argument("--backends", default="memory,sqlite", help="Comma-separated backends (default: memory,sqlite)")
parser.add_argument("--updates", default="10,100", help="Comma-separated message update counts (default: 10,100)")
parser.add_argument("--payload-bytes", default="128", help="Comma-separated exact message content sizes (default: 128)")
parser.add_argument("--repetitions", type=int, default=3)
parser.add_argument("--seed", type=int, default=1)
parser.add_argument("--timeout-seconds", type=float, default=900)
parser.add_argument(
"--max-estimated-full-bytes",
type=int,
default=DEFAULT_MAX_ESTIMATED_FULL_BYTES,
help=("Skip comparable pairs whose estimated cumulative full-mode payload exceeds this value. The cap applies only when full mode is selected; delta-only diagnostics bypass it."),
)
parser.add_argument("--allow-large-cases", action="store_true", help="Disable the estimated cumulative full-payload safety cap")
parser.add_argument("--profile-dir", type=Path, help="Write one cProfile file per case; profiling inflates timings")
parser.add_argument("--output", type=Path)
parser.add_argument("--worker-case", help=argparse.SUPPRESS)
parser.add_argument("--worker-profile", type=Path, help=argparse.SUPPRESS)
return parser
def _worker_main(encoded_case: str, *, profile_path: Path | None = None) -> int:
try:
case = BenchmarkCase(**json.loads(encoded_case))
except (TypeError, ValueError, json.JSONDecodeError) as exc:
print(json.dumps({"schema_version": SCHEMA_VERSION, "benchmark_version": BENCHMARK_VERSION, "success": False, "error": _safe_error(exc)}, separators=(",", ":")))
return 2
with tempfile.TemporaryDirectory(prefix="deerflow-checkpoint-benchmark-") as temp_dir:
if profile_path is None:
row = _run_case(case, work_dir=Path(temp_dir))
else:
row = _run_profiled_case(case, work_dir=Path(temp_dir), profile_path=profile_path)
print(json.dumps(row, ensure_ascii=False, separators=(",", ":")))
return 0 if row.get("success") else 1
def main(argv: list[str] | None = None) -> int:
parser = _build_parser()
args = parser.parse_args(argv)
if args.worker_case is not None:
return _worker_main(args.worker_case, profile_path=args.worker_profile)
if args.output is None:
parser.error("--output is required")
try:
modes = _parse_choice_csv(args.modes, option="--modes", choices=_MODES)
backends = _parse_choice_csv(args.backends, option="--backends", choices=_BACKENDS)
updates = _parse_positive_int_csv(args.updates, option="--updates")
payload_bytes = _parse_positive_int_csv(args.payload_bytes, option="--payload-bytes")
if args.repetitions <= 0:
raise ValueError("--repetitions must be positive")
if args.timeout_seconds <= 0:
raise ValueError("--timeout-seconds must be positive")
if not args.allow_large_cases and args.max_estimated_full_bytes <= 0:
raise ValueError("--max-estimated-full-bytes must be positive")
except ValueError as exc:
parser.error(str(exc))
cases = _expand_cases(
modes=modes,
backends=backends,
update_counts=updates,
payload_bytes=payload_bytes,
repetitions=args.repetitions,
seed=args.seed,
)
cases, skipped = _filter_oversized_pairs(cases, max_bytes=None if args.allow_large_cases else args.max_estimated_full_bytes)
if skipped:
skipped_pairs = len(skipped) // max(1, len(modes))
print(
f"Skipping {skipped_pairs} oversized comparable case pair(s); use --allow-large-cases to run them.",
file=sys.stderr,
)
if not cases:
print("No benchmark cases remain after applying the safety cap.", file=sys.stderr)
return 2
git_sha = _resolve_git_sha()
rows: list[dict[str, Any]] = []
for index, case in enumerate(cases, start=1):
print(
f"[{index}/{len(cases)}] {case.backend} {case.mode} updates={case.update_count} payload={case.payload_bytes} repetition={case.repetition}",
file=sys.stderr,
)
rows.append(_run_child_case(case, timeout_seconds=args.timeout_seconds, git_sha=git_sha, profile_dir=args.profile_dir))
_validate_cross_mode_rows(rows)
args.output.parent.mkdir(parents=True, exist_ok=True)
with args.output.open("w", encoding="utf-8") as output_file:
for row in rows:
output_file.write(json.dumps(row, ensure_ascii=False, separators=(",", ":")) + "\n")
failures = sum(1 for row in rows if not row.get("success"))
print(f"Wrote {len(rows)} result row(s) to {args.output}; failures={failures}", file=sys.stderr)
return 1 if failures else 0
if __name__ == "__main__":
raise SystemExit(main())

196
backend/scripts/benchmark/summarize_checkpoint_channels.py Normal file
View File

@ -0,0 +1,196 @@
#!/usr/bin/env python3
"""Summarize paired full/delta checkpoint benchmark JSONL results.
Only repetitions with one successful ``full`` row and one successful ``delta``
row enter metric medians. This prevents a failed or missing mode from turning
different workloads into a misleading ratio. Ratios are always ``delta/full``:
values below 1 mean delta used less time or storage.
Examples::
python scripts/benchmark/summarize_checkpoint_channels.py results.jsonl
python scripts/benchmark/summarize_checkpoint_channels.py results/*.jsonl --json
python scripts/benchmark/summarize_checkpoint_channels.py results.jsonl \
--metrics write_total_ms,cold_read_ms,logical_checkpoint_bytes --csv
"""
from __future__ import annotations
import argparse
import csv
import json
import statistics
import sys
from collections import defaultdict
from pathlib import Path
from typing import Any
DEFAULT_METRICS = [
"write_total_ms",
"write_p50_ms",
"write_p95_ms",
"write_last_window_ms",
"warm_read_ms",
"cold_read_ms",
"logical_checkpoint_bytes",
"logical_write_bytes",
"durable_db_bytes",
"reducer_replay_ms",
"peak_rss_bytes",
]
_GROUP_FIELDS = (
"backend",
"scenario",
"snapshot_frequency",
"update_count",
"payload_bytes",
)
_INPUT_INDEX_FIELD = "_benchmark_input_index"
def _load_jsonl(paths: list[Path]) -> list[dict[str, Any]]:
rows: list[dict[str, Any]] = []
errors: list[str] = []
for input_index, path in enumerate(paths):
with path.open("r", encoding="utf-8") as input_file:
for line_number, raw_line in enumerate(input_file, start=1):
line = raw_line.strip()
if not line or line.startswith("#"):
continue
try:
row = json.loads(line)
except json.JSONDecodeError as exc:
errors.append(f"{path}:{line_number}: {exc.msg}")
continue
if not isinstance(row, dict):
errors.append(f"{path}:{line_number}: expected a JSON object")
continue
# Repetition numbers restart at zero for every benchmark
# invocation. Keep the source input in the in-memory row so
# separate result files cannot overwrite or pair each other.
row[_INPUT_INDEX_FIELD] = input_index
rows.append(row)
if errors:
raise ValueError("Malformed JSONL row(s):\n" + "\n".join(errors))
return rows
def _group_key(row: dict[str, Any]) -> tuple[Any, ...]:
return tuple(row.get(field) for field in _GROUP_FIELDS)
def _numeric(row: dict[str, Any], metric: str) -> float | None:
value = row.get(metric)
if isinstance(value, bool) or not isinstance(value, (int, float)):
return None
return float(value)
def _group_sort_key(key: tuple[Any, ...]) -> tuple[tuple[int, Any], ...]:
return tuple((0, value) if isinstance(value, (int, float)) and not isinstance(value, bool) else (1, str(value)) for value in key)
def _summarize(rows: list[dict[str, Any]], *, metrics: list[str]) -> list[dict[str, Any]]:
groups: dict[tuple[Any, ...], list[dict[str, Any]]] = defaultdict(list)
for row in rows:
if row.get("profiled"):
continue
groups[_group_key(row)].append(row)
summaries: list[dict[str, Any]] = []
for key, group in sorted(groups.items(), key=lambda item: _group_sort_key(item[0])):
by_repetition: dict[Any, dict[str, dict[str, Any]]] = defaultdict(dict)
for row in group:
if row.get("success") and row.get("mode") in {"full", "delta"}:
repetition_key = (row.get(_INPUT_INDEX_FIELD), row.get("repetition"))
by_repetition[repetition_key][row["mode"]] = row
pairs = [pair for pair in by_repetition.values() if "full" in pair and "delta" in pair]
if not pairs:
continue
summary: dict[str, Any] = {field: key[index] for index, field in enumerate(_GROUP_FIELDS)}
summary["paired_repetitions"] = len(pairs)
summary["failed_rows"] = sum(1 for row in group if not row.get("success"))
for metric in metrics:
full_values: list[float] = []
delta_values: list[float] = []
for pair in pairs:
full_value = _numeric(pair["full"], metric)
delta_value = _numeric(pair["delta"], metric)
if full_value is None or delta_value is None:
continue
full_values.append(full_value)
delta_values.append(delta_value)
if not full_values:
continue
full_median = statistics.median(full_values)
delta_median = statistics.median(delta_values)
summary[f"full_{metric}"] = round(full_median, 6)
summary[f"delta_{metric}"] = round(delta_median, 6)
summary[f"ratio_{metric}"] = round(delta_median / full_median, 6) if full_median != 0 else None
summaries.append(summary)
return summaries
def _columns(rows: list[dict[str, Any]]) -> list[str]:
preferred = [*_GROUP_FIELDS, "paired_repetitions", "failed_rows"]
extras = sorted({key for row in rows for key in row if key not in preferred})
return [field for field in preferred if any(field in row for row in rows)] + extras
def _print_table(rows: list[dict[str, Any]]) -> None:
if not rows:
print("(no comparable full/delta pairs)")
return
columns = _columns(rows)
widths = {column: len(column) for column in columns}
for row in rows:
for column in columns:
widths[column] = max(widths[column], len(str(row.get(column, ""))))
print(" ".join(column.rjust(widths[column]) for column in columns))
for row in rows:
print(" ".join(str(row.get(column, "")).rjust(widths[column]) for column in columns))
def _print_csv(rows: list[dict[str, Any]]) -> None:
if not rows:
return
writer = csv.DictWriter(sys.stdout, fieldnames=_columns(rows), extrasaction="ignore")
writer.writeheader()
writer.writerows(rows)
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="Summarize paired full/delta checkpoint benchmark results")
parser.add_argument("inputs", nargs="+", type=Path)
parser.add_argument("--metrics", default=",".join(DEFAULT_METRICS), help="Comma-separated numeric result fields")
output = parser.add_mutually_exclusive_group()
output.add_argument("--json", action="store_true")
output.add_argument("--csv", action="store_true")
args = parser.parse_args(argv)
metrics = [metric.strip() for metric in args.metrics.split(",") if metric.strip()]
if not metrics:
parser.error("--metrics must contain at least one field")
try:
rows = _load_jsonl(args.inputs)
except (OSError, ValueError) as exc:
print(str(exc), file=sys.stderr)
return 1
profiled_rows = sum(1 for row in rows if row.get("profiled"))
if profiled_rows:
print(f"Skipping {profiled_rows} profiled row(s); profiled timings are excluded from baseline summaries.", file=sys.stderr)
summaries = _summarize(rows, metrics=metrics)
if args.json:
json.dump(summaries, sys.stdout, ensure_ascii=False, indent=2)
print()
elif args.csv:
_print_csv(summaries)
else:
_print_table(summaries)
return 0 if summaries else 1
if __name__ == "__main__":
raise SystemExit(main())

390
backend/tests/test_authorization_enforcement.py Normal file
View File

@ -0,0 +1,390 @@
"""Tests for Phase 1B tool authorization enforcement."""
from __future__ import annotations
import asyncio
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock
import pytest
from langchain_core.tools import StructuredTool
from deerflow.agents.lead_agent import agent as lead_agent_module
from deerflow.agents.middlewares.tool_error_handling_middleware import (
build_lead_runtime_middlewares,
build_subagent_runtime_middlewares,
)
from deerflow.authz.adapter import GuardrailAuthorizationAdapter
from deerflow.authz.enforcement import filter_tools_by_authorization
from deerflow.authz.provider import AuthzDecision, AuthzReason, Principal
from deerflow.authz.rbac import RbacAuthorizationProvider
from deerflow.config.app_config import AppConfig
from deerflow.config.authorization_config import AuthorizationConfig, AuthorizationProviderConfig
from deerflow.config.guardrails_config import GuardrailProviderConfig, GuardrailsConfig
from deerflow.config.model_config import ModelConfig
from deerflow.config.sandbox_config import SandboxConfig
from deerflow.guardrails.middleware import GuardrailMiddleware
from deerflow.tools.builtins.tool_search import assemble_deferred_tools
from deerflow.tools.mcp_metadata import tag_mcp_tool
def _tool(name: str) -> StructuredTool:
return StructuredTool.from_function(lambda: name, name=name, description=name)
class _FilterProvider:
name = "filter"
def __init__(self, allowed: list[str]) -> None:
self.allowed = allowed
self.calls: list[tuple[Principal, str, list[str]]] = []
def authorize(self, request):
return AuthzDecision(allow=True)
async def aauthorize(self, request):
return self.authorize(request)
def filter_resources(self, principal: Principal, resource_type: str, candidates: list[str]) -> list[str]:
self.calls.append((principal, resource_type, candidates))
return [candidate for candidate in candidates if candidate in self.allowed]
class _ExplodingFilterProvider(_FilterProvider):
def __init__(self) -> None:
super().__init__([])
def filter_resources(self, principal: Principal, resource_type: str, candidates: list[str]) -> list[str]:
raise RuntimeError("provider failed")
def _app_config(
*,
authorization: AuthorizationConfig,
guardrails: GuardrailsConfig | None = None,
models: list[ModelConfig] | None = None,
) -> AppConfig:
return AppConfig(
models=models or [],
sandbox=SandboxConfig(use="test"),
authorization=authorization,
guardrails=guardrails or GuardrailsConfig(),
)
class TestAuthorizationToolFilter:
def test_keeps_only_provider_allowed_tools_and_preserves_input_order(self):
provider = _FilterProvider(["web_search", "read_file"])
tools = [_tool("bash"), _tool("web_search"), _tool("read_file")]
filtered = filter_tools_by_authorization(
tools,
provider=provider,
principal=Principal(role="user"),
fail_closed=True,
)
assert [tool.name for tool in filtered] == ["web_search", "read_file"]
assert provider.calls == [(Principal(role="user"), "tool", ["bash", "web_search", "read_file"])]
def test_provider_error_fails_closed_to_an_empty_tool_set(self):
tools = [_tool("bash"), _tool("web_search")]
filtered = filter_tools_by_authorization(
tools,
provider=_ExplodingFilterProvider(),
principal=Principal(role="user"),
fail_closed=True,
)
assert filtered == []
def test_provider_error_fails_open_only_when_configured(self):
tools = [_tool("bash"), _tool("web_search")]
filtered = filter_tools_by_authorization(
tools,
provider=_ExplodingFilterProvider(),
principal=Principal(role="user"),
fail_closed=False,
)
assert filtered == tools
@pytest.mark.parametrize("invalid_result", ["bash", ("bash",), ["bash", 1]])
def test_invalid_provider_result_fails_closed(self, invalid_result):
class _InvalidResultProvider(_FilterProvider):
def filter_resources(self, principal, resource_type, candidates):
return invalid_result
filtered = filter_tools_by_authorization(
[_tool("bash")],
provider=_InvalidResultProvider([]),
principal=Principal(role="user"),
fail_closed=True,
)
assert filtered == []
def test_provider_cannot_add_tools_outside_the_candidate_set(self):
class _InjectingProvider(_FilterProvider):
def filter_resources(self, principal, resource_type, candidates):
return [*candidates, "injected_tool"]
filtered = filter_tools_by_authorization(
[_tool("bash")],
provider=_InjectingProvider([]),
principal=Principal(role="user"),
fail_closed=True,
)
assert [tool.name for tool in filtered] == ["bash"]
class TestAuthorizationGuardrailWiring:
def test_deferred_tool_search_bypasses_layer_two_for_filtered_catalog(self):
provider = RbacAuthorizationProvider(roles={"guest": {"tools": {"allow": ["mcp_allowed"]}}})
filtered_tools = filter_tools_by_authorization(
[tag_mcp_tool(_tool("mcp_allowed"))],
provider=provider,
principal=Principal(role="guest"),
fail_closed=True,
)
_final_tools, deferred_setup = assemble_deferred_tools(filtered_tools, enabled=True)
config = _app_config(
authorization=AuthorizationConfig(
enabled=True,
default_role="guest",
provider=AuthorizationProviderConfig(use="unused:Provider"),
)
)
middlewares = build_lead_runtime_middlewares(
app_config=config,
authorization_provider=provider,
deferred_setup=deferred_setup,
)
authorization_middleware = next(middleware for middleware in middlewares if isinstance(middleware, GuardrailMiddleware) and isinstance(middleware.provider, GuardrailAuthorizationAdapter))
request = MagicMock()
request.tool_call = {
"name": "tool_search",
"args": {"query": "mcp_allowed"},
"id": "call-search",
}
request.runtime = SimpleNamespace(context={"user_role": "guest"})
expected = MagicMock()
handler = MagicMock(return_value=expected)
result = authorization_middleware.wrap_tool_call(request, handler)
assert result is expected
handler.assert_called_once_with(request)
def test_tool_search_without_deferred_catalog_is_not_exempt(self):
provider = RbacAuthorizationProvider(roles={"guest": {"tools": {"allow": ["web_search"]}}})
config = _app_config(
authorization=AuthorizationConfig(
enabled=True,
default_role="guest",
provider=AuthorizationProviderConfig(use="unused:Provider"),
)
)
middlewares = build_lead_runtime_middlewares(
app_config=config,
authorization_provider=provider,
)
authorization_middleware = next(middleware for middleware in middlewares if isinstance(middleware, GuardrailMiddleware) and isinstance(middleware.provider, GuardrailAuthorizationAdapter))
request = MagicMock()
request.tool_call = {"name": "tool_search", "args": {}, "id": "call-search"}
request.runtime = SimpleNamespace(context={"user_role": "guest"})
handler = MagicMock()
result = authorization_middleware.wrap_tool_call(request, handler)
assert result.status == "error"
handler.assert_not_called()
def test_subagent_deferred_tool_search_bypasses_layer_two_async(self):
provider = RbacAuthorizationProvider(roles={"guest": {"tools": {"allow": ["mcp_allowed"]}}})
filtered_tools = filter_tools_by_authorization(
[tag_mcp_tool(_tool("mcp_allowed"))],
provider=provider,
principal=Principal(role="guest"),
fail_closed=True,
)
_final_tools, deferred_setup = assemble_deferred_tools(filtered_tools, enabled=True)
config = _app_config(
authorization=AuthorizationConfig(
enabled=True,
default_role="guest",
provider=AuthorizationProviderConfig(use="unused:Provider"),
)
)
middlewares = build_subagent_runtime_middlewares(
app_config=config,
authorization_provider=provider,
deferred_setup=deferred_setup,
)
authorization_middleware = next(middleware for middleware in middlewares if isinstance(middleware, GuardrailMiddleware) and isinstance(middleware.provider, GuardrailAuthorizationAdapter))
request = MagicMock()
request.tool_call = {
"name": "tool_search",
"args": {"query": "mcp_allowed"},
"id": "call-search",
}
request.runtime = SimpleNamespace(context={"user_role": "guest"})
expected = MagicMock()
handler = AsyncMock(return_value=expected)
result = asyncio.run(authorization_middleware.awrap_tool_call(request, handler))
assert result is expected
handler.assert_awaited_once_with(request)
def test_authorization_wires_adapter_with_the_build_provider_instance(self):
provider = _FilterProvider(["bash"])
config = _app_config(
authorization=AuthorizationConfig(
enabled=True,
provider=AuthorizationProviderConfig(use="deerflow.authz.rbac:RbacAuthorizationProvider", config={"roles": {"user": {}}}),
)
)
middlewares = build_lead_runtime_middlewares(app_config=config, authorization_provider=provider)
authorization_middleware = next(middleware for middleware in middlewares if isinstance(middleware, GuardrailMiddleware) and isinstance(middleware.provider, GuardrailAuthorizationAdapter))
assert authorization_middleware.provider._provider is provider
assert authorization_middleware.fail_closed is True
def test_authorization_and_explicit_guardrail_both_run(self):
provider = _FilterProvider(["bash"])
config = _app_config(
authorization=AuthorizationConfig(
enabled=True,
provider=AuthorizationProviderConfig(use="deerflow.authz.rbac:RbacAuthorizationProvider", config={"roles": {"user": {}}}),
),
guardrails=GuardrailsConfig(
enabled=True,
provider=GuardrailProviderConfig(
use="deerflow.guardrails.builtin:AllowlistProvider",
config={"allowed_tools": ["bash"]},
),
),
)
middlewares = build_lead_runtime_middlewares(app_config=config, authorization_provider=provider)
guardrails = [middleware for middleware in middlewares if isinstance(middleware, GuardrailMiddleware)]
assert len(guardrails) == 2
assert isinstance(guardrails[0].provider, GuardrailAuthorizationAdapter)
assert guardrails[0].provider._provider is provider
assert type(guardrails[1].provider).__name__ == "AllowlistProvider"
def test_wired_authorization_middleware_denies_execution_with_runtime_principal(self):
class _DenyingProvider(_FilterProvider):
def __init__(self):
super().__init__(["bash"])
self.requests = []
def authorize(self, request):
self.requests.append(request)
return AuthzDecision(
allow=False,
reasons=[AuthzReason(code="authz.denied", message="blocked")],
)
provider = _DenyingProvider()
config = _app_config(
authorization=AuthorizationConfig(
enabled=True,
provider=AuthorizationProviderConfig(use="unused:Provider"),
)
)
middlewares = build_lead_runtime_middlewares(
app_config=config,
authorization_provider=provider,
)
authorization_middleware = next(middleware for middleware in middlewares if isinstance(middleware, GuardrailMiddleware) and isinstance(middleware.provider, GuardrailAuthorizationAdapter))
request = MagicMock()
request.tool_call = {"name": "bash", "args": {"command": "whoami"}, "id": "call-1"}
request.runtime = SimpleNamespace(
context={
"user_id": "u1",
"user_role": "guest",
"thread_id": "t1",
"run_id": "r1",
}
)
handler = MagicMock()
result = authorization_middleware.wrap_tool_call(request, handler)
handler.assert_not_called()
assert result.status == "error"
assert "authz.denied" in result.content
assert provider.requests[0].principal == Principal(user_id="u1", role="guest")
assert provider.requests[0].target == "bash"
assert provider.requests[0].context["run_id"] == "r1"
@pytest.mark.parametrize("is_bootstrap", [False, True])
def test_lead_agent_filters_all_model_visible_tools_and_reuses_provider(monkeypatch, is_bootstrap):
"""Layer 1 covers late framework tools and Layer 2 receives its provider."""
config = _app_config(
authorization=AuthorizationConfig(
enabled=True,
provider=AuthorizationProviderConfig(
use="deerflow.authz.rbac:RbacAuthorizationProvider",
config={"roles": {"user": {"tools": {"allow": ["safe_tool"]}}}},
),
),
models=[
ModelConfig(
name="test-model",
display_name="Test model",
use="langchain_openai:ChatOpenAI",
model="test-model",
)
],
)
config.skills.deferred_discovery = True
monkeypatch.setattr(lead_agent_module, "_resolve_model_name", lambda *args, **kwargs: "test-model")
monkeypatch.setattr(lead_agent_module, "create_chat_model", lambda **kwargs: object())
monkeypatch.setattr(lead_agent_module, "create_agent", lambda **kwargs: kwargs)
monkeypatch.setattr(lead_agent_module, "apply_prompt_template", lambda **kwargs: "prompt")
monkeypatch.setattr(lead_agent_module, "build_tracing_callbacks", lambda: [])
monkeypatch.setattr(lead_agent_module, "_load_enabled_available_skills", lambda *args, **kwargs: [])
monkeypatch.setattr(
lead_agent_module,
"build_skill_search_setup",
lambda *args, **kwargs: SimpleNamespace(
describe_skill_tool=_tool("describe_skill"),
skill_names=frozenset({"example"}),
),
raising=False,
)
monkeypatch.setattr("deerflow.skills.describe.build_skill_search_setup", lead_agent_module.build_skill_search_setup)
monkeypatch.setattr("deerflow.tools.get_available_tools", lambda **kwargs: [_tool("safe_tool"), _tool("denied_tool")])
monkeypatch.setattr(lead_agent_module, "should_use_memory_tools", lambda memory_config: True)
monkeypatch.setattr(
lead_agent_module,
"_append_memory_tools_without_name_conflicts",
lambda tools: tools.append(_tool("memory_search")),
)
captured: dict[str, object] = {}
def _capture_middlewares(*args, **kwargs):
captured.update(kwargs)
return []
monkeypatch.setattr(lead_agent_module, "build_middlewares", _capture_middlewares)
runtime_context = {"user_role": "user"}
if is_bootstrap:
runtime_context["is_bootstrap"] = True
result = lead_agent_module._make_lead_agent({"context": runtime_context}, app_config=config)
assert [tool.name for tool in result["tools"]] == ["safe_tool"]
assert captured["authorization_provider"] is not None

15
backend/tests/test_authorization_runtime.py
View File

@ -40,6 +40,7 @@ class TestValidProvider:
def test_builtin_rbac_resolves(self):
config = AuthorizationConfig(
enabled=True,
default_role="admin",
provider=AuthorizationProviderConfig(
use="deerflow.authz.rbac:RbacAuthorizationProvider",
config={"roles": {"admin": {"tools": {"allow": "*"}}}},
@ -160,6 +161,19 @@ class TestRbacErrorPropagation:
else:
pytest.fail("Expected ValueError for invalid RBAC config")
def test_unknown_default_role_fails_during_provider_resolution(self):
config = AuthorizationConfig(
enabled=True,
default_role="missing",
provider=AuthorizationProviderConfig(
use="deerflow.authz.rbac:RbacAuthorizationProvider",
config={"roles": {"user": {"tools": {"allow": "*"}}}},
),
)
with pytest.raises(ValueError, match="default_role.*missing.*known roles"):
resolve_authorization_provider(config)
class TestNoFactoryInjection:
"""Factory must not inject fail_closed or default_role into provider kwargs."""
@ -187,6 +201,7 @@ class TestNoCaching:
def test_each_call_returns_new_instance(self):
config = AuthorizationConfig(
enabled=True,
default_role="admin",
provider=AuthorizationProviderConfig(
use="deerflow.authz.rbac:RbacAuthorizationProvider",
config={"roles": {"admin": {"tools": {"allow": "*"}}}},

245
backend/tests/test_authorization_tool_filter.py Normal file
View File

@ -0,0 +1,245 @@
"""Tests for Phase 1B tool authorization enforcement.
Covers Layer 1 (tool filtering before deferred assembly) and Layer 2
(GuardrailMiddleware via adapter) across the authorization enforcement
helpers, plus disabled-parity and fail-closed semantics.
"""
from __future__ import annotations
from langchain_core.tools import BaseTool, StructuredTool
from deerflow.authz.enforcement import filter_tools_by_authorization
from deerflow.authz.provider import Principal
from deerflow.authz.rbac import RbacAuthorizationProvider
from deerflow.authz.tool_filter import apply_tool_authorization
from deerflow.config.app_config import AppConfig
from deerflow.config.authorization_config import AuthorizationConfig, AuthorizationProviderConfig
from deerflow.config.sandbox_config import SandboxConfig
# --- Helpers ---
def _make_tool(name: str) -> BaseTool:
"""Create a minimal named tool for testing."""
return StructuredTool.from_function(lambda: None, name=name, description="test tool")
def _make_app_config(authz_config: AuthorizationConfig) -> AppConfig:
return AppConfig(
sandbox=SandboxConfig(use="test"),
authorization=authz_config,
)
def _rbac_provider() -> RbacAuthorizationProvider:
return RbacAuthorizationProvider(
roles={
"admin": {"tools": {"allow": "*"}},
"user": {"tools": {"allow": "*", "deny": ["update_agent", "bash"]}},
"guest": {"tools": {"allow": ["web_search", "read_file"]}},
}
)
# --- filter_tools_by_authorization ---
class TestFilterToolsByAuthorization:
def test_provider_none_returns_original(self):
"""When authorization is disabled (provider=None), tools pass through unchanged."""
tools = [_make_tool("bash"), _make_tool("web_search")]
result = filter_tools_by_authorization(
tools,
provider=None,
principal=Principal(role="user"),
fail_closed=True,
)
assert result == tools
def test_filters_denied_tools(self):
"""Denied tools are removed."""
provider = _rbac_provider()
tools = [_make_tool("bash"), _make_tool("web_search"), _make_tool("read_file")]
result = filter_tools_by_authorization(
tools,
provider=provider,
principal=Principal(role="user"),
fail_closed=True,
)
names = [t.name for t in result]
assert "bash" not in names
assert "web_search" in names
assert "read_file" in names
def test_preserves_order(self):
"""Filtering preserves the original tool order."""
provider = _rbac_provider()
tools = [_make_tool("web_search"), _make_tool("bash"), _make_tool("read_file")]
result = filter_tools_by_authorization(
tools,
provider=provider,
principal=Principal(role="user"),
fail_closed=True,
)
assert [t.name for t in result] == ["web_search", "read_file"]
def test_guest_narrow_allowlist(self):
"""Guest role only sees allowed tools."""
provider = _rbac_provider()
tools = [_make_tool("bash"), _make_tool("web_search"), _make_tool("read_file"), _make_tool("write_file")]
result = filter_tools_by_authorization(
tools,
provider=provider,
principal=Principal(role="guest"),
fail_closed=True,
)
assert [t.name for t in result] == ["web_search", "read_file"]
def test_fail_closed_on_provider_error(self):
"""When provider raises and fail_closed=True, deny all tools."""
class _ExplodingProvider:
name = "exploding"
def authorize(self, request):
raise RuntimeError("boom")
async def aauthorize(self, request):
raise RuntimeError("boom")
def filter_resources(self, principal, resource_type, candidates):
raise RuntimeError("boom")
tools = [_make_tool("bash"), _make_tool("web_search")]
result = filter_tools_by_authorization(
tools,
provider=_ExplodingProvider(),
principal=Principal(role="user"),
fail_closed=True,
)
assert result == []
def test_fail_open_on_provider_error(self):
"""When provider raises and fail_closed=False, keep original tools."""
class _ExplodingProvider:
name = "exploding"
def authorize(self, request):
raise RuntimeError("boom")
async def aauthorize(self, request):
raise RuntimeError("boom")
def filter_resources(self, principal, resource_type, candidates):
raise RuntimeError("boom")
tools = [_make_tool("bash"), _make_tool("web_search")]
result = filter_tools_by_authorization(
tools,
provider=_ExplodingProvider(),
principal=Principal(role="user"),
fail_closed=False,
)
assert result == tools
def test_unknown_role_raises_and_fail_closed_denies_all(self):
"""Unknown role causes filter_resources to raise ValueError;
fail_closed=True → deny all tools."""
provider = _rbac_provider()
tools = [_make_tool("bash"), _make_tool("web_search")]
result = filter_tools_by_authorization(
tools,
provider=provider,
principal=Principal(role="nonexistent"),
fail_closed=True,
)
assert result == []
# --- apply_tool_authorization (wrapper) ---
class TestApplyToolAuthorization:
def test_disabled_returns_original_and_none(self):
"""When authorization is disabled, tools unchanged and provider=None."""
app_config = _make_app_config(AuthorizationConfig(enabled=False))
tools = [_make_tool("bash")]
result, provider = apply_tool_authorization(
tools,
context={},
app_config=app_config,
)
assert result == tools
assert provider is None
def test_enabled_filters_and_returns_provider(self):
"""When enabled, tools are filtered and provider is returned for Layer 2 reuse."""
app_config = _make_app_config(
AuthorizationConfig(
enabled=True,
provider=AuthorizationProviderConfig(
use="deerflow.authz.rbac:RbacAuthorizationProvider",
config={"roles": {"user": {"tools": {"allow": "*", "deny": ["bash"]}}}},
),
)
)
tools = [_make_tool("bash"), _make_tool("web_search")]
result, provider = apply_tool_authorization(
tools,
context={"user_role": "user"},
app_config=app_config,
)
assert provider is not None
assert [t.name for t in result] == ["web_search"]
def test_disabled_does_not_resolve_provider(self):
"""Disabled mode must not attempt to resolve/import the provider."""
app_config = _make_app_config(
AuthorizationConfig(
enabled=False,
provider=AuthorizationProviderConfig(use="nonexistent.module:FakeProvider"),
)
)
tools = [_make_tool("bash")]
result, provider = apply_tool_authorization(
tools,
context={},
app_config=app_config,
)
assert result == tools
assert provider is None
def test_reuses_provided_provider(self):
"""When caller provides a provider, it's reused (not re-resolved)."""
rbac = _rbac_provider()
app_config = _make_app_config(AuthorizationConfig(enabled=True))
tools = [_make_tool("bash"), _make_tool("web_search")]
result, provider = apply_tool_authorization(
tools,
context={"user_role": "user"},
app_config=app_config,
authorization_provider=rbac,
)
assert provider is rbac
assert "bash" not in [t.name for t in result]
# --- Disabled parity ---
class TestDisabledParity:
"""When authorization.enabled=false, the tool set must be completely unchanged."""
def test_lead_agent_context_disabled_noop(self):
"""Simulate what the lead agent does: apply_tool_authorization with disabled config."""
app_config = _make_app_config(AuthorizationConfig(enabled=False))
tools = [_make_tool("bash"), _make_tool("web_search"), _make_tool("read_file")]
result, provider = apply_tool_authorization(
tools,
context={"user_role": "admin", "user_id": "u1", "is_internal": True},
app_config=app_config,
)
assert result == tools
assert provider is None

281
backend/tests/test_bench_checkpoint_channels.py Normal file
View File

@ -0,0 +1,281 @@
from __future__ import annotations
import importlib.util
import json
import sys
from pathlib import Path
import pytest
def _load_module():
path = Path(__file__).resolve().parents[1] / "scripts/benchmark/bench_checkpoint_channels.py"
spec = importlib.util.spec_from_file_location("bench_checkpoint_channels", path)
assert spec is not None
module = importlib.util.module_from_spec(spec)
assert spec.loader is not None
sys.modules[spec.name] = module
spec.loader.exec_module(module)
return module
bench = _load_module()
def test_parse_positive_int_csv_deduplicates_in_input_order(capsys: pytest.CaptureFixture[str]) -> None:
assert bench._parse_positive_int_csv("100,10,100,500", option="--updates") == [100, 10, 500]
assert "ignored duplicate value(s): 100" in capsys.readouterr().err
def test_parse_choice_csv_reports_duplicate_values(capsys: pytest.CaptureFixture[str]) -> None:
assert bench._parse_choice_csv("full,full,delta", option="--modes", choices=("full", "delta")) == ["full", "delta"]
assert "ignored duplicate value(s): full" in capsys.readouterr().err
@pytest.mark.parametrize("value", ["", "0", "-1", "one", "1,,2"])
def test_parse_positive_int_csv_rejects_invalid_values(value: str) -> None:
with pytest.raises(ValueError, match="--updates"):
bench._parse_positive_int_csv(value, option="--updates")
def test_deterministic_message_has_exact_payload_bytes_and_stable_identity() -> None:
first = bench._message_for_update(3, 128)
second = bench._message_for_update(3, 128)
assert first == second
assert first.id == "bench-message-00000003"
assert first.type == "ai"
assert len(first.content.encode("utf-8")) == 128
def test_expand_cases_alternates_modes_without_cross_product_reordering() -> None:
cases = bench._expand_cases(
modes=["full", "delta"],
backends=["sqlite"],
update_counts=[10, 100],
payload_bytes=[128],
repetitions=2,
seed=7,
)
assert [(case.repetition, case.update_count, case.mode) for case in cases] == [
(0, 10, "full"),
(0, 10, "delta"),
(0, 100, "delta"),
(0, 100, "full"),
(1, 10, "delta"),
(1, 10, "full"),
(1, 100, "full"),
(1, 100, "delta"),
]
def test_oversized_filter_skips_full_and_delta_as_a_comparable_pair() -> None:
cases = bench._expand_cases(
modes=["full", "delta"],
backends=["memory"],
update_counts=[10, 100],
payload_bytes=[128],
repetitions=1,
seed=1,
)
kept, skipped = bench._filter_oversized_pairs(cases, max_bytes=100_000)
assert {(case.update_count, case.mode) for case in kept} == {(10, "full"), (10, "delta")}
assert {(case.update_count, case.mode) for case in skipped} == {(100, "full"), (100, "delta")}
def test_oversized_filter_does_not_suppress_a_delta_only_diagnostic() -> None:
case = bench.BenchmarkCase(
mode="delta",
backend="memory",
update_count=2000,
payload_bytes=4096,
repetition=0,
seed=1,
)
kept, skipped = bench._filter_oversized_pairs([case], max_bytes=1)
assert kept == [case]
assert skipped == []
def test_help_explains_delta_only_runs_bypass_full_payload_cap() -> None:
assert "delta-only" in bench._build_parser().format_help()
def test_cross_mode_validation_rejects_materialized_state_mismatch() -> None:
rows = [
{
"success": True,
"mode": "full",
"backend": "sqlite",
"scenario": "append",
"snapshot_frequency": 1000,
"update_count": 10,
"payload_bytes": 128,
"repetition": 0,
"actual_message_count": 10,
"content_sha256": "full-digest",
},
{
"success": True,
"mode": "delta",
"backend": "sqlite",
"scenario": "append",
"snapshot_frequency": 1000,
"update_count": 10,
"payload_bytes": 128,
"repetition": 0,
"actual_message_count": 10,
"content_sha256": "delta-digest",
},
]
bench._validate_cross_mode_rows(rows)
assert all(row["success"] is False for row in rows)
assert all("cross-mode" in row["error"] for row in rows)
@pytest.mark.parametrize("mode", ["full", "delta"])
def test_memory_smoke_case_materializes_expected_state(mode: str, tmp_path: Path) -> None:
case = bench.BenchmarkCase(
mode=mode,
backend="memory",
update_count=4,
payload_bytes=64,
repetition=0,
seed=1,
)
row = bench._run_case(case, work_dir=tmp_path)
assert row["success"] is True
assert row["expected_message_count"] == 4
assert row["actual_message_count"] == 4
assert row["warm_read_ms"] >= 0
assert row["cold_read_ms"] >= 0
assert row["saver_reopen_ms"] == 0
assert len(row["content_sha256"]) == 64
assert row["db_bytes"] is None
assert row["logical_checkpoint_bytes"] is not None
def test_memory_case_keeps_timings_when_private_storage_stats_change(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
case = bench.BenchmarkCase(
mode="delta",
backend="memory",
update_count=2,
payload_bytes=32,
repetition=0,
seed=1,
)
def fail_storage_stats(*_args, **_kwargs):
raise AttributeError("private saver layout changed")
monkeypatch.setattr(bench, "_memory_storage_stats", fail_storage_stats)
row = bench._run_case(case, work_dir=tmp_path)
assert row["success"] is True
assert row["write_total_ms"] >= 0
assert row["logical_checkpoint_bytes"] is None
assert row["logical_write_bytes"] is None
assert row["checkpoint_rows"] is None
assert row["write_rows"] is None
assert "private saver layout changed" in row["storage_stats_error"]
@pytest.mark.parametrize("mode", ["full", "delta"])
def test_sqlite_smoke_case_reports_durable_and_logical_storage(mode: str, tmp_path: Path) -> None:
case = bench.BenchmarkCase(
mode=mode,
backend="sqlite",
update_count=3,
payload_bytes=64,
repetition=0,
seed=2,
)
row = bench._run_case(case, work_dir=tmp_path)
assert row["success"] is True
assert row["db_bytes"] > 0
assert row["durable_db_bytes"] > 0
assert row["logical_checkpoint_bytes"] > 0
assert row["logical_write_bytes"] > 0
assert row["checkpoint_rows"] > 0
assert row["write_rows"] > 0
def test_controller_writes_versioned_jsonl_without_sensitive_case_paths(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
output = tmp_path / "result.jsonl"
child_git_shas = []
def fake_run_child(case, *, timeout_seconds, git_sha, profile_dir=None):
child_git_shas.append(git_sha)
return {
"schema_version": 1,
"benchmark_version": 1,
"success": True,
"error": None,
"mode": case.mode,
"backend": case.backend,
"scenario": case.scenario,
"snapshot_frequency": case.snapshot_frequency,
"update_count": case.update_count,
"payload_bytes": case.payload_bytes,
"repetition": case.repetition,
"actual_message_count": case.update_count,
"content_sha256": "same",
}
monkeypatch.setattr(bench, "_run_child_case", fake_run_child)
monkeypatch.setattr(bench, "_resolve_git_sha", lambda: "controller-sha")
rc = bench.main(
[
"--modes",
"full,delta",
"--backends",
"memory",
"--updates",
"2",
"--payload-bytes",
"32",
"--repetitions",
"1",
"--output",
str(output),
]
)
assert rc == 0
rows = [json.loads(line) for line in output.read_text(encoding="utf-8").splitlines()]
assert [row["mode"] for row in rows] == ["full", "delta"]
assert all(row["schema_version"] == 1 for row in rows)
assert all("work_dir" not in row and "database_path" not in row for row in rows)
assert child_git_shas == ["controller-sha", "controller-sha"]
def test_profiled_case_writes_loadable_stats(tmp_path: Path) -> None:
case = bench.BenchmarkCase(
mode="delta",
backend="memory",
update_count=2,
payload_bytes=32,
repetition=0,
seed=3,
)
profile_path = tmp_path / bench._profile_filename(case)
row = bench._run_profiled_case(case, work_dir=tmp_path / "work", profile_path=profile_path)
assert row["success"] is True
assert row["profiled"] is True
assert profile_path.is_file()
assert profile_path.stat().st_size > 0

181
backend/tests/test_branch_history_seed.py Normal file
View File

@ -0,0 +1,181 @@
"""Unit tests for branch-history run-event seeding (#4380 problem 2).
``build_branch_history_seed_events`` must mirror RunJournal's message-event
contract exactly: same event types, ``category="message"``,
``content=message.model_dump()``, the same hidden-message rules, and the same
original-user-text restoration — so the thread feed cannot tell a seeded row
from a journaled one.
"""
import asyncio
from langchain_core.messages import AIMessage, HumanMessage, SystemMessage, ToolMessage
from deerflow.runtime.events.store.memory import MemoryRunEventStore
from deerflow.runtime.journal import build_branch_history_seed_events
def _seed(messages):
return build_branch_history_seed_events(
messages,
thread_id="branch-thread",
run_id="branch-seed-branch-thread",
parent_thread_id="parent-thread",
)
def test_seed_serializes_visible_history_in_order() -> None:
events = _seed(
[
HumanMessage(id="h1", content="question"),
AIMessage(id="a1", content="answer"),
ToolMessage(id="t1", content="tool output", tool_call_id="call-1"),
]
)
assert [event["event_type"] for event in events] == ["llm.human.input", "llm.ai.response", "llm.tool.result"]
assert all(event["category"] == "message" for event in events)
assert all(event["thread_id"] == "branch-thread" for event in events)
assert all(event["run_id"] == "branch-seed-branch-thread" for event in events)
assert [event["content"]["id"] for event in events] == ["h1", "a1", "t1"]
# Human/AI rows carry the journal's caller tag; every row carries the
# seed provenance marker.
assert events[0]["metadata"]["caller"] == "lead_agent"
assert events[1]["metadata"]["caller"] == "lead_agent"
assert "caller" not in events[2]["metadata"]
assert all(event["metadata"]["branch_seed"] is True for event in events)
assert all(event["metadata"]["branch_parent_thread_id"] == "parent-thread" for event in events)
def test_seed_skips_system_summary_and_nonmessage() -> None:
"""System prompts, summary-named/hidden human turns, and non-messages never
enter the thread feed — same as RunJournal's message path."""
events = _seed(
[
SystemMessage(id="s1", content="system prompt"),
HumanMessage(id="h-hidden", content="internal", additional_kwargs={"hide_from_ui": True}),
HumanMessage(id="h-summary", content="compacted", name="summary"),
HumanMessage(id="h1", content="question"),
AIMessage(id="a1", content="answer"),
"not-a-message",
]
)
assert [event["content"]["id"] for event in events] == ["h1", "a1"]
def test_seed_persists_hidden_ai_and_tool_rows_like_runjournal() -> None:
"""RunJournal's on_llm_end / _persist_tool_result_message write hide_from_ui
AI and tool rows unconditionally (the frontend hides them client-side), so
the seed must too — otherwise seeded rows diverge from journaled ones and a
hidden turn disappears from a forked feed."""
events = _seed(
[
HumanMessage(id="h1", content="question"),
AIMessage(id="a-hidden", content="internal", additional_kwargs={"hide_from_ui": True}),
AIMessage(id="a1", content="answer"),
ToolMessage(id="t-hidden", content="internal", tool_call_id="call-h", additional_kwargs={"hide_from_ui": True}),
]
)
assert [event["content"]["id"] for event in events] == ["h1", "a-hidden", "a1", "t-hidden"]
assert [event["event_type"] for event in events] == [
"llm.human.input",
"llm.ai.response",
"llm.ai.response",
"llm.tool.result",
]
# The hidden marker is preserved in content so the frontend still hides them.
assert events[1]["content"]["additional_kwargs"]["hide_from_ui"] is True
assert events[3]["content"]["additional_kwargs"]["hide_from_ui"] is True
def test_seed_deserializes_dict_shaped_checkpoint_messages() -> None:
"""Checkpoint messages can arrive as model_dump()-shaped dicts (the
branch-matching helpers in threads.py already handle both); the seed must
deserialize them instead of silently producing an empty batch."""
dict_messages = [
HumanMessage(id="h1", content="question").model_dump(),
AIMessage(
id="a1",
content="answer",
tool_calls=[{"name": "search", "args": {"q": "x"}, "id": "call-1", "type": "tool_call"}],
).model_dump(),
ToolMessage(id="t1", content="tool output", tool_call_id="call-1").model_dump(),
]
events = _seed(dict_messages)
assert [event["event_type"] for event in events] == ["llm.human.input", "llm.ai.response", "llm.tool.result"]
assert [event["content"]["id"] for event in events] == ["h1", "a1", "t1"]
# Faithful reconstruction: AI tool_calls and the tool's tool_call_id survive.
assert events[1]["content"]["tool_calls"][0]["id"] == "call-1"
assert events[2]["content"]["tool_call_id"] == "call-1"
def test_seed_drops_unparseable_dict_message() -> None:
"""A dict with no usable ``type`` can't be reconstructed; it is dropped
rather than crashing the seed (best-effort — the branch stays usable)."""
events = _seed([{"content": "orphan", "no_type": True}, HumanMessage(id="h1", content="q")])
assert [event["content"]["id"] for event in events] == ["h1"]
def test_seed_persists_allowlisted_hidden_human_input_response() -> None:
"""Mirrors RunJournal: answered clarification replies stay recoverable."""
response = {
"version": 1,
"kind": "human_input_response",
"source": "ask_clarification",
"request_id": "req-1",
"value": "yes",
"response_kind": "text",
}
events = _seed(
[
HumanMessage(
id="h-response",
content="yes",
additional_kwargs={"hide_from_ui": True, "human_input_response": response},
)
]
)
assert [event["content"]["id"] for event in events] == ["h-response"]
assert events[0]["event_type"] == "llm.human.input"
def test_seed_restores_original_user_content() -> None:
"""The model-facing wrapper must not leak into the seeded feed."""
message = HumanMessage(
id="h1",
content="<wrapped>question with injected context</wrapped>",
additional_kwargs={"original_user_content": "question"},
)
events = _seed([message])
assert events[0]["content"]["content"] == "question"
assert "original_user_content" not in events[0]["content"]["additional_kwargs"]
def test_seed_roundtrips_through_memory_store_feed() -> None:
"""put_batch preserves order and list_messages returns the seeded feed."""
store = MemoryRunEventStore()
events = _seed(
[
HumanMessage(id="h1", content="question"),
AIMessage(id="a1", content="answer"),
]
)
async def roundtrip():
await store.put_batch(events)
return await store.list_messages("branch-thread", user_id=None)
rows = asyncio.run(roundtrip())
assert [row["content"]["id"] for row in rows] == ["h1", "a1"]
seqs = [row["seq"] for row in rows]
assert seqs == sorted(seqs)
assert all(row["category"] == "message" for row in rows)

184
backend/tests/test_client.py
View File

@ -7,10 +7,12 @@ import tempfile
import zipfile
from enum import Enum
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import ANY, MagicMock, patch
import pytest
from langchain_core.messages import AIMessage, AIMessageChunk, HumanMessage, SystemMessage, ToolMessage # noqa: F401
from langchain_core.tools import StructuredTool
from app.gateway.routers.mcp import McpConfigResponse
from app.gateway.routers.memory import MemoryConfigResponse, MemoryStatusResponse
@ -21,9 +23,11 @@ from app.gateway.routers.uploads import UploadResponse
from deerflow.agents.middlewares.view_image_middleware import ViewImageMiddleware
from deerflow.agents.thread_state import DeltaThreadState, ThreadState
from deerflow.client import DeerFlowClient
from deerflow.config.authorization_config import AuthorizationConfig, AuthorizationProviderConfig
from deerflow.config.extensions_config import ExtensionsConfig, McpServerConfig
from deerflow.config.paths import Paths
from deerflow.skills.types import SkillCategory
from deerflow.tools.mcp_metadata import tag_mcp_tool
from deerflow.uploads.manager import PathTraversalError
# ---------------------------------------------------------------------------
@ -48,6 +52,7 @@ def mock_app_config():
config.skills.container_path = "/mnt/skills"
config.tool_search.enabled = False
config.database.checkpoint_channel_mode = "full"
config.authorization = AuthorizationConfig(enabled=False)
return config
@ -1023,6 +1028,135 @@ class TestExtractText:
class TestEnsureAgent:
def test_authorization_filters_framework_tools_and_reuses_provider(self, client, mock_app_config):
class Provider:
name = "test"
def filter_resources(self, principal, resource_type, candidates):
return [name for name in candidates if name == "safe_tool"]
def authorize(self, request):
raise AssertionError("not called while assembling")
async def aauthorize(self, request):
raise AssertionError("not called while assembling")
provider = Provider()
mock_app_config.authorization = AuthorizationConfig(
enabled=True,
provider=AuthorizationProviderConfig(use="unused:Provider"),
)
mock_app_config.skills.deferred_discovery = True
client._app_config = mock_app_config
safe_tool = StructuredTool.from_function(lambda: "safe", name="safe_tool", description="safe")
denied_tool = StructuredTool.from_function(lambda: "denied", name="denied_tool", description="denied")
describe_tool = StructuredTool.from_function(lambda: "describe", name="describe_skill", description="describe")
with (
patch("deerflow.client.create_chat_model"),
patch("deerflow.client.create_agent", return_value=MagicMock()) as mock_create_agent,
patch("deerflow.client.build_middlewares", return_value=[]) as mock_build_middlewares,
patch("deerflow.client.apply_prompt_template", return_value="prompt"),
patch("deerflow.client.get_enabled_skills_for_config", return_value=[MagicMock()]),
patch("deerflow.client.build_skill_search_setup", return_value=SimpleNamespace(describe_skill_tool=describe_tool, skill_names=frozenset({"example"}))),
patch.object(client, "_get_tools", return_value=[safe_tool, denied_tool]),
patch("deerflow.authz.tool_filter.resolve_authorization_provider", return_value=provider),
patch("deerflow.runtime.checkpointer.get_checkpointer", return_value=None),
):
client._ensure_agent(client._get_runnable_config("t1"), context={"user_role": "user"})
assert [tool.name for tool in mock_create_agent.call_args.kwargs["tools"]] == ["safe_tool"]
assert mock_build_middlewares.call_args.kwargs["authorization_provider"] is provider
def test_authorization_cache_key_uses_complete_principal(self, client, mock_app_config):
mock_app_config.authorization = AuthorizationConfig(
enabled=True,
provider=AuthorizationProviderConfig(
use="deerflow.authz.rbac:RbacAuthorizationProvider",
config={"roles": {"user": {"tools": {"allow": "*"}}}},
),
)
client._app_config = mock_app_config
with (
patch("deerflow.client.create_chat_model"),
patch("deerflow.client.create_agent", side_effect=[MagicMock(), MagicMock()]) as mock_create_agent,
patch("deerflow.client.build_middlewares", return_value=[]),
patch("deerflow.client.apply_prompt_template", return_value="prompt"),
patch("deerflow.client.get_enabled_skills_for_config", return_value=[]),
patch.object(client, "_get_tools", return_value=[]),
patch("deerflow.runtime.checkpointer.get_checkpointer", return_value=None),
):
config = client._get_runnable_config("t1")
client._ensure_agent(config, context={"user_id": "u1", "user_role": "user", "authz_attributes": {"department": "eng"}})
client._ensure_agent(config, context={"user_id": "u2", "user_role": "user", "authz_attributes": {"department": "eng"}})
assert mock_create_agent.call_count == 2
def test_authorization_cache_key_snapshots_nested_attributes(self, client, mock_app_config):
mock_app_config.authorization = AuthorizationConfig(
enabled=True,
provider=AuthorizationProviderConfig(
use="deerflow.authz.rbac:RbacAuthorizationProvider",
config={"roles": {"user": {"tools": {"allow": "*"}}}},
),
)
client._app_config = mock_app_config
attributes = {"groups": ["reader"]}
with (
patch("deerflow.client.create_chat_model"),
patch("deerflow.client.create_agent", side_effect=[MagicMock(), MagicMock()]) as mock_create_agent,
patch("deerflow.client.build_middlewares", return_value=[]),
patch("deerflow.client.apply_prompt_template", return_value="prompt"),
patch("deerflow.client.get_enabled_skills_for_config", return_value=[]),
patch.object(client, "_get_tools", return_value=[]),
patch("deerflow.runtime.checkpointer.get_checkpointer", return_value=None),
):
config = client._get_runnable_config("t1")
context = {
"user_role": "user",
"authz_attributes": attributes,
}
client._ensure_agent(config, context=context)
attributes["groups"].append("admin")
client._ensure_agent(config, context=context)
assert mock_create_agent.call_count == 2
def test_disabled_authorization_preserves_framework_tool_order(self, client, mock_app_config):
mock_app_config.authorization = AuthorizationConfig(enabled=False)
mock_app_config.tool_search.enabled = True
mock_app_config.skills.deferred_discovery = True
client._app_config = mock_app_config
mcp_tool = tag_mcp_tool(StructuredTool.from_function(lambda: "mcp", name="mcp_tool", description="mcp"))
describe_tool = StructuredTool.from_function(lambda: "describe", name="describe_skill", description="describe")
with (
patch("deerflow.client.create_chat_model"),
patch("deerflow.client.create_agent", return_value=MagicMock()) as mock_create_agent,
patch("deerflow.client.build_middlewares", return_value=[]),
patch("deerflow.client.apply_prompt_template", return_value="prompt"),
patch("deerflow.client.get_enabled_skills_for_config", return_value=[MagicMock()]),
patch(
"deerflow.client.build_skill_search_setup",
return_value=SimpleNamespace(
describe_skill_tool=describe_tool,
skill_names=frozenset({"example"}),
),
),
patch.object(client, "_get_tools", return_value=[mcp_tool]),
patch("deerflow.runtime.checkpointer.get_checkpointer", return_value=None),
):
client._ensure_agent(client._get_runnable_config("t1"))
assert [tool.name for tool in mock_create_agent.call_args.kwargs["tools"]] == [
"mcp_tool",
"tool_search",
"describe_skill",
]
def test_creates_agent(self, client):
"""_ensure_agent creates an agent on first call."""
mock_agent = MagicMock()
@ -1141,7 +1275,7 @@ class TestEnsureAgent:
"""_ensure_agent does not recreate if config key unchanged."""
mock_agent = MagicMock()
client._agent = mock_agent
client._agent_config_key = (None, True, False, False, None, None, None, None, "full")
client._agent_config_key = (None, True, False, False, None, None, None, None, "full", None)
config = client._get_runnable_config("t1")
client._ensure_agent(config)
@ -2538,7 +2672,7 @@ class TestScenarioAgentRecreation:
agents_created = []
def fake_ensure(config):
def fake_ensure(config, **kwargs):
key = tuple(config.get("configurable", {}).get(k) for k in ["model_name", "thinking_enabled", "is_plan_mode", "subagent_enabled"])
agents_created.append(key)
client._agent = agent
@ -2551,6 +2685,52 @@ class TestScenarioAgentRecreation:
assert len(agents_created) == 2
assert agents_created[0] != agents_created[1]
def test_stream_propagates_trusted_authorization_context(self, client):
ai = AIMessage(content="ok", id="ai-1")
agent = _make_agent_mock([{"messages": [ai]}])
captured: dict = {}
def fake_ensure(config, *, context):
captured.update(context)
client._agent = agent
with patch.object(client, "_ensure_agent", side_effect=fake_ensure):
list(
client.stream(
"hi",
thread_id="t1",
user_id="u1",
user_role="guest",
is_internal=True,
authz_attributes={"department": "eng"},
)
)
assert captured["user_id"] == "u1"
assert captured["user_role"] == "guest"
assert captured["is_internal"] is True
assert captured["authz_attributes"] == {"department": "eng"}
assert agent.stream.call_args.kwargs["context"]["user_role"] == "guest"
def test_stream_uses_effective_user_for_authorization_context(self, client, mock_app_config):
mock_app_config.authorization = AuthorizationConfig(
enabled=True,
provider=AuthorizationProviderConfig(use="unused:Provider"),
)
client._app_config = mock_app_config
agent = _make_agent_mock([{"messages": [AIMessage(content="ok", id="ai-1")]}])
captured: dict = {}
def fake_ensure(config, *, context):
captured.update(context)
client._agent = agent
with patch.object(client, "_ensure_agent", side_effect=fake_ensure):
list(client.stream("hi", thread_id="t1"))
assert captured["user_id"] == "test-user-autouse"
assert agent.stream.call_args.kwargs["context"]["user_id"] == "test-user-autouse"
class TestScenarioThreadIsolation:
"""Scenario: Operations on different threads don't interfere."""

5
backend/tests/test_client_langfuse_metadata.py
View File

@ -15,6 +15,7 @@ from typing import Any
import pytest
from deerflow.client import DeerFlowClient
from deerflow.config.authorization_config import AuthorizationConfig
from deerflow.trace_context import DEERFLOW_TRACE_METADATA_KEY, request_trace_context
@ -49,8 +50,9 @@ def _stub_agent_creation(monkeypatch, fake_agent: _FakeAgent) -> dict[str, Any]:
"""
captured: dict[str, Any] = {}
def _stub_ensure_agent(self, config):
def _stub_ensure_agent(self, config, **kwargs):
captured["config"] = config
captured["context"] = kwargs.get("context")
self._agent = fake_agent
self._agent_config_key = ("stub",)
@ -69,6 +71,7 @@ def _make_client(_monkeypatch, *, enhance_enabled: bool = True) -> DeerFlowClien
fake_app_config = SimpleNamespace(
models=[SimpleNamespace(name="stub-model")],
logging=SimpleNamespace(enhance=SimpleNamespace(enabled=enhance_enabled)),
authorization=AuthorizationConfig(enabled=False),
)
client = DeerFlowClient.__new__(DeerFlowClient)
client._app_config = fake_app_config

304
backend/tests/test_custom_events.py Normal file
View File

@ -0,0 +1,304 @@
from __future__ import annotations
import asyncio
import importlib
from enum import Enum
from types import SimpleNamespace
from typing import TypedDict
import pytest
from langchain.agents import create_agent
from langchain_core.language_models import BaseChatModel
from langchain_core.messages import AIMessage, HumanMessage
from langchain_core.outputs import ChatGeneration, ChatResult
from langgraph.config import get_stream_writer
from langgraph.errors import GraphInterrupt
from langgraph.graph import END, START, StateGraph
from langgraph.types import Interrupt
from deerflow.subagents.config import SubagentConfig
from deerflow.utils import custom_events as custom_events_module
from deerflow.utils.custom_events import aemit_custom_event, emit_custom_event
task_tool_module = importlib.import_module("deerflow.tools.builtins.task_tool")
class _State(TypedDict):
value: int
def _compile_graph(node):
builder = StateGraph(_State)
builder.add_node("emit", node)
builder.add_edge(START, "emit")
builder.add_edge("emit", END)
return builder.compile()
def _sync_node(state: _State) -> _State:
payload = {"type": "sync_probe", "value": state["value"]}
emit_custom_event(payload, writer=get_stream_writer())
return state
async def _async_node(state: _State) -> _State:
payload = {"type": "async_probe", "value": state["value"]}
await aemit_custom_event(payload, writer=get_stream_writer())
return state
async def _custom_events(graph) -> list[dict]:
return [chunk async for chunk in graph.astream({"value": 7}, stream_mode="custom")]
async def _astream_events(graph) -> list[dict]:
return [event async for event in graph.astream_events({"value": 7}, version="v2") if event["event"] == "on_custom_event"]
@pytest.mark.anyio
@pytest.mark.parametrize(
("node", "event_name"),
[
(_sync_node, "sync_probe"),
(_async_node, "async_probe"),
],
)
async def test_custom_event_is_emitted_once_to_each_streaming_api(node, event_name):
graph = _compile_graph(node)
custom_chunks = await _custom_events(graph)
callback_events = await _astream_events(graph)
expected = {"type": event_name, "value": 7}
assert custom_chunks == [expected]
assert len(callback_events) == 1
assert callback_events[0]["name"] == event_name
assert callback_events[0]["data"] == expected
class _TaskCallingModel(BaseChatModel):
call_count: int = 0
@property
def _llm_type(self) -> str:
return "fake-task-caller"
def bind_tools(self, tools, **kwargs):
return self
def _generate(self, messages, stop=None, run_manager=None, **kwargs):
self.call_count += 1
if self.call_count == 1:
message = AIMessage(
content="",
tool_calls=[
{
"id": "task-call-1",
"name": "task",
"args": {
"description": "validate streaming",
"prompt": "run the delegated task",
"subagent_type": "general-purpose",
},
}
],
response_metadata={"finish_reason": "tool_calls"},
)
else:
message = AIMessage(content="done", response_metadata={"finish_reason": "stop"})
return ChatResult(generations=[ChatGeneration(message=message)])
async def _agenerate(self, messages, stop=None, run_manager=None, **kwargs):
return self._generate(messages, stop=stop, run_manager=run_manager, **kwargs)
@pytest.mark.anyio
async def test_real_task_tool_events_reach_astream_events(monkeypatch):
"""Exercise the real ToolNode/runtime callback context used by task_tool."""
class _SubagentStatus(Enum):
COMPLETED = "completed"
config = SubagentConfig(
name="general-purpose",
description="General helper",
system_prompt="Test prompt",
model="test-model",
timeout_seconds=10,
)
completed = SimpleNamespace(
status=_SubagentStatus.COMPLETED,
ai_messages=[],
result="delegated result",
error=None,
stop_reason=None,
token_usage_records=[],
usage_reported=False,
)
class _Executor:
def __init__(self, **_kwargs):
pass
def execute_async(self, _prompt, task_id=None):
return task_id
monkeypatch.setattr(task_tool_module, "SubagentStatus", _SubagentStatus)
monkeypatch.setattr(task_tool_module, "SubagentExecutor", _Executor)
monkeypatch.setattr(task_tool_module, "get_available_subagent_names", lambda: ["general-purpose"])
monkeypatch.setattr(task_tool_module, "get_subagent_config", lambda _name: config)
monkeypatch.setattr(task_tool_module, "get_background_task_result", lambda _task_id: completed)
monkeypatch.setattr(task_tool_module, "cleanup_background_task", lambda _task_id: None)
monkeypatch.setattr(task_tool_module, "_token_usage_cache_enabled", lambda _config: False)
monkeypatch.setattr("deerflow.tools.get_available_tools", lambda **_kwargs: [])
agent = create_agent(
model=_TaskCallingModel(),
tools=[task_tool_module.task_tool],
context_schema=dict,
)
events = [
event
async for event in agent.astream_events(
{"messages": [HumanMessage(content="delegate this")]},
version="v2",
context={"thread_id": "task-stream-thread"},
)
if event["event"] == "on_custom_event"
]
assert [event["name"] for event in events] == ["task_started", "task_completed"]
assert [event["data"]["type"] for event in events] == ["task_started", "task_completed"]
assert all(event["data"]["task_id"] == "task-call-1" for event in events)
assert events[0]["data"]["description"] == "validate streaming"
assert events[1]["data"]["result"] == "delegated result"
def test_sync_dispatch_failure_does_not_break_writer(monkeypatch):
payload = {"type": "sync_probe", "value": 1}
written: list[dict] = []
def fail_dispatch(*_args, **_kwargs):
raise RuntimeError("callback failed")
monkeypatch.setattr(custom_events_module, "dispatch_custom_event", fail_dispatch)
emit_custom_event(payload, writer=written.append)
assert written == [payload]
def test_sync_dispatch_without_parent_run_does_not_break_writer():
payload = {"type": "sync_probe", "value": 1}
written: list[dict] = []
emit_custom_event(payload, writer=written.append)
assert written == [payload]
@pytest.mark.anyio
async def test_async_dispatch_failure_does_not_break_writer(monkeypatch):
payload = {"type": "async_probe", "value": 1}
written: list[dict] = []
async def fail_dispatch(*_args, **_kwargs):
raise RuntimeError("callback failed")
monkeypatch.setattr(custom_events_module, "adispatch_custom_event", fail_dispatch)
await aemit_custom_event(payload, writer=written.append)
assert written == [payload]
@pytest.mark.anyio
async def test_async_dispatch_without_parent_run_does_not_break_writer():
payload = {"type": "async_probe", "value": 1}
written: list[dict] = []
await aemit_custom_event(payload, writer=written.append)
assert written == [payload]
def test_missing_event_type_preserves_writer_and_skips_dispatch(monkeypatch):
payload = {"value": 1}
written: list[dict] = []
dispatched: list[tuple] = []
monkeypatch.setattr(custom_events_module, "dispatch_custom_event", lambda *args, **kwargs: dispatched.append((args, kwargs)))
emit_custom_event(payload, writer=written.append)
assert written == [payload]
assert dispatched == []
def test_writer_failure_propagates_before_dispatch(monkeypatch):
dispatched: list[tuple] = []
def fail_writer(_payload):
raise RuntimeError("writer failed")
monkeypatch.setattr(custom_events_module, "dispatch_custom_event", lambda *args, **kwargs: dispatched.append((args, kwargs)))
with pytest.raises(RuntimeError, match="writer failed"):
emit_custom_event({"type": "sync_probe"}, writer=fail_writer)
assert dispatched == []
@pytest.mark.anyio
async def test_async_writer_failure_propagates_before_dispatch(monkeypatch):
dispatched: list[tuple] = []
def fail_writer(_payload):
raise RuntimeError("writer failed")
async def record_dispatch(*args, **kwargs):
dispatched.append((args, kwargs))
monkeypatch.setattr(custom_events_module, "adispatch_custom_event", record_dispatch)
with pytest.raises(RuntimeError, match="writer failed"):
await aemit_custom_event({"type": "async_probe"}, writer=fail_writer)
assert dispatched == []
@pytest.mark.anyio
async def test_async_cancellation_is_not_swallowed(monkeypatch):
async def cancel_dispatch(*_args, **_kwargs):
raise asyncio.CancelledError
monkeypatch.setattr(custom_events_module, "adispatch_custom_event", cancel_dispatch)
with pytest.raises(asyncio.CancelledError):
await aemit_custom_event({"type": "async_probe"}, writer=lambda _payload: None)
@pytest.mark.parametrize("async_dispatch", [False, True])
def test_langgraph_control_flow_is_not_swallowed(monkeypatch, async_dispatch):
control_flow = GraphInterrupt((Interrupt(value="pause"),))
if async_dispatch:
async def interrupt_dispatch(*_args, **_kwargs):
raise control_flow
monkeypatch.setattr(custom_events_module, "adispatch_custom_event", interrupt_dispatch)
with pytest.raises(GraphInterrupt) as raised:
asyncio.run(aemit_custom_event({"type": "async_probe"}, writer=lambda _payload: None))
else:
def interrupt_dispatch(*_args, **_kwargs):
raise control_flow
monkeypatch.setattr(custom_events_module, "dispatch_custom_event", interrupt_dispatch)
with pytest.raises(GraphInterrupt) as raised:
emit_custom_event({"type": "sync_probe"}, writer=lambda _payload: None)
assert raised.value is control_flow

25
backend/tests/test_github_webhooks.py
View File

@ -547,16 +547,25 @@ def test_dispatch_failure_returns_503_not_200(client: TestClient, monkeypatch: p
The earlier behaviour swallowed every fan-out exception into a 200 OK
response (``dispatch={"error": "fanout failed"}``). GitHub treats 200
as final success and does not automatically retry any failure,
as final success and never automatically retries any failure,
including 5xx (see
https://docs.github.com/en/webhooks/using-webhooks/handling-failed-webhook-deliveries)
— so a 200 ack permanently drops the delivery. The route now lets
runtime failures propagate as 503 so the delivery is correctly
recorded as failed, recoverable via a manual "Redeliver", the REST
API, or an operator's own recovery script. The startup-time
``is_route_enabled`` check still handles *configuration* failures
fail-closed (route absent → 404); 503 is reserved for runtime
failures worth making recoverable this way.
— so a mistaken 200 ack hides the failure from GitHub's Recent
Deliveries status and from any recovery script filtering on non-OK
deliveries (the pattern GitHub's own docs recommend). That is a
discoverability gap, not literal unrecoverability: GitHub's manual
"Redeliver" button and its REST redelivery endpoint place no
failed-status precondition on the delivery id (see
https://docs.github.com/en/webhooks/testing-and-troubleshooting-webhooks/redelivering-webhooks),
so a 200'd delivery can still be redelivered by an operator who
independently finds it — they just get no signal to. The route now
lets runtime failures propagate as 503 instead, so the delivery is
correctly recorded as failed and actually surfaces for a manual
"Redeliver" click, the REST API, or an operator's own recovery
script to find and recover. The startup-time ``is_route_enabled``
check still handles *configuration* failures fail-closed (route
absent → 404); 503 is reserved for runtime failures worth making
discoverable and recoverable this way.
"""
async def fake_fanout(*args, **kwargs) -> dict:

6
backend/tests/test_guardrail_middleware.py
View File

@ -431,17 +431,19 @@ class TestGuardrailMiddleware:
assert journal.calls == []
# Journal: a recording failure must not alter the guardrail denial outcome.
def test_guardrail_event_recording_failure_does_not_change_denial(self):
def test_guardrail_event_recording_failure_warns_without_changing_denial(self, caplog):
journal = _FakeJournal(fail=True)
mw = GuardrailMiddleware(_DenyAllProvider())
req = _make_tool_call_request("bash", context={"__run_journal": journal})
handler = MagicMock()
result = mw.wrap_tool_call(req, handler)
with caplog.at_level("WARNING"):
result = mw.wrap_tool_call(req, handler)
handler.assert_not_called()
assert result.status == "error"
assert "oap.denied" in result.content
assert "Failed to record middleware:guardrail event" in caplog.text
# Journal: the async denial path records the same guardrail audit event.
def test_async_denied_tool_records_guardrail_event(self):

7
backend/tests/test_lead_agent_model_resolution.py
View File

@ -565,9 +565,12 @@ def test_build_middlewares_passes_explicit_app_config_to_shared_factory(monkeypa
def _raise_get_app_config():
raise AssertionError("ambient get_app_config() must not be used when app_config is explicit")
def _fake_build_lead_runtime_middlewares(*, app_config, lazy_init):
provider = object()
def _fake_build_lead_runtime_middlewares(*, app_config, lazy_init, authorization_provider):
captured["app_config"] = app_config
captured["lazy_init"] = lazy_init
captured["authorization_provider"] = authorization_provider
return ["base-middleware"]
monkeypatch.setattr(lead_agent_module, "get_app_config", _raise_get_app_config)
@ -593,11 +596,13 @@ def test_build_middlewares_passes_explicit_app_config_to_shared_factory(monkeypa
{"configurable": {"is_plan_mode": False, "subagent_enabled": False}},
model_name="safe-model",
app_config=app_config,
authorization_provider=provider,
)
assert captured == {
"app_config": app_config,
"lazy_init": True,
"authorization_provider": provider,
"title_app_config": app_config,
"memory_config": app_config.memory,
}

56
backend/tests/test_llm_error_handling_middleware.py
View File

@ -95,6 +95,7 @@ def test_async_model_call_retries_busy_provider_then_succeeds(
attempts = 0
waits: list[float] = []
events: list[dict] = []
dispatched_events: list[dict] = []
async def fake_sleep(delay: float) -> None:
waits.append(delay)
@ -102,6 +103,10 @@ def test_async_model_call_retries_busy_provider_then_succeeds(
def fake_writer():
return events.append
async def fake_emit_custom_event(payload, *, writer):
writer(payload)
dispatched_events.append(payload)
async def handler(_request) -> AIMessage:
nonlocal attempts
attempts += 1
@ -114,6 +119,10 @@ def test_async_model_call_retries_busy_provider_then_succeeds(
"langgraph.config.get_stream_writer",
fake_writer,
)
monkeypatch.setattr(
"deerflow.agents.middlewares.llm_error_handling_middleware.aemit_custom_event",
fake_emit_custom_event,
)
result = asyncio.run(middleware.awrap_model_call(SimpleNamespace(), handler))
@ -122,6 +131,7 @@ def test_async_model_call_retries_busy_provider_then_succeeds(
assert attempts == 3
assert waits == [0.025, 0.025]
assert [event["type"] for event in events] == ["llm_retry", "llm_retry"]
assert dispatched_events == events
def test_async_model_call_returns_user_message_for_quota_errors() -> None:
@ -168,11 +178,17 @@ def test_async_model_call_marks_transient_retry_exhaustion_as_error_fallback(
def test_sync_model_call_uses_retry_after_header(monkeypatch: pytest.MonkeyPatch) -> None:
middleware = _build_middleware(retry_max_attempts=2, retry_base_delay_ms=10, retry_cap_delay_ms=10)
waits: list[float] = []
events: list[dict] = []
dispatched_events: list[dict] = []
attempts = 0
def fake_sleep(delay: float) -> None:
waits.append(delay)
def fake_emit_custom_event(payload, *, writer):
writer(payload)
dispatched_events.append(payload)
def handler(_request) -> AIMessage:
nonlocal attempts
attempts += 1
@ -185,12 +201,52 @@ def test_sync_model_call_uses_retry_after_header(monkeypatch: pytest.MonkeyPatch
return AIMessage(content="ok")
monkeypatch.setattr("time.sleep", fake_sleep)
monkeypatch.setattr("langgraph.config.get_stream_writer", lambda: events.append)
monkeypatch.setattr(
"deerflow.agents.middlewares.llm_error_handling_middleware.emit_custom_event",
fake_emit_custom_event,
)
result = middleware.wrap_model_call(SimpleNamespace(), handler)
assert isinstance(result, AIMessage)
assert result.content == "ok"
assert waits == [2.0]
assert dispatched_events == events
assert [event["type"] for event in events] == ["llm_retry"]
def test_sync_retry_event_preserves_langgraph_control_flow(monkeypatch: pytest.MonkeyPatch) -> None:
middleware = _build_middleware()
def interrupt_dispatch(*_args, **_kwargs):
raise GraphBubbleUp
monkeypatch.setattr("langgraph.config.get_stream_writer", lambda: lambda _payload: None)
monkeypatch.setattr(
"deerflow.agents.middlewares.llm_error_handling_middleware.emit_custom_event",
interrupt_dispatch,
)
with pytest.raises(GraphBubbleUp):
middleware._emit_retry_event(1, 10, "busy", max_attempts=2)
@pytest.mark.anyio
async def test_async_retry_event_preserves_langgraph_control_flow(monkeypatch: pytest.MonkeyPatch) -> None:
middleware = _build_middleware()
async def interrupt_dispatch(*_args, **_kwargs):
raise GraphBubbleUp
monkeypatch.setattr("langgraph.config.get_stream_writer", lambda: lambda _payload: None)
monkeypatch.setattr(
"deerflow.agents.middlewares.llm_error_handling_middleware.aemit_custom_event",
interrupt_dispatch,
)
with pytest.raises(GraphBubbleUp):
await middleware._aemit_retry_event(1, 10, "busy", max_attempts=2)
def test_sync_model_call_propagates_graph_bubble_up() -> None:

4
backend/tests/test_memory_tools.py
View File

@ -494,6 +494,7 @@ class TestModeGating:
def test_lead_agent_deduplicates_memory_tools_after_appending(self, monkeypatch):
"""Configured tools should not duplicate tool-mode memory tools."""
from deerflow.agents.lead_agent import agent as lead_agent_module
from deerflow.config.authorization_config import AuthorizationConfig
from deerflow.config.memory_config import MemoryConfig
monkeypatch.setattr(lead_agent_module, "_resolve_model_name", lambda x=None, **kwargs: "default-model")
@ -516,6 +517,7 @@ class TestModeGating:
skills=SimpleNamespace(deferred_discovery=False, container_path="/tmp/skills"),
tool_search=SimpleNamespace(enabled=False, auto_promote_top_k=0),
database=SimpleNamespace(checkpoint_channel_mode="full"),
authorization=AuthorizationConfig(enabled=False),
)
agent_kwargs = lead_agent_module._make_lead_agent({"configurable": {"agent_name": "test-agent"}}, app_config=app_config)
@ -527,6 +529,7 @@ class TestModeGating:
def test_lead_agent_preserves_non_memory_duplicate_tool_names(self, monkeypatch):
"""Memory-tool collision handling should not drop unrelated duplicate tools."""
from deerflow.agents.lead_agent import agent as lead_agent_module
from deerflow.config.authorization_config import AuthorizationConfig
from deerflow.config.memory_config import MemoryConfig
monkeypatch.setattr(lead_agent_module, "_resolve_model_name", lambda x=None, **kwargs: "default-model")
@ -549,6 +552,7 @@ class TestModeGating:
skills=SimpleNamespace(deferred_discovery=False, container_path="/tmp/skills"),
tool_search=SimpleNamespace(enabled=False, auto_promote_top_k=0),
database=SimpleNamespace(checkpoint_channel_mode="full"),
authorization=AuthorizationConfig(enabled=False),
)
agent_kwargs = lead_agent_module._make_lead_agent({"configurable": {"agent_name": "test-agent"}}, app_config=app_config)

43
backend/tests/test_multi_worker_run_ownership.py
View File

@ -260,6 +260,49 @@ async def test_reconciliation_skips_active_lease_runs():
assert stored["status"] == "running"
@pytest.mark.anyio
async def test_reconciliation_skips_candidate_when_owner_renews_lease_after_scan():
"""A renewed lease between scan and claim must keep the run active."""
store = MemoryRunStore()
grace = 10
expired_lease = (datetime.now(UTC) - timedelta(seconds=grace + 5)).isoformat()
await store.put(
"race-run",
thread_id="thread-1",
status="running",
owner_worker_id="worker-alive",
lease_expires_at=expired_lease,
created_at=(datetime.now(UTC) - timedelta(seconds=120)).isoformat(),
)
original_list = store.list_inflight_with_expired_lease
async def list_then_owner_renews(*, before=None, grace_seconds=10):
rows = [dict(row) for row in await original_list(before=before, grace_seconds=grace_seconds)]
renewed_lease = (datetime.now(UTC) + timedelta(seconds=60)).isoformat()
updated = await store.update_lease(
"race-run",
owner_worker_id="worker-alive",
lease_expires_at=renewed_lease,
)
assert updated is True
return rows
store.list_inflight_with_expired_lease = list_then_owner_renews
manager = _make_manager(
store=store,
run_ownership_config=_lease_config(heartbeat_enabled=True, grace_seconds=grace),
)
recovered = await manager.reconcile_orphaned_inflight_runs(
error="Gateway restarted before this run reached a durable final state.",
)
assert recovered == []
stored = await store.get("race-run")
assert stored["status"] == "running"
assert datetime.fromisoformat(stored["lease_expires_at"]) > datetime.now(UTC)
@pytest.mark.anyio
async def test_reconciliation_claims_null_lease_runs():
"""Pre-ownership rows (NULL lease) must be reclaimed."""

549
backend/tests/test_run_event_stream_contract.py Normal file
View File

@ -0,0 +1,549 @@
"""Conformance tests for the documented run event stream contract."""
from __future__ import annotations
import ast
import json
from pathlib import Path
from uuid import uuid4
import pytest
from jsonschema import Draft202012Validator, FormatChecker
from langchain_core.messages import AIMessage, HumanMessage, ToolMessage
from langchain_core.outputs import ChatGeneration, LLMResult
from deerflow.runtime.events.catalog import (
FIXED_RUN_EVENT_DEFINITIONS,
JOURNAL_RUN_EVENT_DEFINITIONS,
MIDDLEWARE_EVENT_PATTERN,
MIDDLEWARE_EVENT_TAG_MAX_LENGTH,
MIDDLEWARE_EVENT_TAGS,
RUN_EVENT_CATEGORY_MAX_LENGTH,
RUN_EVENT_TYPE_MAX_LENGTH,
SUBAGENT_RUN_EVENT_DEFINITIONS,
WORKSPACE_RUN_EVENT_DEFINITIONS,
RunEventDefinition,
RunEventPattern,
)
from deerflow.runtime.events.store.memory import MemoryRunEventStore
from deerflow.runtime.journal import RunJournal
from deerflow.subagents.step_events import SUBAGENT_STEP_MAX_CHARS, capture_step_message, subagent_run_event
REPO_ROOT = Path(__file__).resolve().parents[2]
CONTRACT_PATH = REPO_ROOT / "contracts" / "run_event_stream_contract.json"
def _load_contract() -> dict:
return json.loads(CONTRACT_PATH.read_text(encoding="utf-8"))
def _contract_events() -> dict[str, dict]:
return {event["event_type"]: event for event in _load_contract()["events"]}
def _assert_schema_valid(schema: dict | bool, instance: object) -> None:
Draft202012Validator.check_schema(schema)
validator = Draft202012Validator(schema, format_checker=FormatChecker())
errors = sorted(validator.iter_errors(instance), key=lambda error: list(error.path))
assert not errors, "; ".join(error.message for error in errors)
def _assert_fixed_event_valid(event: dict, *, persisted: bool = False) -> None:
contract_event = _contract_events()[event["event_type"]]
assert event["category"] == contract_event["category"]
_assert_schema_valid(contract_event["content_schema"], event["content"])
_assert_schema_valid(contract_event["metadata_schema"], event.get("metadata", {}))
if persisted:
_assert_schema_valid(_load_contract()["record_schema"], event)
def _make_llm_response(content: str = "answer", usage: dict | None = None) -> LLMResult:
message = AIMessage(
content=content,
id=f"msg-{uuid4()}",
response_metadata={"model_name": "test-model"},
usage_metadata=usage,
)
return LLMResult(generations=[[ChatGeneration(message=message)]])
def _subagent_batch() -> list[dict]:
chunks = [
{"type": "task_started", "task_id": "call-batch", "description": "research"},
{
"type": "task_running",
"task_id": "call-batch",
"message": {
"type": "ai",
"content": "searching",
"tool_calls": [{"name": "web_search", "args": {"query": "deerflow"}}],
},
"message_index": 1,
},
{"type": "task_completed", "task_id": "call-batch", "result": "done"},
]
events = [subagent_run_event(chunk) for chunk in chunks]
assert all(event is not None for event in events)
return [{"thread_id": "thread-batch", "run_id": "run-batch", **event} for event in events if event is not None]
async def _persist_subagent_batch(store) -> list[dict]:
await store.put_batch(_subagent_batch())
return await store.list_events("thread-batch", "run-batch")
async def _record_run_end(store) -> dict:
journal = RunJournal("run-output", "thread-output", store, flush_threshold=100)
journal.on_chain_end(
{"messages": [AIMessage(content="final answer", id="final-message")]},
run_id=uuid4(),
parent_run_id=None,
)
await journal.flush()
events = await store.list_events("thread-output", "run-output", event_types=["run.end"])
assert len(events) == 1
return events[0]
def test_contract_and_runtime_catalog_have_the_same_fixed_events():
contract = _load_contract()
contract_types = [event["event_type"] for event in contract["events"]]
runtime_types = [definition.event_type for definition in FIXED_RUN_EVENT_DEFINITIONS]
contract_pairs = {(event["event_type"], event["category"]) for event in contract["events"]}
runtime_pairs = {(definition.event_type, definition.category) for definition in FIXED_RUN_EVENT_DEFINITIONS}
assert len(set(contract_types)) == len(contract_types)
assert len(set(runtime_types)) == len(runtime_types)
assert contract_pairs == runtime_pairs
assert set(contract["categories"]) == {definition.category for definition in FIXED_RUN_EVENT_DEFINITIONS} | {MIDDLEWARE_EVENT_PATTERN.category}
event_type_schema = contract["record_schema"]["properties"]["event_type"]
category_schema = contract["record_schema"]["properties"]["category"]
middleware_pattern = contract["dynamic_event_patterns"][0]
assert event_type_schema["maxLength"] == RUN_EVENT_TYPE_MAX_LENGTH
assert category_schema["maxLength"] == RUN_EVENT_CATEGORY_MAX_LENGTH
assert middleware_pattern["event_type_schema"]["maxLength"] == RUN_EVENT_TYPE_MAX_LENGTH
assert middleware_pattern["tag_schema"]["maxLength"] == MIDDLEWARE_EVENT_TAG_MAX_LENGTH
from deerflow.persistence.models.run_event import RunEventRow
assert RunEventRow.__table__.c.event_type.type.length == RUN_EVENT_TYPE_MAX_LENGTH
assert RunEventRow.__table__.c.category.type.length == RUN_EVENT_CATEGORY_MAX_LENGTH
@pytest.mark.parametrize(
("definition_type", "kwargs"),
[
(RunEventDefinition, {"event_type": "test.event"}),
(RunEventPattern, {"pattern": "test:{tag}", "prefix": "test:"}),
],
)
def test_runtime_catalog_rejects_categories_that_do_not_fit_persistence(definition_type, kwargs):
assert definition_type(category="x" * RUN_EVENT_CATEGORY_MAX_LENGTH, **kwargs).category
for invalid_category in ("", "x" * (RUN_EVENT_CATEGORY_MAX_LENGTH + 1)):
with pytest.raises(ValueError, match="category"):
definition_type(category=invalid_category, **kwargs)
@pytest.mark.parametrize(
"relative_path",
[
"persistence/models/run_event.py",
"workspace_changes/types.py",
],
)
def test_lower_level_run_event_modules_do_not_import_runtime(relative_path):
module_path = REPO_ROOT / "backend" / "packages" / "harness" / "deerflow" / relative_path
tree = ast.parse(module_path.read_text(encoding="utf-8"), filename=str(module_path))
imports = [node.module for node in ast.walk(tree) if isinstance(node, ast.ImportFrom) and node.module is not None]
imports.extend(alias.name for node in ast.walk(tree) if isinstance(node, ast.Import) for alias in node.names)
assert not [module for module in imports if module == "deerflow.runtime" or module.startswith("deerflow.runtime.")]
def test_legacy_aliases_are_read_only_and_outside_the_current_catalog():
contract = _load_contract()
aliases = {alias["event_type"]: alias for alias in contract["legacy_event_aliases"]}
current_types = {definition.event_type for definition in FIXED_RUN_EVENT_DEFINITIONS}
assert set(aliases) == {"ai_message"}
assert aliases["ai_message"]["canonical_event_type"] == "llm.ai.response"
assert aliases["ai_message"]["produced_by_current_runtime"] is False
assert "/messages/page" in aliases["ai_message"]["compatibility_scope"]
assert "legacy /messages endpoint" in aliases["ai_message"]["known_limitations"]
assert set(aliases).isdisjoint(current_types)
def test_record_envelope_accepts_every_json_content_type():
schema = _load_contract()["record_schema"]
envelope = {
"thread_id": "thread-1",
"run_id": "run-1",
"seq": 1,
"event_type": "run.end",
"category": "outputs",
"metadata": {},
"created_at": "2026-07-21T00:00:00+00:00",
}
for content in ("text", {"key": "value"}, ["value"], 1, 1.5, True, None):
_assert_schema_valid(schema, {**envelope, "content": content})
def test_contract_schemas_are_valid_json_schema():
contract = _load_contract()
Draft202012Validator.check_schema(contract["record_schema"])
for event in contract["events"]:
Draft202012Validator.check_schema(event["content_schema"])
Draft202012Validator.check_schema(event["metadata_schema"])
for pattern in contract["dynamic_event_patterns"]:
Draft202012Validator.check_schema(pattern["event_type_schema"])
Draft202012Validator.check_schema(pattern["tag_schema"])
Draft202012Validator.check_schema(pattern["content_schema"])
Draft202012Validator.check_schema(pattern["metadata_schema"])
@pytest.mark.anyio
@pytest.mark.parametrize("backend", ["memory", "jsonl"])
async def test_non_database_stores_return_contract_records(backend, tmp_path):
if backend == "memory":
store = MemoryRunEventStore()
else:
from deerflow.runtime.events.store.jsonl import JsonlRunEventStore
store = JsonlRunEventStore(base_dir=tmp_path / "events")
record = await store.put(
thread_id="thread-1",
run_id="run-1",
event_type="context:memory",
category="context",
content={"content_sha256": "a" * 64},
metadata={},
)
_assert_fixed_event_valid(record, persisted=True)
assert record["seq"] == 1
@pytest.mark.anyio
async def test_database_store_returns_contract_record_with_backend_fields(tmp_path):
from deerflow.persistence.engine import close_engine, get_session_factory, init_engine
from deerflow.runtime.events.store.db import DbRunEventStore
url = f"sqlite+aiosqlite:///{tmp_path / 'events.db'}"
await init_engine("sqlite", url=url, sqlite_dir=str(tmp_path))
try:
store = DbRunEventStore(get_session_factory())
record = await store.put(
thread_id="thread-1",
run_id="run-1",
event_type="context:memory",
category="context",
content={"content_sha256": "a" * 64},
metadata={},
)
_assert_fixed_event_valid(record, persisted=True)
assert "user_id" in record
assert record["metadata"]["content_is_json"] is True
finally:
await close_engine()
@pytest.mark.anyio
async def test_run_end_backend_storage_semantics_match_contract(tmp_path):
from deerflow.persistence.engine import close_engine, get_session_factory, init_engine
from deerflow.runtime.events.store.db import DbRunEventStore
from deerflow.runtime.events.store.jsonl import JsonlRunEventStore
contract_event = _contract_events()["run.end"]
assert set(contract_event["storage_semantics"]) == {"memory", "jsonl", "database"}
memory_event = await _record_run_end(MemoryRunEventStore())
jsonl_event = await _record_run_end(JsonlRunEventStore(base_dir=tmp_path / "events"))
url = f"sqlite+aiosqlite:///{tmp_path / 'run-output.db'}"
await init_engine("sqlite", url=url, sqlite_dir=str(tmp_path))
try:
database_event = await _record_run_end(DbRunEventStore(get_session_factory()))
finally:
await close_engine()
for event in (memory_event, jsonl_event, database_event):
_assert_fixed_event_valid(event, persisted=True)
assert isinstance(memory_event["content"]["messages"][0], AIMessage)
assert isinstance(jsonl_event["content"]["messages"][0], str)
assert isinstance(database_event["content"]["messages"][0], str)
assert "final answer" in jsonl_event["content"]["messages"][0]
assert "final answer" in database_event["content"]["messages"][0]
@pytest.mark.anyio
async def test_run_journal_observed_events_exactly_match_its_catalog():
store = MemoryRunEventStore()
journal = RunJournal("run-1", "thread-1", store, flush_threshold=100)
root_run_id = uuid4()
llm_run_id = uuid4()
journal.on_chain_start(
{"name": "root"},
{},
run_id=root_run_id,
parent_run_id=None,
tags=["lead_agent"],
metadata={"langgraph_step": 1},
)
journal.on_chat_model_start(
{},
[[HumanMessage(content="question", id="human-1")]],
run_id=llm_run_id,
tags=["lead_agent"],
)
journal.on_llm_end(
_make_llm_response("answer", usage={"input_tokens": 3, "output_tokens": 4, "total_tokens": 7}),
run_id=llm_run_id,
parent_run_id=None,
tags=["lead_agent"],
)
journal.on_tool_end(
ToolMessage(content="tool result", tool_call_id="call-1", name="web_search", id="tool-1"),
run_id=uuid4(),
)
journal.on_llm_error(RuntimeError("model failed"), run_id=uuid4())
journal.on_chain_error(ValueError("run failed"), run_id=uuid4())
journal.on_chain_end({"messages": []}, run_id=root_run_id, parent_run_id=None)
journal.record_memory_context(content_sha256="a" * 64)
await journal.flush()
events = await store.list_events("thread-1", "run-1")
expected_types = {definition.event_type for definition in JOURNAL_RUN_EVENT_DEFINITIONS}
assert {event["event_type"] for event in events} == expected_types
for event in events:
_assert_fixed_event_valid(event, persisted=True)
@pytest.mark.anyio
@pytest.mark.parametrize("tag", MIDDLEWARE_EVENT_TAGS)
async def test_dynamic_middleware_event_matches_pattern_contract(tag):
store = MemoryRunEventStore()
journal = RunJournal("run-1", "thread-1", store, flush_threshold=100)
journal.record_middleware(
tag,
name="GuardrailMiddleware",
hook="wrap_tool_call",
action="deny",
changes={"reason": "policy"},
)
await journal.flush()
event = (await store.list_events("thread-1", "run-1"))[0]
pattern = _load_contract()["dynamic_event_patterns"][0]
assert pattern["pattern"] == MIDDLEWARE_EVENT_PATTERN.pattern
assert set(pattern["known_tags"]) == set(MIDDLEWARE_EVENT_TAGS)
assert event["event_type"] == MIDDLEWARE_EVENT_PATTERN.event_type(tag)
assert event["category"] == MIDDLEWARE_EVENT_PATTERN.category == pattern["category"]
_assert_schema_valid(pattern["event_type_schema"], event["event_type"])
_assert_schema_valid(pattern["tag_schema"], tag)
_assert_schema_valid(pattern["content_schema"], event["content"])
_assert_schema_valid(pattern["metadata_schema"], event["metadata"])
@pytest.mark.parametrize("tag", ["", "x" * (MIDDLEWARE_EVENT_TAG_MAX_LENGTH + 1)])
def test_dynamic_middleware_event_rejects_tags_that_do_not_fit_persistence(tag):
journal = RunJournal("run-1", "thread-1", MemoryRunEventStore(), flush_threshold=100)
with pytest.raises(ValueError):
journal.record_middleware(
tag,
name="CustomMiddleware",
hook="after_model",
action="record",
changes={},
)
def test_subagent_observed_events_exactly_match_its_catalog_and_payloads():
long_result = "r" * (SUBAGENT_STEP_MAX_CHARS + 1)
long_error = "e" * (SUBAGENT_STEP_MAX_CHARS + 1)
cases = [
{"type": "task_started", "task_id": "call-1", "description": "research"},
{
"type": "task_running",
"task_id": "call-1",
"message": {
"type": "ai",
"content": "searching",
"tool_calls": [{"name": "web_search", "args": {"query": "deerflow"}}],
},
"message_index": 1,
},
{
"type": "task_running",
"task_id": "call-1",
"message": {"type": "tool", "name": "web_search", "content": "result"},
"message_index": 2,
},
{
"type": "task_completed",
"task_id": "call-1",
"result": "done",
"model_name": "test-model",
"usage": {"input_tokens": 3, "output_tokens": 4, "total_tokens": 7},
},
{"type": "task_failed", "task_id": "call-2", "error": "boom"},
{"type": "task_cancelled", "task_id": "call-3"},
{"type": "task_timed_out", "task_id": "call-4", "error": "timed out"},
{"type": "task_completed", "task_id": "call-5", "result": long_result},
{"type": "task_failed", "task_id": "call-6", "error": long_error},
]
records = [subagent_run_event(case) for case in cases]
assert all(record is not None for record in records)
typed_records = [record for record in records if record is not None]
expected_types = {definition.event_type for definition in SUBAGENT_RUN_EVENT_DEFINITIONS}
assert {record["event_type"] for record in typed_records} == expected_types
for record in typed_records:
_assert_fixed_event_valid(record)
ai_step, tool_step = typed_records[1]["content"], typed_records[2]["content"]
completed, failed = typed_records[3]["content"], typed_records[4]["content"]
timed_out = typed_records[6]["content"]
truncated_result, truncated_error = typed_records[7]["content"], typed_records[8]["content"]
assert ai_step["tool_calls"][0]["name"] == "web_search"
assert tool_step["tool_name"] == "web_search"
assert completed["result"] == "done"
assert completed["model_name"] == "test-model"
assert completed["usage"]["total_tokens"] == 7
assert failed["error"] == "boom"
assert timed_out["error"] == "timed out"
assert len(truncated_result["result"]) == SUBAGENT_STEP_MAX_CHARS
assert truncated_result["result_truncated"] is True
assert len(truncated_error["error"]) == SUBAGENT_STEP_MAX_CHARS
assert truncated_error["error_truncated"] is True
assert {record["content"]["status"] for record in typed_records[3:]} == {
"completed",
"failed",
"cancelled",
"timed_out",
}
def test_captured_subagent_message_survives_task_running_conversion():
captured: list[dict] = []
assert capture_step_message(
AIMessage(
content="searching",
id="ai-step-1",
tool_calls=[{"id": "call-1", "name": "web_search", "args": {"query": "deerflow"}}],
),
captured,
set(),
)
event = subagent_run_event(
{
"type": "task_running",
"task_id": "task-1",
"message": captured[0],
"message_index": 0,
}
)
assert event is not None
assert event["event_type"] == "subagent.step"
assert event["content"]["task_id"] == "task-1"
assert event["content"]["message_index"] == 0
assert event["content"]["text"] == "searching"
assert event["content"]["tool_calls"] == [{"name": "web_search", "args": {"query": "deerflow"}}]
_assert_fixed_event_valid(event)
@pytest.mark.parametrize(
"chunk",
[
{"type": "task_started", "description": "missing task id"},
{"type": "task_started", "task_id": "", "description": "empty task id"},
{"type": "task_started", "task_id": "call-1", "description": 42},
{"type": "task_running", "task_id": "call-1", "message": {"type": "ai"}},
{"type": "task_running", "task_id": "call-1", "message": {"type": "ai"}, "message_index": -1},
{"type": "task_running", "task_id": "call-1", "message": {"type": "ai"}, "message_index": True},
{"type": "task_running", "task_id": "call-1", "message": "not-an-object", "message_index": 0},
{"type": "task_completed"},
],
)
def test_subagent_producer_rejects_chunks_missing_contract_fields(chunk):
assert subagent_run_event(chunk) is None
@pytest.mark.anyio
@pytest.mark.parametrize("backend", ["memory", "jsonl"])
async def test_subagent_batch_round_trip_matches_contract_for_non_database_stores(backend, tmp_path):
if backend == "memory":
store = MemoryRunEventStore()
else:
from deerflow.runtime.events.store.jsonl import JsonlRunEventStore
store = JsonlRunEventStore(base_dir=tmp_path / "subagent-events")
records = await _persist_subagent_batch(store)
assert [record["event_type"] for record in records] == ["subagent.start", "subagent.step", "subagent.end"]
for record in records:
_assert_fixed_event_valid(record, persisted=True)
@pytest.mark.anyio
async def test_subagent_batch_round_trip_matches_contract_for_database_store(tmp_path):
from deerflow.persistence.engine import close_engine, get_session_factory, init_engine
from deerflow.runtime.events.store.db import DbRunEventStore
url = f"sqlite+aiosqlite:///{tmp_path / 'subagent-events.db'}"
await init_engine("sqlite", url=url, sqlite_dir=str(tmp_path))
try:
records = await _persist_subagent_batch(DbRunEventStore(get_session_factory()))
finally:
await close_engine()
assert [record["event_type"] for record in records] == ["subagent.start", "subagent.step", "subagent.end"]
for record in records:
_assert_fixed_event_valid(record, persisted=True)
@pytest.mark.anyio
async def test_workspace_change_producer_matches_catalog_and_payload(monkeypatch, tmp_path):
from deerflow.workspace_changes import WorkspaceRoot, scan_workspace_roots
from deerflow.workspace_changes import recorder as recorder_module
workspace = tmp_path / "workspace"
outputs = tmp_path / "outputs"
workspace.mkdir()
outputs.mkdir()
roots = [
WorkspaceRoot("workspace", workspace, "/mnt/user-data/workspace"),
WorkspaceRoot("outputs", outputs, "/mnt/user-data/outputs"),
]
before = scan_workspace_roots(roots)
(workspace / "report.md").write_text("# Report\n", encoding="utf-8")
monkeypatch.setattr(recorder_module, "build_thread_workspace_roots", lambda *_args, **_kwargs: roots)
store = MemoryRunEventStore()
record = await recorder_module.record_workspace_changes(store, "thread-1", "run-1", before)
assert record is not None
assert {record["event_type"]} == {definition.event_type for definition in WORKSPACE_RUN_EVENT_DEFINITIONS}
_assert_fixed_event_valid(record, persisted=True)
def test_known_gaps_do_not_reclassify_current_events_as_missing():
contract = _load_contract()
gap_ids = {gap["id"] for gap in contract["known_gaps"]}
current_types = {definition.event_type for definition in FIXED_RUN_EVENT_DEFINITIONS}
assert {"tool-call-intent", "terminal-run-status"}.issubset(gap_ids)
assert all(gap.get("event_type") not in current_types for gap in contract["known_gaps"])

18
backend/tests/test_run_manager.py
View File

@ -61,15 +61,15 @@ class PermanentStatusRunStore(MemoryRunStore):
)
class FailingStatusRunStore(MemoryRunStore):
"""Memory run store that always fails status updates."""
class FailingTakeoverRunStore(MemoryRunStore):
"""Memory run store that always fails takeover claims."""
def __init__(self) -> None:
super().__init__()
self.status_update_attempts = 0
self.takeover_attempts = 0
async def update_status(self, run_id, status, *, error=None, stop_reason=None):
self.status_update_attempts += 1
async def claim_for_takeover(self, run_id, *, grace_seconds, error):
self.takeover_attempts += 1
raise sqlite3.OperationalError("database is locked")
@ -301,9 +301,9 @@ async def test_reconcile_orphaned_inflight_runs_skips_live_local_run():
@pytest.mark.anyio
async def test_reconcile_orphaned_inflight_runs_skips_rows_when_error_status_is_not_persisted():
"""Startup recovery must not report a row as recovered if the error update failed."""
store = FailingStatusRunStore()
async def test_reconcile_orphaned_inflight_runs_skips_rows_when_takeover_claim_fails():
"""Startup recovery must not report a row as recovered if the takeover claim failed."""
store = FailingTakeoverRunStore()
await store.put("running-run", thread_id="thread-1", status="running", created_at="2026-01-01T00:00:00+00:00")
manager = RunManager(
store=store,
@ -318,7 +318,7 @@ async def test_reconcile_orphaned_inflight_runs_skips_rows_when_error_status_is_
stored = await store.get("running-run")
assert recovered == []
assert stored["status"] == "running"
assert store.status_update_attempts == 2
assert store.takeover_attempts == 2
@pytest.mark.anyio

87
backend/tests/test_safety_finish_reason_graph_integration.py
View File

@ -20,6 +20,7 @@ from __future__ import annotations
from typing import Any
import pytest
from langchain.agents import create_agent
from langchain.agents.middleware import AgentMiddleware
from langchain.agents.middleware.types import ModelRequest, ModelResponse
@ -140,6 +141,42 @@ def test_content_filter_with_tool_calls_does_not_invoke_tool_node():
assert final_ai.response_metadata.get("finish_reason") == "content_filter"
@pytest.mark.anyio
async def test_safety_termination_event_reaches_astream_events():
"""Exercise the middleware's real async graph hook and callback context."""
_TOOL_INVOCATIONS.clear()
agent = create_agent(
model=_ContentFilteredModel(),
tools=[write_file],
middleware=[SafetyFinishReasonMiddleware()],
context_schema=dict,
)
events = [
event
async for event in agent.astream_events(
{"messages": [HumanMessage(content="write me a report")]},
version="v2",
context={"thread_id": "safety-stream-thread"},
)
if event["event"] == "on_custom_event"
]
assert len(events) == 1
assert events[0]["name"] == "safety_termination"
assert events[0]["data"] == {
"type": "safety_termination",
"detector": "openai_compatible_content_filter",
"reason_field": "finish_reason",
"reason_value": "content_filter",
"suppressed_tool_call_count": 1,
"suppressed_tool_call_names": ["write_file"],
"thread_id": "safety-stream-thread",
}
assert _TOOL_INVOCATIONS == []
def test_content_filter_without_tool_calls_passes_through_unchanged():
"""No tool calls => issue scope says don't intervene; the partial
response should be delivered as-is so the user sees what they got."""
@ -179,6 +216,56 @@ def test_content_filter_without_tool_calls_passes_through_unchanged():
assert _TOOL_INVOCATIONS == []
def test_content_filter_empty_no_tool_calls_is_backfilled_not_persisted_empty():
"""#4393: a content_filter response with empty content and no tool calls
must not survive in graph state as an empty AIMessage — strict
OpenAI-compatible providers reject an empty assistant message on the next
request, poisoning the whole thread. The middleware backfills a
user-facing explanation so the persisted message is non-empty."""
_TOOL_INVOCATIONS.clear()
class _EmptyContentFilterModel(BaseChatModel):
"""Mimics Kimi/Moonshot refusing a sensitive question: an empty
assistant message flagged finish_reason='content_filter'."""
@property
def _llm_type(self) -> str:
return "fake-empty-content-filter"
def bind_tools(self, tools, **kwargs):
return self
def _generate(self, messages, stop=None, run_manager=None, **kwargs):
msg = AIMessage(content="", response_metadata={"finish_reason": "content_filter"})
return ChatResult(generations=[ChatGeneration(message=msg)])
async def _agenerate(self, messages, stop=None, run_manager=None, **kwargs):
return self._generate(messages, stop=stop, run_manager=run_manager, **kwargs)
agent = create_agent(
model=_EmptyContentFilterModel(),
tools=[write_file],
middleware=[SafetyFinishReasonMiddleware()],
)
result = agent.invoke({"messages": [HumanMessage(content="a sensitive question")]})
final_ai = next(m for m in reversed(result["messages"]) if isinstance(m, AIMessage))
# The poison condition: the persisted assistant message must not be empty.
assert isinstance(final_ai.content, str)
assert final_ai.content.strip(), "empty assistant message would be rejected by strict providers on the next turn"
assert "safety-related signal" in final_ai.content
assert "returned no content" in final_ai.content
# Observability stamp present with zero suppressed tool calls.
record = final_ai.additional_kwargs.get("safety_termination")
assert record is not None
assert record["suppressed_tool_call_count"] == 0
# Real provider reason preserved for downstream SSE / converters.
assert final_ai.response_metadata.get("finish_reason") == "content_filter"
assert _TOOL_INVOCATIONS == []
def test_normal_tool_call_round_trip_is_not_affected():
"""Regression: a healthy finish_reason='tool_calls' response must still
execute the tool. The middleware must not over-fire."""

181
backend/tests/test_safety_finish_reason_middleware.py
View File

@ -4,6 +4,7 @@ from unittest.mock import MagicMock
import pytest
from langchain_core.messages import AIMessage, HumanMessage, SystemMessage
from langgraph.errors import GraphBubbleUp
from deerflow.agents.middlewares.safety_finish_reason_middleware import SafetyFinishReasonMiddleware
from deerflow.agents.middlewares.safety_termination_detectors import (
@ -113,6 +114,93 @@ class TestTriggerCriteria:
}
assert mw._apply(state, _runtime()) is None
def test_content_filter_blank_content_no_tool_calls_backfills(self):
"""#4393: an empty content_filter response with no tool calls would be
persisted empty and rejected by strict providers on the next request.
Backfill an explanation so the persisted message is non-empty."""
mw = SafetyFinishReasonMiddleware()
state = {
"messages": [
_ai(
content="",
response_metadata={"finish_reason": "content_filter"},
)
]
}
result = mw._apply(state, _runtime())
assert result is not None
patched = result["messages"][0]
assert patched.tool_calls == []
assert isinstance(patched.content, str)
assert patched.content.strip() # never persisted empty
assert "safety-related signal" in patched.content
assert "returned no content" in patched.content
# It must not claim tool calls were suppressed — none existed.
assert "were suppressed" not in patched.content
record = patched.additional_kwargs["safety_termination"]
assert record["suppressed_tool_call_count"] == 0
assert record["suppressed_tool_call_names"] == []
def test_content_filter_whitespace_content_no_tool_calls_backfills(self):
"""Whitespace-only content is still blank to a strict provider."""
mw = SafetyFinishReasonMiddleware()
state = {
"messages": [
_ai(
content=" \n ",
response_metadata={"finish_reason": "content_filter"},
)
]
}
result = mw._apply(state, _runtime())
assert result is not None
patched = result["messages"][0]
assert patched.tool_calls == []
assert "returned no content" in patched.content
def test_content_filter_none_content_no_tool_calls_backfills(self):
"""content=None is reachable via model_copy rewrites (which skip
validation) and must be treated as blank, not stringified to 'None'."""
mw = SafetyFinishReasonMiddleware()
none_content = _ai(response_metadata={"finish_reason": "content_filter"}).model_copy(update={"content": None})
assert none_content.content is None # precondition for the regression
result = mw._apply({"messages": [none_content]}, _runtime())
assert result is not None
patched = result["messages"][0]
assert patched.tool_calls == []
assert isinstance(patched.content, str)
assert patched.content.strip()
assert "returned no content" in patched.content
def test_anthropic_refusal_blank_content_no_tool_calls_backfills(self):
"""The empty-content backfill is detector-agnostic (#4393)."""
mw = SafetyFinishReasonMiddleware()
state = {
"messages": [
_ai(
content="",
response_metadata={"stop_reason": "refusal"},
)
]
}
result = mw._apply(state, _runtime())
assert result is not None
assert result["messages"][0].content.strip()
def test_blank_content_no_tool_calls_without_safety_signal_passes_through(self):
"""A blank response with no safety signal is out of scope: only a
detected safety termination triggers the backfill."""
mw = SafetyFinishReasonMiddleware()
state = {
"messages": [
_ai(
content="",
response_metadata={"finish_reason": "stop"},
)
]
}
assert mw._apply(state, _runtime()) is None
def test_normal_tool_calls_pass_through(self):
mw = SafetyFinishReasonMiddleware()
state = {
@ -562,7 +650,7 @@ class TestAuditEvent:
assert result is not None
assert result["messages"][0].tool_calls == []
def test_journal_record_exception_does_not_break_run(self):
def test_journal_record_exception_warns_without_breaking_run(self, caplog):
"""Buggy journal must never propagate an exception into the agent loop."""
journal = MagicMock()
journal.record_middleware.side_effect = RuntimeError("db down")
@ -576,9 +664,12 @@ class TestAuditEvent:
]
}
# Must not raise.
result = mw._apply(state, self._runtime_with_journal(journal))
with caplog.at_level("WARNING"):
result = mw._apply(state, self._runtime_with_journal(journal))
assert result is not None
assert result["messages"][0].tool_calls == []
assert "Failed to record middleware:safety_termination event" in caplog.text
def test_no_record_when_passthrough(self):
"""When the middleware does NOT intervene, no audit event is written."""
@ -599,14 +690,23 @@ class TestAuditEvent:
class TestStreamEvent:
def test_emits_event_when_writer_available(self, monkeypatch):
captured: list = []
dispatched: list = []
def fake_writer(payload):
captured.append(payload)
def fake_emit_custom_event(payload, *, writer):
writer(payload)
dispatched.append(payload)
# Patch get_stream_writer at the symbol-resolution site.
import langgraph.config
monkeypatch.setattr(langgraph.config, "get_stream_writer", lambda: fake_writer)
monkeypatch.setattr(
"deerflow.agents.middlewares.safety_finish_reason_middleware.emit_custom_event",
fake_emit_custom_event,
)
mw = SafetyFinishReasonMiddleware()
state = {
@ -628,6 +728,83 @@ class TestStreamEvent:
assert payload["suppressed_tool_call_count"] == 1
assert payload["suppressed_tool_call_names"] == ["write_file"]
assert payload["thread_id"] == "t-stream"
assert dispatched == captured
@pytest.mark.anyio
async def test_async_hook_uses_async_event_dispatch(self, monkeypatch):
captured: list = []
dispatched: list = []
async def fake_emit_custom_event(payload, *, writer):
writer(payload)
dispatched.append(payload)
import langgraph.config
monkeypatch.setattr(langgraph.config, "get_stream_writer", lambda: captured.append)
monkeypatch.setattr(
"deerflow.agents.middlewares.safety_finish_reason_middleware.aemit_custom_event",
fake_emit_custom_event,
)
mw = SafetyFinishReasonMiddleware()
state = {
"messages": [
_ai(
tool_calls=[_write_call()],
response_metadata={"finish_reason": "content_filter"},
)
]
}
result = await mw.aafter_model(state, _runtime("t-async-stream"))
assert result is not None
assert result["messages"][0].tool_calls == []
assert dispatched == captured
assert [payload["type"] for payload in captured] == ["safety_termination"]
assert captured[0]["thread_id"] == "t-async-stream"
def test_sync_event_preserves_langgraph_control_flow(self, monkeypatch):
import langgraph.config
def interrupt_dispatch(*_args, **_kwargs):
raise GraphBubbleUp
monkeypatch.setattr(langgraph.config, "get_stream_writer", lambda: lambda _payload: None)
monkeypatch.setattr(
"deerflow.agents.middlewares.safety_finish_reason_middleware.emit_custom_event",
interrupt_dispatch,
)
termination = SafetyTermination(
detector="test",
reason_field="finish_reason",
reason_value="content_filter",
)
with pytest.raises(GraphBubbleUp):
SafetyFinishReasonMiddleware()._emit_event(termination, ["write_file"], _runtime())
@pytest.mark.anyio
async def test_async_event_preserves_langgraph_control_flow(self, monkeypatch):
import langgraph.config
async def interrupt_dispatch(*_args, **_kwargs):
raise GraphBubbleUp
monkeypatch.setattr(langgraph.config, "get_stream_writer", lambda: lambda _payload: None)
monkeypatch.setattr(
"deerflow.agents.middlewares.safety_finish_reason_middleware.aemit_custom_event",
interrupt_dispatch,
)
termination = SafetyTermination(
detector="test",
reason_field="finish_reason",
reason_value="content_filter",
)
with pytest.raises(GraphBubbleUp):
await SafetyFinishReasonMiddleware()._aemit_event(termination, ["write_file"], _runtime())
def test_writer_unavailable_does_not_break(self, monkeypatch):
import langgraph.config

20
backend/tests/test_skill_request_scoped_secrets.py
View File

@ -898,6 +898,26 @@ class TestInContextBindsSecrets:
# Values must never reach the audit journal.
assert "tok-secret-value" not in str(bind_calls[0])
def test_binding_audit_failure_warns_without_breaking_binding(self, tmp_path, monkeypatch, caplog):
from deerflow.runtime.secret_context import read_active_secrets
skill = _make_secret_skill(tmp_path, "erp-report", [SecretRequirement("ERP_TOKEN")])
journal = MagicMock()
journal.record_middleware.side_effect = RuntimeError("db down")
context = {"secrets": {"ERP_TOKEN": "tok-123"}, "__run_journal": journal}
with caplog.at_level("WARNING"):
self._run_call(
tmp_path,
monkeypatch,
[skill],
context=context,
skill_context=[_skill_context_entry(skill)],
)
assert read_active_secrets(context) == {"ERP_TOKEN": "tok-123"}
assert "Failed to record skill secret binding audit event" in caplog.text
def test_slash_binding_persists_across_model_calls_in_same_run(self, tmp_path, monkeypatch):
"""#3861 semantics preserved under per-call recompute: after the single
activation call, the tool loop issues more model calls without a fresh

6
backend/tests/test_slash_skills.py
View File

@ -505,7 +505,7 @@ def test_skill_activation_middleware_async_records_activation_audit_event(monkey
assert kwargs["changes"]["content_hash"] == hashlib.sha256(b"# Data Analysis\nUse pandas.").hexdigest()
def test_skill_activation_middleware_ignores_activation_audit_errors(monkeypatch, tmp_path):
def test_skill_activation_middleware_warns_and_ignores_activation_audit_errors(monkeypatch, tmp_path, caplog):
skill = _make_skill(tmp_path, "data-analysis", content="# Data Analysis\nUse pandas.")
monkeypatch.setattr(middleware_module, "get_or_new_skill_storage", lambda **kwargs: _make_storage(tmp_path, [skill]))
@ -517,10 +517,12 @@ def test_skill_activation_middleware_ignores_activation_audit_errors(monkeypatch
def handler(model_request: ModelRequest):
return AIMessage(content="ok")
result = middleware.wrap_model_call(_make_model_request([original], runtime=runtime), handler)
with caplog.at_level("WARNING"):
result = middleware.wrap_model_call(_make_model_request([original], runtime=runtime), handler)
assert isinstance(result, AIMessage)
assert result.content == "ok"
assert "Failed to record slash skill activation audit event" in caplog.text
def test_skill_activation_middleware_activates_only_latest_real_user_message(monkeypatch, tmp_path):

77
backend/tests/test_subagent_executor.py
View File

@ -46,7 +46,10 @@ _LANGGRAPH_HAS_ROOT_LINEAGE_STREAM_REGRESSION = Version(package_version("langgra
def _default_app_config():
return SimpleNamespace(tool_search=SimpleNamespace(enabled=False))
return SimpleNamespace(
tool_search=SimpleNamespace(enabled=False),
authorization=SimpleNamespace(enabled=False),
)
def _patch_default_get_app_config(executor_module):
@ -312,10 +315,13 @@ class TestAgentConstruction:
app_config=app_config,
parent_model="parent-model",
)
provider = object()
executor._authz_provider = provider
result = executor._create_agent()
assert result is agent
assert captured["middlewares"]["authorization_provider"] is provider
assert captured["model"] == {
"name": "parent-model",
"thinking_enabled": False,
@ -332,6 +338,7 @@ class TestAgentConstruction:
"lazy_init": True,
"deferred_setup": None,
"agent_name": "test-agent",
"authorization_provider": provider,
}
assert captured["agent"]["model"] is model
assert captured["agent"]["middleware"] is middlewares
@ -599,7 +606,14 @@ class TestAgentConstruction:
"get_or_new_user_skill_storage",
lambda user_id, *, app_config=None: SimpleNamespace(load_skills=lambda *, enabled_only: []),
)
monkeypatch.setattr(executor_module, "get_app_config", lambda: SimpleNamespace(tool_search=SimpleNamespace(enabled=True)))
monkeypatch.setattr(
executor_module,
"get_app_config",
lambda: SimpleNamespace(
tool_search=SimpleNamespace(enabled=True),
authorization=SimpleNamespace(enabled=False),
),
)
@as_tool
def mcp_calc(expression: str) -> str:
@ -640,7 +654,14 @@ class TestAgentConstruction:
"get_or_new_user_skill_storage",
lambda user_id, *, app_config=None: SimpleNamespace(load_skills=lambda *, enabled_only: []),
)
monkeypatch.setattr(executor_module, "get_app_config", lambda: SimpleNamespace(tool_search=SimpleNamespace(enabled=False)))
monkeypatch.setattr(
executor_module,
"get_app_config",
lambda: SimpleNamespace(
tool_search=SimpleNamespace(enabled=False),
authorization=SimpleNamespace(enabled=False),
),
)
@as_tool
def mcp_calc(expression: str) -> str:
@ -655,6 +676,47 @@ class TestAgentConstruction:
assert deferred_setup.deferred_names == frozenset()
assert "<available-deferred-tools>" not in state["messages"][0].content
@pytest.mark.anyio
async def test_build_initial_state_applies_authorization_before_deferral(
self,
classes,
base_config,
monkeypatch: pytest.MonkeyPatch,
):
from deerflow.config.authorization_config import AuthorizationConfig, AuthorizationProviderConfig
SubagentExecutor = classes["SubagentExecutor"]
monkeypatch.setattr(
sys.modules["deerflow.skills.storage"],
"get_or_new_skill_storage",
lambda *, app_config=None: SimpleNamespace(load_skills=lambda *, enabled_only: []),
)
app_config = SimpleNamespace(
authorization=AuthorizationConfig(
enabled=True,
provider=AuthorizationProviderConfig(
use="deerflow.authz.rbac:RbacAuthorizationProvider",
config={"roles": {"user": {"tools": {"allow": ["safe_tool"]}}}},
),
),
models=[SimpleNamespace(name="test-model")],
tool_search=SimpleNamespace(enabled=False),
)
executor = SubagentExecutor(
config=base_config,
tools=[NamedTool("safe_tool"), NamedTool("denied_tool")],
app_config=app_config,
parent_model="test-model",
user_role="user",
thread_id="test-thread",
)
_state, final_tools, deferred_setup = await executor._build_initial_state("Do the task")
assert [tool.name for tool in final_tools] == ["safe_tool"]
assert deferred_setup.deferred_names == frozenset()
assert executor._authz_provider is not None
@pytest.mark.anyio
async def test_build_initial_state_deferral_respects_tool_policy_and_tool_search_is_infra(
self,
@ -685,7 +747,14 @@ class TestAgentConstruction:
"get_or_new_user_skill_storage",
lambda user_id, *, app_config=None: SimpleNamespace(load_skills=lambda *, enabled_only: []),
)
monkeypatch.setattr(executor_module, "get_app_config", lambda: SimpleNamespace(tool_search=SimpleNamespace(enabled=True)))
monkeypatch.setattr(
executor_module,
"get_app_config",
lambda: SimpleNamespace(
tool_search=SimpleNamespace(enabled=True),
authorization=SimpleNamespace(enabled=False),
),
)
@as_tool
def active_tool(x: str) -> str:

168
backend/tests/test_summarize_checkpoint_channels.py Normal file
View File

@ -0,0 +1,168 @@
from __future__ import annotations
import importlib.util
import json
import sys
from pathlib import Path
import pytest
def _load_module():
path = Path(__file__).resolve().parents[1] / "scripts/benchmark/summarize_checkpoint_channels.py"
spec = importlib.util.spec_from_file_location("summarize_checkpoint_channels", path)
assert spec is not None
module = importlib.util.module_from_spec(spec)
assert spec.loader is not None
sys.modules[spec.name] = module
spec.loader.exec_module(module)
return module
summarize = _load_module()
def _row(mode: str, repetition: int, write_ms: float, checkpoint_bytes: int, *, success: bool = True) -> dict:
return {
"success": success,
"mode": mode,
"backend": "sqlite",
"scenario": "append",
"snapshot_frequency": 1000,
"update_count": 100,
"payload_bytes": 128,
"repetition": repetition,
"write_total_ms": write_ms,
"logical_checkpoint_bytes": checkpoint_bytes,
}
def test_summarize_uses_only_successful_paired_repetitions() -> None:
rows = [
_row("full", 0, 10, 1000),
_row("delta", 0, 5, 200),
_row("full", 1, 30, 3000),
_row("delta", 1, 15, 600),
_row("full", 2, 999, 9999),
_row("delta", 2, 1, 1, success=False),
]
result = summarize._summarize(rows, metrics=["write_total_ms", "logical_checkpoint_bytes"])
assert result == [
{
"backend": "sqlite",
"scenario": "append",
"snapshot_frequency": 1000,
"update_count": 100,
"payload_bytes": 128,
"paired_repetitions": 2,
"failed_rows": 1,
"full_write_total_ms": 20.0,
"delta_write_total_ms": 10.0,
"ratio_write_total_ms": 0.5,
"full_logical_checkpoint_bytes": 2000.0,
"delta_logical_checkpoint_bytes": 400.0,
"ratio_logical_checkpoint_bytes": 0.2,
}
]
def test_summarize_omits_group_without_a_successful_pair() -> None:
assert summarize._summarize([_row("full", 0, 10, 1000)], metrics=["write_total_ms"]) == []
def test_summarize_excludes_profiled_pairs_from_baseline_medians() -> None:
rows = [
_row("full", 0, 10, 1000),
_row("delta", 0, 5, 200),
{**_row("full", 1, 1000, 1000), "profiled": True},
{**_row("delta", 1, 1000, 200), "profiled": True},
]
result = summarize._summarize(rows, metrics=["write_total_ms"])
assert result[0]["paired_repetitions"] == 1
assert result[0]["full_write_total_ms"] == 10.0
assert result[0]["delta_write_total_ms"] == 5.0
def test_summarize_sorts_numeric_update_counts_numerically() -> None:
rows = []
for update_count in (10, 2):
full = _row("full", 0, 10, 1000)
delta = _row("delta", 0, 5, 200)
full["update_count"] = update_count
delta["update_count"] = update_count
rows.extend([full, delta])
result = summarize._summarize(rows, metrics=["write_total_ms"])
assert [row["update_count"] for row in result] == [2, 10]
def test_load_jsonl_reports_file_and_line_for_malformed_input(tmp_path: Path) -> None:
path = tmp_path / "results.jsonl"
path.write_text('{"success": true}\nnot-json\n', encoding="utf-8")
with pytest.raises(ValueError, match=r"results\.jsonl:2"):
summarize._load_jsonl([path])
def test_multiple_inputs_keep_same_numbered_repetitions_separate(tmp_path: Path) -> None:
first = tmp_path / "first.jsonl"
second = tmp_path / "second.jsonl"
first.write_text(
"".join(json.dumps(row) + "\n" for row in [_row("full", 0, 10, 1000), _row("delta", 0, 5, 200)]),
encoding="utf-8",
)
second.write_text(
"".join(json.dumps(row) + "\n" for row in [_row("full", 0, 30, 3000), _row("delta", 0, 15, 600)]),
encoding="utf-8",
)
result = summarize._summarize(summarize._load_jsonl([first, second]), metrics=["write_total_ms"])
assert result[0]["paired_repetitions"] == 2
assert result[0]["full_write_total_ms"] == 20.0
assert result[0]["delta_write_total_ms"] == 10.0
def test_multiple_inputs_do_not_cross_pair_single_mode_results(tmp_path: Path) -> None:
full_only = tmp_path / "full.jsonl"
delta_only = tmp_path / "delta.jsonl"
full_only.write_text(json.dumps(_row("full", 0, 10, 1000)) + "\n", encoding="utf-8")
delta_only.write_text(json.dumps(_row("delta", 0, 5, 200)) + "\n", encoding="utf-8")
rows = summarize._load_jsonl([full_only, delta_only])
assert summarize._summarize(rows, metrics=["write_total_ms"]) == []
def test_main_writes_json_summary(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
path = tmp_path / "results.jsonl"
rows = [_row("full", 0, 10, 1000), _row("delta", 0, 5, 200)]
path.write_text("".join(json.dumps(row) + "\n" for row in rows), encoding="utf-8")
rc = summarize.main([str(path), "--metrics", "write_total_ms", "--json"])
assert rc == 0
output = json.loads(capsys.readouterr().out)
assert output[0]["ratio_write_total_ms"] == 0.5
def test_main_warns_when_profiled_rows_are_skipped(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
path = tmp_path / "results.jsonl"
rows = [
_row("full", 0, 10, 1000),
_row("delta", 0, 5, 200),
{**_row("full", 1, 1000, 1000), "profiled": True},
{**_row("delta", 1, 1000, 200), "profiled": True},
]
path.write_text("".join(json.dumps(row) + "\n" for row in rows), encoding="utf-8")
rc = summarize.main([str(path), "--metrics", "write_total_ms", "--json"])
captured = capsys.readouterr()
assert rc == 0
assert "Skipping 2 profiled row(s)" in captured.err

14
backend/tests/test_task_tool_core_logic.py
View File

@ -499,9 +499,14 @@ def test_task_tool_emits_running_and_completed_events(monkeypatch):
runtime = _make_runtime()
runtime.context["deerflow_trace_id"] = "task-trace-1"
events = []
dispatched_events = []
captured = {}
get_available_tools = MagicMock(return_value=["tool-a", "tool-b"])
async def fake_emit_custom_event(payload, *, writer):
writer(payload)
dispatched_events.append(payload)
class DummyExecutor:
def __init__(self, **kwargs):
captured["executor_kwargs"] = kwargs
@ -529,6 +534,7 @@ def test_task_tool_emits_running_and_completed_events(monkeypatch):
monkeypatch.setattr(task_tool_module, "get_background_task_result", lambda _: next(responses))
monkeypatch.setattr(task_tool_module, "get_stream_writer", lambda: events.append)
monkeypatch.setattr(task_tool_module, "aemit_custom_event", fake_emit_custom_event)
monkeypatch.setattr(task_tool_module.asyncio, "sleep", _no_sleep)
# task_tool lazily imports from deerflow.tools at call time, so patch that module-level function.
monkeypatch.setattr("deerflow.tools.get_available_tools", get_available_tools)
@ -557,6 +563,7 @@ def test_task_tool_emits_running_and_completed_events(monkeypatch):
event_types = [e["type"] for e in events]
assert event_types == ["task_started", "task_running", "task_running", "task_completed"]
assert dispatched_events == events
assert events[0]["model_name"] == "ark-model"
assert events[-1]["result"] == "all done"
@ -1701,6 +1708,11 @@ def test_terminal_events_include_usage(monkeypatch, status, expected_type):
config = _make_subagent_config()
runtime = _make_runtime()
events = []
dispatched_events = []
async def fake_emit_custom_event(payload, *, writer):
writer(payload)
dispatched_events.append(payload)
records = [
{"source_run_id": "r1", "caller": "subagent:general-purpose", "input_tokens": 100, "output_tokens": 50, "total_tokens": 150},
@ -1712,6 +1724,7 @@ def test_terminal_events_include_usage(monkeypatch, status, expected_type):
monkeypatch.setattr(task_tool_module, "get_subagent_config", lambda _: config)
monkeypatch.setattr(task_tool_module, "get_background_task_result", lambda _: result)
monkeypatch.setattr(task_tool_module, "get_stream_writer", lambda: events.append)
monkeypatch.setattr(task_tool_module, "aemit_custom_event", fake_emit_custom_event)
monkeypatch.setattr(task_tool_module.asyncio, "sleep", _no_sleep)
monkeypatch.setattr(task_tool_module, "_report_subagent_usage", lambda *_: None)
monkeypatch.setattr(task_tool_module, "cleanup_background_task", lambda _: None)
@ -1727,6 +1740,7 @@ def test_terminal_events_include_usage(monkeypatch, status, expected_type):
terminal_events = [e for e in events if e["type"] == expected_type]
assert len(terminal_events) == 1
assert dispatched_events == events
assert terminal_events[0]["usage"] == {
"input_tokens": 300,
"output_tokens": 130,

304
backend/tests/test_thread_regenerate_prepare.py
View File

@ -1,12 +1,15 @@
from __future__ import annotations
import asyncio
import copy
from types import SimpleNamespace
from unittest.mock import AsyncMock, patch
import pytest
from fastapi import HTTPException
from langchain_core.messages import AIMessage, HumanMessage
from langgraph.checkpoint.base import empty_checkpoint, uuid6
from langgraph.checkpoint.memory import InMemorySaver
from deerflow.runtime import RunStatus
from deerflow.utils.messages import ORIGINAL_USER_CONTENT_KEY
@ -27,6 +30,38 @@ def _checkpoint(checkpoint_id: str, messages: list[object], *, metadata: dict |
)
async def _put_memory_checkpoint(
checkpointer: InMemorySaver,
thread_id: str,
messages: list[object],
*,
step: int,
parent_config: dict | None = None,
metadata: dict | None = None,
) -> dict:
checkpoint = empty_checkpoint()
checkpoint["id"] = str(uuid6())
checkpoint["channel_values"] = {"messages": messages}
checkpoint["channel_versions"] = {"messages": step}
checkpoint_metadata = {
"step": step,
"source": "loop",
"writes": {"test": {"messages": messages}},
"parents": {},
}
checkpoint_metadata.update(metadata or {})
return await checkpointer.aput(
parent_config or {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}},
checkpoint,
checkpoint_metadata,
{"messages": step},
)
async def _collect_checkpoints(checkpointer: InMemorySaver, config: dict) -> list:
return [checkpoint async for checkpoint in checkpointer.alist(config)]
class FakeCheckpointer:
def __init__(self, history, *, latest=None, materialized_history=None, materialized_latest=None):
self.history = history
@ -72,19 +107,26 @@ class FakeAccessor:
values=dict(checkpoint.checkpoint.get("channel_values", {})),
config=checkpoint.config,
metadata=checkpoint.metadata,
parent_config=getattr(checkpoint, "parent_config", None),
)
async def aget(self, _config):
if self.checkpointer.materialized_latest is not None:
return self.checkpointer.materialized_latest
raw = self.checkpointer.latest or (self.checkpointer.history[0] if self.checkpointer.history else None)
async def aget(self, config):
materialized_latest = getattr(self.checkpointer, "materialized_latest", None)
if materialized_latest is not None and not config.get("configurable", {}).get("checkpoint_id"):
return materialized_latest
raw = await self.checkpointer.aget_tuple(config)
return self._from_raw(raw) if raw is not None else SimpleNamespace(values={}, config={}, metadata={})
async def ahistory(self, _config, *, limit=None):
self.checkpointer.alist_limits.append(limit)
history = self.checkpointer.materialized_history
async def ahistory(self, config, *, limit=None):
alist_limits = getattr(self.checkpointer, "alist_limits", None)
if alist_limits is not None:
alist_limits.append(limit)
history = getattr(self.checkpointer, "materialized_history", None)
if history is None:
history = [self._from_raw(item) for item in self.checkpointer.history]
if hasattr(self.checkpointer, "history"):
history = [self._from_raw(item) for item in self.checkpointer.history]
else:
history = [self._from_raw(item) async for item in self.checkpointer.alist(config, limit=limit)]
return history[:limit]
@ -354,6 +396,110 @@ def test_prepare_regenerate_payload_returns_clean_input_and_base_checkpoint():
assert regenerated_human["additional_kwargs"] == {"files": [{"filename": "data.csv", "path": "/mnt/user-data/uploads/data.csv"}]}
def test_prepare_regenerate_payload_does_not_mutate_legacy_single_checkpoint_branch():
from app.gateway.routers.thread_runs import _prepare_regenerate_payload
checkpointer = InMemorySaver()
source_thread_id = "source-thread"
branch_thread_id = "legacy-branch"
source_run_id = "source-run"
human = HumanMessage(id="human-1", content="question", additional_kwargs={"run_id": source_run_id})
ai = AIMessage(id="ai-1", content="answer")
async def _seed() -> str:
source_base_config = await _put_memory_checkpoint(checkpointer, source_thread_id, [], step=0)
after_human = await _put_memory_checkpoint(
checkpointer,
source_thread_id,
[human],
step=1,
parent_config=source_base_config,
)
source_head_config = await _put_memory_checkpoint(
checkpointer,
source_thread_id,
[human, ai],
step=2,
parent_config=after_human,
)
source_head = await checkpointer.aget_tuple(source_head_config)
assert source_head is not None
legacy_head = copy.deepcopy(source_head.checkpoint)
legacy_head_id = str(uuid6())
legacy_head["id"] = legacy_head_id
legacy_metadata = copy.deepcopy(source_head.metadata)
legacy_metadata.update(
{
"source": "branch",
"deerflow_branch": True,
"branch_parent_thread_id": source_thread_id,
"branch_parent_checkpoint_id": source_head_config["configurable"]["checkpoint_id"],
"branch_parent_message_id": "ai-1",
}
)
await checkpointer.aput(
{"configurable": {"thread_id": branch_thread_id, "checkpoint_ns": ""}},
legacy_head,
legacy_metadata,
dict(legacy_head["channel_versions"]),
)
return legacy_head_id
legacy_head_id = asyncio.run(_seed())
request = _request(checkpointer, FakeEventStore([]))
with pytest.raises(HTTPException) as exc:
asyncio.run(_prepare_regenerate_payload(branch_thread_id, "ai-1", request))
assert exc.value.status_code == 409
assert exc.value.detail == "Could not find an addressable checkpoint before the target user message"
latest = asyncio.run(checkpointer.aget_tuple({"configurable": {"thread_id": branch_thread_id, "checkpoint_ns": ""}}))
assert latest is not None
assert latest.config["configurable"]["checkpoint_id"] == legacy_head_id
branch_history = asyncio.run(_collect_checkpoints(checkpointer, {"configurable": {"thread_id": branch_thread_id, "checkpoint_ns": ""}}))
assert [item.config["configurable"]["checkpoint_id"] for item in branch_history] == [legacy_head_id]
def test_prepare_regenerate_payload_rejects_legacy_branch_when_source_checkpoint_is_missing():
from app.gateway.routers.thread_runs import _prepare_regenerate_payload
checkpointer = InMemorySaver()
branch_thread_id = "legacy-orphan"
human = HumanMessage(id="human-1", content="question", additional_kwargs={"run_id": "source-run"})
ai = AIMessage(id="ai-1", content="answer")
async def _seed() -> None:
await _put_memory_checkpoint(
checkpointer,
branch_thread_id,
[human, ai],
step=1,
metadata={
"source": "branch",
"deerflow_branch": True,
"branch_parent_thread_id": "deleted-source",
"branch_parent_checkpoint_id": "missing-checkpoint",
"branch_parent_message_id": "ai-1",
},
)
asyncio.run(_seed())
with pytest.raises(HTTPException) as exc:
asyncio.run(
_prepare_regenerate_payload(
branch_thread_id,
"ai-1",
_request(checkpointer, FakeEventStore([])),
)
)
assert exc.value.status_code == 409
assert exc.value.detail == "Could not find an addressable checkpoint before the target user message"
branch_history = asyncio.run(_collect_checkpoints(checkpointer, {"configurable": {"thread_id": branch_thread_id, "checkpoint_ns": ""}}))
assert len(branch_history) == 1
def test_prepare_regenerate_uses_materialized_history_when_raw_messages_are_omitted():
from app.gateway.routers.thread_runs import _prepare_regenerate_payload
@ -402,6 +548,124 @@ def test_prepare_regenerate_uses_materialized_history_when_raw_messages_are_omit
assert checkpointer.alist_limits == [400]
def test_prepare_regenerate_rejects_cyclic_lineage_without_chronological_fallback():
from app.gateway.routers import thread_runs
human = HumanMessage(id="human-1", content="question")
ai = AIMessage(id="ai-1", content="answer")
def linked_snapshot(checkpoint_id: str, messages: list[object], parent_id: str) -> SimpleNamespace:
return SimpleNamespace(
values={"messages": messages},
config={
"configurable": {
"thread_id": "thread-1",
"checkpoint_ns": "",
"checkpoint_id": checkpoint_id,
}
},
metadata={},
parent_config={
"configurable": {
"thread_id": "thread-1",
"checkpoint_ns": "",
"checkpoint_id": parent_id,
}
},
)
head = linked_snapshot("head", [human, ai], "cycle")
cycle = linked_snapshot("cycle", [human], "head")
wrong_sibling_base = linked_snapshot("wrong-sibling", [], "root")
by_id = {"head": head, "cycle": cycle}
async def aget(config):
return by_id[config["configurable"]["checkpoint_id"]]
accessor = SimpleNamespace(
aget=aget,
ahistory=AsyncMock(return_value=[head, wrong_sibling_base]),
)
builder = AsyncMock(
return_value=(
accessor,
{"configurable": {"thread_id": "thread-1", "checkpoint_ns": ""}},
)
)
with patch.object(thread_runs, "build_thread_checkpoint_state_accessor", builder):
with pytest.raises(HTTPException) as exc:
asyncio.run(
thread_runs._find_base_checkpoint_before_human(
"thread-1",
"human-1",
_request(FakeCheckpointer([]), FakeEventStore([])),
head_checkpoint=head,
)
)
assert exc.value.status_code == 409
assert exc.value.detail == "Could not safely resolve the checkpoint before the target user message"
accessor.ahistory.assert_not_awaited()
def test_prepare_regenerate_rejects_dangling_parent_without_chronological_fallback():
from app.gateway.routers import thread_runs
human = HumanMessage(id="human-1", content="question")
ai = AIMessage(id="ai-1", content="answer")
head = SimpleNamespace(
values={"messages": [human, ai]},
config={
"configurable": {
"thread_id": "thread-1",
"checkpoint_ns": "",
"checkpoint_id": "head",
}
},
metadata={},
parent_config={
"configurable": {
"thread_id": "thread-1",
"checkpoint_ns": "",
"checkpoint_id": "missing",
}
},
)
missing = SimpleNamespace(
values={},
config=head.parent_config,
metadata=None,
created_at=None,
parent_config=None,
)
accessor = SimpleNamespace(
aget=AsyncMock(return_value=missing),
ahistory=AsyncMock(return_value=[head, _snapshot("wrong-sibling", [])]),
)
builder = AsyncMock(
return_value=(
accessor,
{"configurable": {"thread_id": "thread-1", "checkpoint_ns": ""}},
)
)
with patch.object(thread_runs, "build_thread_checkpoint_state_accessor", builder):
with pytest.raises(HTTPException) as exc:
asyncio.run(
thread_runs._find_base_checkpoint_before_human(
"thread-1",
"human-1",
_request(FakeCheckpointer([]), FakeEventStore([])),
head_checkpoint=head,
)
)
assert exc.value.status_code == 409
assert exc.value.detail == "Could not safely resolve the checkpoint before the target user message"
accessor.ahistory.assert_not_awaited()
def test_prepare_regenerate_payload_rejects_non_latest_assistant():
from app.gateway.routers.thread_runs import _prepare_regenerate_payload
@ -459,6 +723,30 @@ def test_prepare_regenerate_payload_falls_back_to_matching_run_when_events_are_m
assert response.metadata["regenerate_from_run_id"] == "run-latest"
def test_prepare_regenerate_payload_uses_server_stamped_human_run_id_without_parent_events():
from app.gateway.routers.thread_runs import _prepare_regenerate_payload
human = HumanMessage(id="human-1", content="question", additional_kwargs={"run_id": "parent-run"})
ai = AIMessage(id="ai-1", content="answer")
base = _checkpoint("ckpt-base", [])
after_human = _checkpoint("ckpt-human", [human])
latest = _checkpoint(
"ckpt-ai",
[human, ai],
metadata={
"deerflow_branch": True,
"branch_parent_thread_id": "parent-thread",
"branch_parent_checkpoint_id": "parent-checkpoint",
},
)
checkpointer = FakeCheckpointer([latest, after_human, base])
event_store = FakeEventStore([])
response = asyncio.run(_prepare_regenerate_payload("thread-1", "ai-1", _request(checkpointer, event_store)))
assert response.target_run_id == "parent-run"
def test_prepare_regenerate_payload_rejects_unverified_run_fallback_when_events_are_missing():
from app.gateway.routers.thread_runs import _prepare_regenerate_payload

581
backend/tests/test_threads_router.py
View File

@ -14,7 +14,7 @@ from langgraph.store.memory import InMemoryStore
from langgraph.types import Overwrite
from app.gateway import services as gateway_services
from app.gateway.routers import threads
from app.gateway.routers import thread_runs, threads
from deerflow.config.paths import Paths
from deerflow.persistence.thread_meta import InvalidMetadataFilterError
from deerflow.persistence.thread_meta.memory import THREADS_NS, MemoryThreadMetaStore
@ -165,6 +165,7 @@ def _patch_checkpoint_state_builder(monkeypatch):
monkeypatch.setattr(threads, "build_checkpoint_state_mutation_accessor", _mutation_builder)
monkeypatch.setattr(threads, "build_thread_checkpoint_state_accessor", _read_boundary)
monkeypatch.setattr(threads, "build_thread_checkpoint_state_mutation_accessor", _mutation_boundary)
monkeypatch.setattr(thread_runs, "build_thread_checkpoint_state_accessor", _read_boundary)
class _FakeStateAccessor:
@ -209,6 +210,7 @@ async def _write_checkpoint(
*,
step: int,
metadata: dict | None = None,
parent_config: dict | None = None,
) -> dict:
checkpoint = empty_checkpoint()
checkpoint["id"] = checkpoint_id
@ -223,7 +225,7 @@ async def _write_checkpoint(
}
checkpoint_metadata.update(metadata or {})
return await checkpointer.aput(
{"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}},
parent_config or {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}},
checkpoint,
checkpoint_metadata,
{"messages": step},
@ -1100,6 +1102,86 @@ def test_ai_message_lacks_duration_only_for_unannotated_ai_messages() -> None:
# ── branch threads from completed assistant turns ─────────────────────────────
def test_branch_thread_can_prepare_regenerate_without_branch_run_events() -> None:
app, _store, checkpointer = _build_thread_app()
app.include_router(thread_runs.router)
source_thread_id = "source-regenerate"
source_run_id = "source-run"
async def list_messages(_thread_id: str, *, limit: int, **_kwargs) -> list[dict]:
assert limit == thread_runs.REGENERATE_HISTORY_SCAN_LIMIT
return []
async def list_by_thread(_thread_id: str, *, user_id=None, limit: int = 100) -> list:
return []
app.state.run_event_store = SimpleNamespace(list_messages=list_messages)
app.state.run_manager = SimpleNamespace(list_by_thread=list_by_thread)
human = HumanMessage(id="human-1", content="Question", additional_kwargs={"run_id": source_run_id})
ai = AIMessage(id="ai-1", content="Answer")
with TestClient(app) as client:
created = client.post("/api/threads", json={"thread_id": source_thread_id, "metadata": {}})
assert created.status_code == 200, created.text
initial = asyncio.run(checkpointer.aget_tuple({"configurable": {"thread_id": source_thread_id, "checkpoint_ns": ""}}))
assert initial is not None
after_human = asyncio.run(
_write_checkpoint(
checkpointer,
source_thread_id,
str(uuid6()),
[human],
step=1,
parent_config=initial.config,
)
)
asyncio.run(
_write_checkpoint(
checkpointer,
source_thread_id,
str(uuid6()),
[human, ai],
step=2,
parent_config=after_human,
)
)
branch_response = client.post(
f"/api/threads/{source_thread_id}/branches",
json={"message_id": "ai-1", "message_ids": ["ai-1"]},
)
assert branch_response.status_code == 200, branch_response.text
branch_thread_id = branch_response.json()["thread_id"]
prepare_response = client.post(
f"/api/threads/{branch_thread_id}/runs/regenerate/prepare",
json={"message_id": "ai-1"},
)
assert prepare_response.status_code == 200, prepare_response.text
prepared = prepare_response.json()
assert prepared["target_run_id"] == source_run_id
branch_base_id = prepared["checkpoint"]["checkpoint_id"]
assert prepared["input"]["messages"][0]["id"] == "human-1"
assert prepared["input"]["messages"][0]["content"] == [{"type": "text", "text": "Question"}]
branch_base = asyncio.run(
checkpointer.aget_tuple(
{
"configurable": {
"thread_id": branch_thread_id,
"checkpoint_ns": "",
"checkpoint_id": branch_base_id,
}
}
)
)
assert branch_base is not None
assert branch_base.checkpoint.get("channel_values", {}).get("messages", []) == []
def test_branch_thread_from_older_assistant_turn_creates_truncated_thread() -> None:
app, store, checkpointer = _build_thread_app()
source_thread_id = "source-thread"
@ -1111,16 +1193,49 @@ def test_branch_thread_from_older_assistant_turn_creates_truncated_thread() -> N
human_3 = HumanMessage(id="human-3", content="Third question")
ai_3 = AIMessage(id="ai-3", content="Third answer")
async def _seed() -> None:
await _write_checkpoint(checkpointer, source_thread_id, "0001", [human_1, ai_1], step=1)
await _write_checkpoint(checkpointer, source_thread_id, "0002", [human_1, ai_1, human_2, ai_2], step=2)
await _write_checkpoint(checkpointer, source_thread_id, "0003", [human_1, ai_1, human_2, ai_2, human_3, ai_3], step=3)
asyncio.run(_seed())
async def _seed(parent_config: dict) -> dict:
after_human_1 = await _write_checkpoint(checkpointer, source_thread_id, str(uuid6()), [human_1], step=1, parent_config=parent_config)
after_ai_1 = await _write_checkpoint(checkpointer, source_thread_id, str(uuid6()), [human_1, ai_1], step=2, parent_config=after_human_1)
after_human_2 = await _write_checkpoint(
checkpointer,
source_thread_id,
str(uuid6()),
[human_1, ai_1, human_2],
step=3,
parent_config=after_ai_1,
)
after_ai_2 = await _write_checkpoint(
checkpointer,
source_thread_id,
str(uuid6()),
[human_1, ai_1, human_2, ai_2],
step=4,
parent_config=after_human_2,
)
after_human_3 = await _write_checkpoint(
checkpointer,
source_thread_id,
str(uuid6()),
[human_1, ai_1, human_2, ai_2, human_3],
step=5,
parent_config=after_ai_2,
)
await _write_checkpoint(
checkpointer,
source_thread_id,
str(uuid6()),
[human_1, ai_1, human_2, ai_2, human_3, ai_3],
step=6,
parent_config=after_human_3,
)
return after_ai_2
with TestClient(app) as client:
created = client.post("/api/threads", json={"thread_id": source_thread_id, "metadata": {}, "assistant_id": "agent"})
assert created.status_code == 200, created.text
initial = asyncio.run(checkpointer.aget_tuple({"configurable": {"thread_id": source_thread_id, "checkpoint_ns": ""}}))
assert initial is not None
target_checkpoint_config = asyncio.run(_seed(initial.config))
asyncio.run(
store.aput(
THREADS_NS,
@ -1149,7 +1264,7 @@ def test_branch_thread_from_older_assistant_turn_creates_truncated_thread() -> N
search_response = client.post("/api/threads/search", json={"limit": 10})
assert body["parent_thread_id"] == source_thread_id
assert body["parent_checkpoint_id"] == "0002"
assert body["parent_checkpoint_id"] == target_checkpoint_config["configurable"]["checkpoint_id"]
assert body["branched_from_message_id"] == "ai-2"
assert body["workspace_clone_mode"] == "skipped_historical_turn"
@ -1172,7 +1287,7 @@ def test_branch_thread_uses_materialized_history_and_overwrites_fresh_seed(monke
AIMessage(id="a2", content="Second answer"),
]
def snapshot(checkpoint_id: str, materialized_messages: list[object]) -> SimpleNamespace:
def snapshot(checkpoint_id: str, materialized_messages: list[object], *, parent_id: str | None = None) -> SimpleNamespace:
return SimpleNamespace(
values={"messages": materialized_messages, "title": "Materialized title"},
config={
@ -1183,31 +1298,48 @@ def test_branch_thread_uses_materialized_history_and_overwrites_fresh_seed(monke
}
},
metadata={"step": int(checkpoint_id[-1])},
parent_config=(
{
"configurable": {
"thread_id": source_thread_id,
"checkpoint_ns": "",
"checkpoint_id": parent_id,
}
}
if parent_id is not None
else None
),
)
source_accessor = SimpleNamespace()
source_history = [
snapshot("ckpt-2", messages),
snapshot("ckpt-1", messages[:2]),
snapshot("ckpt-2", messages, parent_id="ckpt-1"),
snapshot("ckpt-1", messages[:2], parent_id="ckpt-0"),
snapshot("ckpt-0", []),
]
branch_updates: list[tuple[dict, dict, str | None]] = []
async def source_ahistory(config, *, limit=None):
assert config["configurable"]["thread_id"] == source_thread_id
assert limit == 200
assert limit == threads._BRANCH_HISTORY_RAW_SCAN_LIMIT
return source_history
async def source_aget(config):
checkpoint_id = config["configurable"]["checkpoint_id"]
return next(item for item in source_history if item.config["configurable"]["checkpoint_id"] == checkpoint_id)
async def branch_aupdate(config, values, *, as_node=None):
branch_updates.append((config, values, as_node))
return {
"configurable": {
"thread_id": config["configurable"]["thread_id"],
"checkpoint_ns": "",
"checkpoint_id": "branch-seed",
"checkpoint_id": f"branch-{len(branch_updates)}",
}
}
source_accessor.ahistory = source_ahistory
source_accessor.aget = source_aget
branch_accessor = SimpleNamespace(aupdate=branch_aupdate)
def build_accessor(_request, *, thread_id, assistant_id=None, checkpoint_id=None):
@ -1244,13 +1376,179 @@ def test_branch_thread_uses_materialized_history_and_overwrites_fresh_seed(monke
assert response.status_code == 200, response.text
body = response.json()
assert body["parent_checkpoint_id"] == "ckpt-1"
assert len(branch_updates) == 1
update_config, update_values, as_node = branch_updates[0]
assert isinstance(update_values["messages"], Overwrite)
assert [message.id for message in update_values["messages"].value] == ["h1", "a1"]
assert update_config["configurable"]["thread_id"] == body["thread_id"]
assert update_config["metadata"]["source"] == "branch"
assert as_node == "branch"
assert len(branch_updates) == 2
replay_config, replay_values, replay_node = branch_updates[0]
assert isinstance(replay_values["messages"], Overwrite)
assert replay_values["messages"].value == []
assert replay_config["configurable"]["thread_id"] == body["thread_id"]
assert replay_config["metadata"]["source"] == "branch"
assert replay_node == "branch"
head_config, head_values, head_node = branch_updates[1]
assert isinstance(head_values["messages"], Overwrite)
assert [message.id for message in head_values["messages"].value] == ["h1", "a1"]
assert head_config["configurable"]["checkpoint_id"] == "branch-1"
assert head_config["metadata"]["source"] == "branch"
assert head_node == "branch"
@pytest.mark.parametrize(
("include_replay_base", "expected_message_ids"),
[
(True, [[], ["h1", "a1"]]),
(False, [["h1", "a1"]]),
],
ids=["chronological-replay-base", "legacy-single-checkpoint"],
)
def test_branch_thread_preserves_unlinked_legacy_histories(
monkeypatch,
include_replay_base: bool,
expected_message_ids: list[list[str]],
) -> None:
app, _store, _checkpointer = _build_thread_app()
source_thread_id = "source-unlinked"
messages = [
HumanMessage(id="h1", content="Question"),
AIMessage(id="a1", content="Answer"),
]
def snapshot(checkpoint_id: str, snapshot_messages: list[object], *, duration_only: bool = False) -> SimpleNamespace:
return SimpleNamespace(
values={"messages": snapshot_messages},
config={
"configurable": {
"thread_id": source_thread_id,
"checkpoint_ns": "",
"checkpoint_id": checkpoint_id,
}
},
metadata={"writes": {"runtime_run_duration": 1}} if duration_only else {},
parent_config=None,
)
source_history = [snapshot("ckpt-1", messages)]
if include_replay_base:
source_history.extend(
[
snapshot("ckpt-duration", [], duration_only=True),
snapshot("ckpt-0", []),
]
)
history_limits: list[int | None] = []
async def source_ahistory(config, *, limit=None):
assert config["configurable"]["thread_id"] == source_thread_id
history_limits.append(limit)
return source_history
async def unexpected_lineage_read(_config):
raise AssertionError("unlinked checkpoints must use chronological history")
branch_updates: list[dict] = []
async def branch_aupdate(config, values, *, as_node=None):
assert as_node == "branch"
branch_updates.append(values)
return {
"configurable": {
"thread_id": config["configurable"]["thread_id"],
"checkpoint_ns": "",
"checkpoint_id": f"branch-{len(branch_updates)}",
}
}
source_accessor = SimpleNamespace(ahistory=source_ahistory, aget=unexpected_lineage_read)
branch_accessor = SimpleNamespace(aupdate=branch_aupdate)
def build_accessor(_request, *, thread_id, assistant_id=None, checkpoint_id=None):
assert thread_id == source_thread_id
return source_accessor, {
"configurable": {
"thread_id": source_thread_id,
"checkpoint_ns": "",
}
}
def build_mutation_accessor(_request, *, thread_id, as_node, checkpoint_id=None, state_schema=None):
return branch_accessor, {
"configurable": {
"thread_id": thread_id,
"checkpoint_ns": "",
}
}
monkeypatch.setattr(threads, "build_checkpoint_state_accessor", build_accessor)
monkeypatch.setattr(threads, "build_checkpoint_state_mutation_accessor", build_mutation_accessor)
with TestClient(app) as client:
created = client.post(
"/api/threads",
json={"thread_id": source_thread_id, "metadata": {}, "assistant_id": "agent"},
)
assert created.status_code == 200, created.text
response = client.post(
f"/api/threads/{source_thread_id}/branches",
json={"message_id": "a1", "message_ids": ["a1"]},
)
assert response.status_code == 200, response.text
assert response.json()["parent_checkpoint_id"] == "ckpt-1"
assert [[message.id for message in update["messages"].value] for update in branch_updates] == expected_message_ids
assert history_limits == [
threads._BRANCH_HISTORY_RAW_SCAN_LIMIT,
threads._BRANCH_HISTORY_RAW_SCAN_LIMIT,
threads._BRANCH_HISTORY_RAW_SCAN_LIMIT,
]
def test_branch_history_scans_budget_for_duration_only_checkpoints() -> None:
target = SimpleNamespace(
values={
"messages": [
HumanMessage(id="h1", content="Question"),
AIMessage(id="a1", content="Answer"),
]
},
config={
"configurable": {
"thread_id": "source-duration-budget",
"checkpoint_ns": "",
"checkpoint_id": "target",
}
},
metadata={},
)
duration_only = [
SimpleNamespace(
values={"messages": []},
config={
"configurable": {
"thread_id": "source-duration-budget",
"checkpoint_ns": "",
"checkpoint_id": f"duration-{index}",
}
},
metadata={"writes": {"runtime_run_duration": index}},
)
for index in range(threads._BRANCH_HISTORY_SCAN_LIMIT)
]
history = [*duration_only, target]
limits: list[int | None] = []
async def ahistory(_config, *, limit=None):
limits.append(limit)
return history[:limit]
accessor = SimpleNamespace(ahistory=ahistory)
config = {"configurable": {"thread_id": "source-duration-budget", "checkpoint_ns": ""}}
found = asyncio.run(threads._find_branch_checkpoint(accessor, config, {"a1"}))
targets_latest = asyncio.run(threads._branch_targets_latest_turn(accessor, config, {"a1"}))
assert found is target
assert targets_latest is True
assert limits == [threads._BRANCH_HISTORY_RAW_SCAN_LIMIT, threads._BRANCH_HISTORY_RAW_SCAN_LIMIT]
def test_branch_thread_real_mutation_graph_finishes_without_scheduling(monkeypatch) -> None:
@ -1264,7 +1562,7 @@ def test_branch_thread_real_mutation_graph_finishes_without_scheduling(monkeypat
AIMessage(id="a2", content="Second answer"),
]
def snapshot(checkpoint_id: str, materialized_messages: list[object]) -> SimpleNamespace:
def snapshot(checkpoint_id: str, materialized_messages: list[object], *, parent_id: str | None = None) -> SimpleNamespace:
return SimpleNamespace(
values={"messages": materialized_messages},
config={
@ -1275,15 +1573,27 @@ def test_branch_thread_real_mutation_graph_finishes_without_scheduling(monkeypat
}
},
metadata={},
parent_config=(
{
"configurable": {
"thread_id": source_thread_id,
"checkpoint_ns": "",
"checkpoint_id": parent_id,
}
}
if parent_id is not None
else None
),
)
source_history = [
snapshot("ckpt-2", messages, parent_id="ckpt-1"),
snapshot("ckpt-1", messages[:2], parent_id="ckpt-0"),
snapshot("ckpt-0", []),
]
source_accessor = SimpleNamespace(
ahistory=AsyncMock(
return_value=[
snapshot("ckpt-2", messages),
snapshot("ckpt-1", messages[:2]),
]
)
ahistory=AsyncMock(return_value=source_history),
aget=AsyncMock(side_effect=lambda config: next(item for item in source_history if item.config["configurable"]["checkpoint_id"] == config["configurable"]["checkpoint_id"])),
)
def source_builder(_request, *, thread_id, assistant_id=None, checkpoint_id=None):
@ -1374,6 +1684,7 @@ def _wire_extension_agent(monkeypatch, app, checkpointer, mode):
monkeypatch.setattr(threads, "build_checkpoint_state_mutation_accessor", gateway_services.build_checkpoint_state_mutation_accessor)
monkeypatch.setattr(threads, "build_thread_checkpoint_state_accessor", gateway_services.build_thread_checkpoint_state_accessor)
monkeypatch.setattr(threads, "build_thread_checkpoint_state_mutation_accessor", gateway_services.build_thread_checkpoint_state_mutation_accessor)
monkeypatch.setattr(thread_runs, "build_thread_checkpoint_state_accessor", gateway_services.build_thread_checkpoint_state_accessor)
return custom_factory
@ -1389,6 +1700,24 @@ async def _seed_extension_source(checkpointer, custom_factory, mode, source_thre
{"messages": [AIMessage(id="a1", content="answer")], "ext_list": ["payload"]},
as_node="model",
)
await accessor.aupdate(
{"configurable": {"thread_id": source_thread_id, "checkpoint_ns": ""}},
{
"messages": [
HumanMessage(
id="h2",
content="follow-up",
additional_kwargs={"run_id": "source-run"},
)
]
},
as_node="model",
)
await accessor.aupdate(
{"configurable": {"thread_id": source_thread_id, "checkpoint_ns": ""}},
{"messages": [AIMessage(id="a2", content="follow-up answer")]},
as_node="model",
)
@pytest.mark.parametrize("mode", ["full", "delta"])
@ -1397,11 +1726,23 @@ def test_state_endpoints_preserve_extension_reducer_channels(monkeypatch, mode)
GET /state must return the extension value (resolved via the thread's
assistant_id), POST /state must replace it, and branch must preserve it
byte-for-byte by copying reducer channels with Overwrite semantics.
byte-for-byte by copying reducer channels with Overwrite semantics. The
copied pre-user checkpoint must also remain materializable for regenerate.
"""
app, _store, checkpointer = _build_thread_app()
app.include_router(thread_runs.router)
custom_factory = _wire_extension_agent(monkeypatch, app, checkpointer, mode)
async def list_messages(_thread_id: str, *, limit: int, **_kwargs) -> list[dict]:
assert limit == thread_runs.REGENERATE_HISTORY_SCAN_LIMIT
return []
async def list_by_thread(_thread_id: str, *, user_id=None, limit: int = 100) -> list:
return []
app.state.run_event_store = SimpleNamespace(list_messages=list_messages)
app.state.run_manager = SimpleNamespace(list_by_thread=list_by_thread)
recorded_updates: list[dict] = []
real_mutation_builder = gateway_services.build_checkpoint_state_mutation_accessor
@ -1444,11 +1785,17 @@ def test_state_endpoints_preserve_extension_reducer_channels(monkeypatch, mode)
branch_response = client.post(
f"/api/threads/{source_thread_id}/branches",
json={"message_id": "a1", "message_ids": ["a1"]},
json={"message_id": "a2", "message_ids": ["a2"]},
)
assert branch_response.status_code == 200, branch_response.text
branch_thread_id = branch_response.json()["thread_id"]
prepare_response = client.post(
f"/api/threads/{branch_thread_id}/runs/regenerate/prepare",
json={"message_id": "a2"},
)
assert prepare_response.status_code == 200, prepare_response.text
# The branch write must copy every reducer channel with replace semantics.
branch_update = recorded_updates[-1]
assert isinstance(branch_update["ext_list"], Overwrite)
@ -1462,7 +1809,129 @@ def test_state_endpoints_preserve_extension_reducer_channels(monkeypatch, mode)
branch_values = asyncio.run(materialize(branch_thread_id))
assert branch_values["ext_list"] == ["replaced"]
assert [message.id for message in branch_values["messages"]] == ["h1", "a1"]
assert [message.id for message in branch_values["messages"]] == ["h1", "a1", "h2", "a2"]
prepared = prepare_response.json()
assert prepared["target_run_id"] == "source-run"
assert prepared["input"]["messages"][0]["id"] == "h2"
base_accessor = CheckpointStateAccessor.bind(custom_factory(), checkpointer, mode=mode)
base_values = asyncio.run(
base_accessor.aget(
{
"configurable": {
"thread_id": branch_thread_id,
"checkpoint_ns": prepared["checkpoint"]["checkpoint_ns"],
"checkpoint_id": prepared["checkpoint"]["checkpoint_id"],
}
}
)
).values
assert [message.id for message in base_values["messages"]] == ["h1", "a1"]
async def _seed_branch_history_source(checkpointer, custom_factory, mode, source_thread_id):
"""Seed a completed turn whose history includes hidden and tool messages."""
accessor = CheckpointStateAccessor.bind(custom_factory(), checkpointer, mode=mode)
config = {"configurable": {"thread_id": source_thread_id, "checkpoint_ns": ""}}
await accessor.aupdate(
config,
{
"messages": [
HumanMessage(id="h1", content="question"),
HumanMessage(id="h-hidden", content="internal", additional_kwargs={"hide_from_ui": True}),
]
},
as_node="model",
)
await accessor.aupdate(
config,
{
"messages": [
ToolMessage(id="t1", content="tool output", tool_call_id="call-1"),
AIMessage(id="a1", content="answer"),
]
},
as_node="model",
)
@pytest.mark.parametrize("mode", ["full", "delta"])
def test_branch_seeds_run_events_with_parent_history(monkeypatch, mode) -> None:
"""Branching must seed the branch's run-event feed with the parent history.
The thread feed (``GET /messages`` / ``/messages/page``) reads the
run-event store, not checkpoints; without seeding, a fresh branch has no
message rows, so the inherited history vanishes from the UI as soon as
the branch's first run refreshes the feed (#4380 problem 2).
"""
from deerflow.runtime.events.store.memory import MemoryRunEventStore
app, _store, checkpointer = _build_thread_app()
custom_factory = _wire_extension_agent(monkeypatch, app, checkpointer, mode)
event_store = MemoryRunEventStore()
app.state.run_event_store = event_store
source_thread_id = f"branch-history-source-{mode}"
with TestClient(app) as client:
created = client.post(
"/api/threads",
json={"thread_id": source_thread_id, "metadata": {}, "assistant_id": "extension-agent"},
)
assert created.status_code == 200, created.text
asyncio.run(_seed_branch_history_source(checkpointer, custom_factory, mode, source_thread_id))
branch_response = client.post(
f"/api/threads/{source_thread_id}/branches",
json={"message_id": "a1", "message_ids": ["a1"]},
)
assert branch_response.status_code == 200, branch_response.text
branch_thread_id = branch_response.json()["thread_id"]
rows = asyncio.run(event_store.list_messages(branch_thread_id, user_id=None))
# The visible parent history is seeded in order; hidden messages are not.
assert [row["content"]["id"] for row in rows] == ["h1", "t1", "a1"]
assert [row["event_type"] for row in rows] == ["llm.human.input", "llm.tool.result", "llm.ai.response"]
assert all(row["category"] == "message" for row in rows)
assert all(row["run_id"] == f"branch-seed-{branch_thread_id}" for row in rows)
assert all((row.get("metadata") or {}).get("branch_seed") is True for row in rows)
seqs = [row["seq"] for row in rows]
assert seqs == sorted(seqs)
assert branch_response.json()["history_seed_mode"] == "seeded"
# The parent thread's feed stays untouched.
assert asyncio.run(event_store.list_messages(source_thread_id, user_id=None)) == []
def test_branch_history_seed_failure_keeps_branch_usable(monkeypatch) -> None:
"""A seeding failure must degrade, not fail the branch (best-effort)."""
class _ExplodingStore:
async def put_batch(self, events):
raise RuntimeError("event store down")
app, _store, checkpointer = _build_thread_app()
custom_factory = _wire_extension_agent(monkeypatch, app, checkpointer, "full")
app.state.run_event_store = _ExplodingStore()
source_thread_id = "branch-history-source-failure"
with TestClient(app) as client:
created = client.post(
"/api/threads",
json={"thread_id": source_thread_id, "metadata": {}, "assistant_id": "extension-agent"},
)
assert created.status_code == 200, created.text
asyncio.run(_seed_branch_history_source(checkpointer, custom_factory, "full", source_thread_id))
branch_response = client.post(
f"/api/threads/{source_thread_id}/branches",
json={"message_id": "a1", "message_ids": ["a1"]},
)
assert branch_response.status_code == 200, branch_response.text
assert branch_response.json()["history_seed_mode"] == "failed"
def test_update_thread_state_rejects_unknown_state_fields(monkeypatch) -> None:
@ -1520,14 +1989,16 @@ def test_branch_thread_rejects_non_assistant_targets() -> None:
human = HumanMessage(id="human-1", content="Question")
ai = AIMessage(id="ai-1", content="Answer")
async def _seed() -> None:
await _write_checkpoint(checkpointer, source_thread_id, "0001", [human, ai], step=1)
asyncio.run(_seed())
async def _seed(parent_config: dict) -> None:
after_human = await _write_checkpoint(checkpointer, source_thread_id, str(uuid6()), [human], step=1, parent_config=parent_config)
await _write_checkpoint(checkpointer, source_thread_id, str(uuid6()), [human, ai], step=2, parent_config=after_human)
with TestClient(app) as client:
created = client.post("/api/threads", json={"thread_id": source_thread_id, "metadata": {}})
assert created.status_code == 200, created.text
initial = asyncio.run(checkpointer.aget_tuple({"configurable": {"thread_id": source_thread_id, "checkpoint_ns": ""}}))
assert initial is not None
asyncio.run(_seed(initial.config))
response = client.post(
f"/api/threads/{source_thread_id}/branches",
@ -1555,10 +2026,9 @@ def test_branch_thread_best_effort_copies_current_workspace(tmp_path) -> None:
human = HumanMessage(id="human-file", content="Make a file")
ai = AIMessage(id="ai-file", content="Done")
async def _seed() -> None:
await _write_checkpoint(checkpointer, source_thread_id, "0001", [human, ai], step=1)
asyncio.run(_seed())
async def _seed(parent_config: dict) -> None:
after_human = await _write_checkpoint(checkpointer, source_thread_id, str(uuid6()), [human], step=1, parent_config=parent_config)
await _write_checkpoint(checkpointer, source_thread_id, str(uuid6()), [human, ai], step=2, parent_config=after_human)
with (
patch("app.gateway.routers.threads.get_paths", return_value=paths),
@ -1567,6 +2037,9 @@ def test_branch_thread_best_effort_copies_current_workspace(tmp_path) -> None:
):
created = client.post("/api/threads", json={"thread_id": source_thread_id, "metadata": {}})
assert created.status_code == 200, created.text
initial = asyncio.run(checkpointer.aget_tuple({"configurable": {"thread_id": source_thread_id, "checkpoint_ns": ""}}))
assert initial is not None
asyncio.run(_seed(initial.config))
response = client.post(
f"/api/threads/{source_thread_id}/branches",
@ -1606,11 +2079,26 @@ def test_branch_thread_from_historical_turn_skips_workspace_clone(tmp_path) -> N
human_2 = HumanMessage(id="human-2", content="Second question")
ai_2 = AIMessage(id="ai-2", content="Second answer")
async def _seed() -> None:
await _write_checkpoint(checkpointer, source_thread_id, "0001", [human_1, ai_1], step=1)
await _write_checkpoint(checkpointer, source_thread_id, "0002", [human_1, ai_1, human_2, ai_2], step=2)
asyncio.run(_seed())
async def _seed(parent_config: dict) -> dict:
after_human_1 = await _write_checkpoint(checkpointer, source_thread_id, str(uuid6()), [human_1], step=1, parent_config=parent_config)
after_ai_1 = await _write_checkpoint(checkpointer, source_thread_id, str(uuid6()), [human_1, ai_1], step=2, parent_config=after_human_1)
after_human_2 = await _write_checkpoint(
checkpointer,
source_thread_id,
str(uuid6()),
[human_1, ai_1, human_2],
step=3,
parent_config=after_ai_1,
)
await _write_checkpoint(
checkpointer,
source_thread_id,
str(uuid6()),
[human_1, ai_1, human_2, ai_2],
step=4,
parent_config=after_human_2,
)
return after_ai_1
with (
patch("app.gateway.routers.threads.get_paths", return_value=paths),
@ -1619,6 +2107,9 @@ def test_branch_thread_from_historical_turn_skips_workspace_clone(tmp_path) -> N
):
created = client.post("/api/threads", json={"thread_id": source_thread_id, "metadata": {}})
assert created.status_code == 200, created.text
initial = asyncio.run(checkpointer.aget_tuple({"configurable": {"thread_id": source_thread_id, "checkpoint_ns": ""}}))
assert initial is not None
target_checkpoint_config = asyncio.run(_seed(initial.config))
response = client.post(
f"/api/threads/{source_thread_id}/branches",
@ -1627,7 +2118,7 @@ def test_branch_thread_from_historical_turn_skips_workspace_clone(tmp_path) -> N
assert response.status_code == 200, response.text
body = response.json()
assert body["parent_checkpoint_id"] == "0001"
assert body["parent_checkpoint_id"] == target_checkpoint_config["configurable"]["checkpoint_id"]
assert body["workspace_clone_mode"] == "skipped_historical_turn"
target_user_data = paths.sandbox_user_data_dir(body["thread_id"], user_id=user_id)

556
backend/tests/test_worker_stream_subgraph_namespace.py Normal file
View File

@ -0,0 +1,556 @@
"""Subgraph stream frames must not impersonate root-graph frames (#4399).
The gateway worker drives ``agent.astream(subgraphs=...)`` and publishes each
frame to the StreamBridge. Delegated subagent graphs inherit the parent's
checkpoint namespace (``subagents/executor.py``), so with ``subgraphs=True``
their values snapshots and token chunks arrive interleaved with root frames.
Publishing them under bare event names lets a subagent's values snapshot
replace the whole thread view in SDK clients and floods the parent message
stream with the subagent's token chunks. The namespace must ride the SSE event
name (LangGraph Platform style ``mode|ns1|ns2``) and namespaced frames must
bypass the root-only consumers (file-tool chunk batcher, subagent event
persistence).
"""
import asyncio
import importlib
import sys
from importlib.metadata import version as package_version
from types import SimpleNamespace
import pytest
from langchain_core.messages import AIMessage, HumanMessage, ToolMessage
from packaging.version import Version
from deerflow.runtime.runs import worker
from deerflow.runtime.runs.manager import RunRecord
from deerflow.runtime.runs.schemas import DisconnectMode, RunStatus
from deerflow.runtime.runs.worker import (
_compose_sse_event,
_publish_stream_item,
_unpack_stream_item,
)
from deerflow.runtime.stream_bridge.memory import MemoryStreamBridge
SUBAGENT_NS = ("tools:call_subagent_1",)
# Delegated graphs inherit the parent checkpoint namespace (and therefore
# stream as subgraphs) only on LangGraph >= 1.2.6 — same gate as
# tests/test_subagent_executor.py::TestSubagentCheckpointLineage.
_LANGGRAPH_INHERITS_SUBGRAPH_NAMESPACE = Version(package_version("langgraph")) >= Version("1.2.6")
class _FakeBridge:
def __init__(self) -> None:
self.published: list[tuple[str, str, object]] = []
async def publish(self, run_id: str, event: str, payload: object) -> None:
self.published.append((run_id, event, payload))
class _FakeSubagentEvents:
def __init__(self) -> None:
self.added: list[object] = []
async def add(self, chunk: object) -> None:
self.added.append(chunk)
class _SpyBatcher:
"""Observable stand-in for _LargeFileToolChunkBatcher."""
def __init__(self) -> None:
self.pushed: list[object] = []
self.finish_calls = 0
self.flush_calls = 0
def push(self, chunk: object) -> list[object]:
self.pushed.append(chunk)
return [chunk]
def finish(self) -> list[object]:
self.finish_calls += 1
return []
def flush(self) -> list[object]:
self.flush_calls += 1
return []
class TestUnpackStreamItem:
def test_root_frame_with_subgraphs_has_empty_namespace(self):
mode, chunk, namespace = _unpack_stream_item(((), "values", {"messages": []}), ["values"], True)
assert mode == "values"
assert namespace == ()
def test_subgraph_frame_preserves_namespace(self):
mode, chunk, namespace = _unpack_stream_item((SUBAGENT_NS, "values", {"messages": []}), ["values"], True)
assert mode == "values"
assert namespace == SUBAGENT_NS
def test_nested_subgraph_namespace_is_preserved_in_order(self):
ns = ("tools:call_a", "model_request:xyz")
_mode, _chunk, namespace = _unpack_stream_item((ns, "messages", object()), ["messages"], True)
assert namespace == ns
def test_two_tuple_under_subgraphs_is_root(self):
mode, _chunk, namespace = _unpack_stream_item(("custom", {"type": "task_started"}), ["custom"], True)
assert mode == "custom"
assert namespace == ()
def test_without_subgraphs_frames_are_root(self):
mode, _chunk, namespace = _unpack_stream_item(("values", {}), ["values"], False)
assert mode == "values"
assert namespace == ()
def test_single_mode_fallback_is_root(self):
mode, chunk, namespace = _unpack_stream_item({"messages": []}, ["values"], False)
assert mode == "values"
assert chunk == {"messages": []}
assert namespace == ()
def test_unparsable_item_under_subgraphs(self):
mode, chunk, namespace = _unpack_stream_item("garbage", ["values"], True)
assert mode is None
assert chunk is None
assert namespace == ()
class TestComposeSseEvent:
def test_root_frame_keeps_bare_event_name(self):
assert _compose_sse_event("values", ()) == "values"
def test_subgraph_frame_gets_namespace_qualified_name(self):
assert _compose_sse_event("values", SUBAGENT_NS) == "values|tools:call_subagent_1"
def test_nested_namespace_joins_all_segments(self):
assert _compose_sse_event("messages", ("tools:call_a", "model_request:xyz")) == "messages|tools:call_a|model_request:xyz"
class TestPublishStreamItem:
@pytest.mark.asyncio
async def test_subagent_values_snapshot_is_never_published_as_bare_values(self):
# The #4399 regression: a delegated subagent's values snapshot published
# as bare "values" replaces the whole thread view in SDK clients.
bridge = _FakeBridge()
await _publish_stream_item(
bridge=bridge,
run_id="run-1",
mode="values",
chunk={"messages": [{"type": "human", "content": "subagent task prompt"}]},
namespace=SUBAGENT_NS,
file_tool_chunk_batcher=None,
subagent_events=_FakeSubagentEvents(),
)
assert [event for _run, event, _payload in bridge.published] == ["values|tools:call_subagent_1"]
@pytest.mark.asyncio
async def test_root_values_snapshot_keeps_bare_event_name(self):
bridge = _FakeBridge()
await _publish_stream_item(
bridge=bridge,
run_id="run-1",
mode="values",
chunk={"messages": []},
namespace=(),
file_tool_chunk_batcher=None,
subagent_events=_FakeSubagentEvents(),
)
assert [event for _run, event, _payload in bridge.published] == ["values"]
@pytest.mark.asyncio
async def test_subagent_message_chunks_are_namespaced(self):
bridge = _FakeBridge()
await _publish_stream_item(
bridge=bridge,
run_id="run-1",
mode="messages",
chunk=({"content": "token"}, {"langgraph_node": "model"}),
namespace=SUBAGENT_NS,
file_tool_chunk_batcher=_SpyBatcher(),
subagent_events=_FakeSubagentEvents(),
)
assert [event for _run, event, _payload in bridge.published] == ["messages|tools:call_subagent_1"]
@pytest.mark.asyncio
async def test_root_custom_event_is_persisted_for_subagent_history(self):
bridge = _FakeBridge()
subagent_events = _FakeSubagentEvents()
chunk = {"type": "task_started", "task_id": "call_1"}
await _publish_stream_item(
bridge=bridge,
run_id="run-1",
mode="custom",
chunk=chunk,
namespace=(),
file_tool_chunk_batcher=None,
subagent_events=subagent_events,
)
assert [event for _run, event, _payload in bridge.published] == ["custom"]
assert subagent_events.added == [chunk]
@pytest.mark.asyncio
async def test_subgraph_custom_event_is_not_persisted(self):
bridge = _FakeBridge()
subagent_events = _FakeSubagentEvents()
await _publish_stream_item(
bridge=bridge,
run_id="run-1",
mode="custom",
chunk={"type": "noise"},
namespace=SUBAGENT_NS,
file_tool_chunk_batcher=None,
subagent_events=subagent_events,
)
assert [event for _run, event, _payload in bridge.published] == ["custom|tools:call_subagent_1"]
assert subagent_events.added == []
@pytest.mark.asyncio
async def test_only_root_frames_drive_the_file_tool_batcher(self):
bridge = _FakeBridge()
batcher = _SpyBatcher()
# A subagent values frame must not finish() a pending root batch...
await _publish_stream_item(
bridge=bridge,
run_id="run-1",
mode="values",
chunk={"messages": []},
namespace=SUBAGENT_NS,
file_tool_chunk_batcher=batcher,
subagent_events=_FakeSubagentEvents(),
)
assert batcher.finish_calls == 0
assert batcher.pushed == []
# ...while a root values frame does.
await _publish_stream_item(
bridge=bridge,
run_id="run-1",
mode="values",
chunk={"messages": []},
namespace=(),
file_tool_chunk_batcher=batcher,
subagent_events=_FakeSubagentEvents(),
)
assert batcher.finish_calls == 1
@pytest.mark.asyncio
async def test_root_message_chunks_go_through_the_batcher(self):
bridge = _FakeBridge()
batcher = _SpyBatcher()
chunk = ({"content": "token"}, {"langgraph_node": "model"})
await _publish_stream_item(
bridge=bridge,
run_id="run-1",
mode="messages",
chunk=chunk,
namespace=(),
file_tool_chunk_batcher=batcher,
subagent_events=_FakeSubagentEvents(),
)
assert batcher.pushed == [chunk]
assert [event for _run, event, _payload in bridge.published] == ["messages"]
# ---------------------------------------------------------------------------
# Production-shaped integration: SubagentExecutor -> astream(subgraphs=...)
# -> run_agent stream loop -> StreamBridge. The namespace must originate from
# LangGraph's own delegation routing (checkpoint-namespace inheritance), not
# be hand-fed to the publishing helper — the #4399 regression lived in that
# interaction, not in any helper in isolation.
# ---------------------------------------------------------------------------
_CHILD_MESSAGE_IDS = frozenset(
{
"child-task-sentinel",
"child-ai-sentinel",
"child-tool-sentinel",
"child-final-sentinel",
}
)
_PARENT_FINAL_ID = "parent-final-sentinel"
_THREAD_ID = "thread-subgraph-stream-integration"
@pytest.fixture
def real_executor_module():
"""Swap the conftest MagicMock for the real subagent executor module.
conftest.py mocks ``deerflow.subagents.executor`` to break a package-init
import cycle; by the time this fixture runs every other deerflow module is
already imported, so a fresh import of the real module is safe.
"""
original = sys.modules.get("deerflow.subagents.executor")
sys.modules.pop("deerflow.subagents.executor", None)
subagents_pkg = sys.modules.get("deerflow.subagents")
if subagents_pkg is not None and hasattr(subagents_pkg, "executor"):
delattr(subagents_pkg, "executor")
module = importlib.import_module("deerflow.subagents.executor")
# Hermetic in CI (no config.yaml) — same defaults as test_subagent_executor.
module.get_app_config = lambda: SimpleNamespace(tool_search=SimpleNamespace(enabled=False))
module.build_tracing_callbacks = lambda: []
yield module
if original is not None:
sys.modules["deerflow.subagents.executor"] = original
else:
sys.modules.pop("deerflow.subagents.executor", None)
subagents_pkg = sys.modules.get("deerflow.subagents")
if subagents_pkg is not None and hasattr(subagents_pkg, "executor"):
delattr(subagents_pkg, "executor")
class _RecordingStreamBridge(MemoryStreamBridge):
"""Real in-memory bridge that also records (event, payload) pairs."""
def __init__(self) -> None:
super().__init__()
self.published: list[tuple[str, object]] = []
async def publish(self, run_id: str, event: str, payload: object) -> None:
self.published.append((event, payload))
await super().publish(run_id, event, payload)
class _IntegrationRunManager:
def __init__(self, record: RunRecord) -> None:
self._record = record
async def wait_for_prior_finalizing(self, *_args, **_kwargs):
return None
async def set_status(self, _run_id, status, **_kwargs):
self._record.status = status
async def update_model_name(self, *_args, **_kwargs):
return None
async def update_run_completion(self, *_args, **_kwargs):
return None
async def has_later_started_run(self, *_args, **_kwargs):
return False
async def set_finalizing(self, *_args, **_kwargs):
return None
def _collect_ids(payload: object) -> set[str]:
"""All string ``id`` values anywhere in a serialized stream payload."""
ids: set[str] = set()
def walk(node: object) -> None:
if isinstance(node, dict):
node_id = node.get("id")
if isinstance(node_id, str):
ids.add(node_id)
for value in node.values():
walk(value)
elif isinstance(node, (list, tuple)):
for value in node:
walk(value)
walk(payload)
return ids
def _build_delegating_parent_graph(executor_module, monkeypatch, *, child_emits_error_fallback: bool = False):
"""Real parent graph whose node delegates a scripted child through the
real ``SubagentExecutor`` and emits ``task_*`` custom events the way the
production task tool does (root-graph ``get_stream_writer``).
With ``child_emits_error_fallback`` the child stream contains an assistant
message carrying the ``deerflow_error_fallback`` marker (not as its final
message, so the delegation itself still completes) — the shape whose leak
would mark the *parent* run as errored (#4399).
"""
from langgraph.config import get_stream_writer
from langgraph.graph import END, START, MessagesState, StateGraph
from deerflow.subagents.config import SubagentConfig
child_builder = StateGraph(MessagesState)
child_builder.add_node(
"child_model",
lambda _state: {
"messages": [
AIMessage(
content="",
id="child-ai-sentinel",
tool_calls=[{"name": "child_tool", "args": {}, "id": "child-tool-call", "type": "tool_call"}],
)
]
},
)
child_builder.add_node(
"child_tool",
lambda _state: {"messages": [ToolMessage(content="child tool output", name="child_tool", tool_call_id="child-tool-call", id="child-tool-sentinel")]},
)
child_builder.add_node(
"child_fallback",
lambda _state: {
"messages": [
AIMessage(
content="child provider failed after retries",
id="child-fallback-sentinel",
additional_kwargs={"deerflow_error_fallback": True},
)
]
},
)
child_builder.add_node(
"child_final",
lambda _state: {"messages": [AIMessage(content="child final answer", id="child-final-sentinel")]},
)
child_builder.add_edge(START, "child_model")
child_builder.add_edge("child_model", "child_tool")
if child_emits_error_fallback:
child_builder.add_edge("child_tool", "child_fallback")
child_builder.add_edge("child_fallback", "child_final")
else:
child_builder.add_edge("child_tool", "child_final")
child_builder.add_edge("child_final", END)
child_graph = child_builder.compile(checkpointer=False)
executor = executor_module.SubagentExecutor(
config=SubagentConfig(
name="general-purpose",
description="Namespace integration test agent",
system_prompt="You are a namespace integration test agent.",
max_turns=5,
timeout_seconds=30,
),
tools=[],
parent_model="test-model",
thread_id=_THREAD_ID,
trace_id="trace-namespace-integration",
)
async def build_initial_state(task):
return ({"messages": [HumanMessage(content=task, id="child-task-sentinel")]}, [], None)
monkeypatch.setattr(executor, "_build_initial_state", build_initial_state)
monkeypatch.setattr(executor, "_create_agent", lambda *_args, **_kwargs: child_graph)
async def delegate(_state):
writer = get_stream_writer()
task_id = executor.execute_async("run the delegated child graph")
writer({"type": "task_started", "task_id": task_id})
try:
deadline = asyncio.get_running_loop().time() + 10
while True:
result = executor_module.get_background_task_result(task_id)
if result is not None and result.status.is_terminal:
break
if asyncio.get_running_loop().time() >= deadline:
pytest.fail("delegated subagent did not complete")
await asyncio.sleep(0.001)
assert result.status.value == "completed", f"delegation failed: {result.error}"
finally:
executor_module.cleanup_background_task(task_id)
writer({"type": "task_completed", "task_id": task_id})
return {"messages": [AIMessage(content="parent final answer", id=_PARENT_FINAL_ID)]}
parent_builder = StateGraph(MessagesState)
parent_builder.add_node("delegate", delegate)
parent_builder.add_edge(START, "delegate")
parent_builder.add_edge("delegate", END)
return parent_builder.compile()
async def _run_delegation_through_worker(executor_module, monkeypatch, *, stream_subgraphs: bool, child_emits_error_fallback: bool = False) -> tuple[RunRecord, _RecordingStreamBridge]:
from langgraph.checkpoint.memory import InMemorySaver
parent_graph = _build_delegating_parent_graph(executor_module, monkeypatch, child_emits_error_fallback=child_emits_error_fallback)
bridge = _RecordingStreamBridge()
record = RunRecord(
run_id=f"run-ns-int-{int(stream_subgraphs)}",
thread_id=_THREAD_ID,
assistant_id="lead-agent",
status=RunStatus.pending,
on_disconnect=DisconnectMode.cancel,
model_name=None,
)
record.abort_event = asyncio.Event()
await worker.run_agent(
bridge,
_IntegrationRunManager(record),
record,
ctx=worker.RunContext(checkpointer=InMemorySaver()),
agent_factory=lambda config: parent_graph,
graph_input={"messages": [HumanMessage(content="delegate to the subagent")]},
config={"configurable": {"thread_id": _THREAD_ID}},
stream_modes=["values", "messages-tuple", "custom"],
stream_subgraphs=stream_subgraphs,
)
return record, bridge
@pytest.mark.skipif(
not _LANGGRAPH_INHERITS_SUBGRAPH_NAMESPACE,
reason="delegated graphs stream as namespaced subgraphs only on LangGraph >= 1.2.6",
)
class TestWorkerSubgraphStreamIntegration:
@pytest.mark.asyncio
async def test_stream_subgraphs_publishes_delegated_frames_namespaced_never_bare(self, real_executor_module, monkeypatch):
record, bridge = await _run_delegation_through_worker(real_executor_module, monkeypatch, stream_subgraphs=True)
assert record.status == RunStatus.success, f"run failed: {bridge.published}"
events = bridge.published
bare_values = [payload for event, payload in events if event == "values"]
bare_messages = [payload for event, payload in events if event == "messages"]
namespaced_values = [(event, payload) for event, payload in events if event.startswith("values|")]
namespaced_messages = [(event, payload) for event, payload in events if event.startswith("messages|")]
# The #4399 takeover: a delegated values snapshot must never be
# published as bare "values" (SDK clients replace the thread view).
for payload in bare_values:
assert not (_collect_ids(payload) & _CHILD_MESSAGE_IDS), f"delegated messages leaked into a bare values frame: {payload}"
for payload in bare_messages:
assert not (_collect_ids(payload) & _CHILD_MESSAGE_IDS), f"delegated message chunk leaked into the bare messages stream: {payload}"
# The delegated frames must actually arrive — namespaced by LangGraph,
# not silently dropped (guards against a vacuous pass).
assert any(_collect_ids(payload) & _CHILD_MESSAGE_IDS for _event, payload in namespaced_values), f"expected namespaced delegated values frames, got events: {[event for event, _ in events]}"
assert any(_collect_ids(payload) & _CHILD_MESSAGE_IDS for _event, payload in namespaced_messages), f"expected namespaced delegated message chunks, got events: {[event for event, _ in events]}"
for event, _payload in namespaced_values + namespaced_messages:
segments = event.split("|")[1:]
assert segments and all(segments), f"namespaced event name has empty namespace segments: {event}"
# Root frames stay bare and intact.
assert any(_PARENT_FINAL_ID in _collect_ids(payload) for payload in bare_values)
custom_types = [payload.get("type") for event, payload in events if event == "custom" and isinstance(payload, dict)]
assert "task_started" in custom_types and "task_completed" in custom_types
@pytest.mark.asyncio
async def test_delegated_error_fallback_does_not_mark_the_parent_run_as_error(self, real_executor_module, monkeypatch):
record, bridge = await _run_delegation_through_worker(real_executor_module, monkeypatch, stream_subgraphs=True, child_emits_error_fallback=True)
# A delegated subagent's LLM error fallback is the executor's to map
# (task_failed); it must not decide the parent run's status.
assert record.status == RunStatus.success, "delegated error fallback leaked into the parent run status"
assert not [payload for event, payload in bridge.published if event == "error"]
# Non-vacuous: the marked child message really rode the stream —
# namespaced, where the root-only fallback detector must ignore it.
namespaced_payloads = [payload for event, payload in bridge.published if event.startswith(("values|", "messages|"))]
assert any("child-fallback-sentinel" in _collect_ids(payload) for payload in namespaced_payloads)
@pytest.mark.asyncio
async def test_without_stream_subgraphs_delegated_frames_stay_out_while_task_events_remain(self, real_executor_module, monkeypatch):
record, bridge = await _run_delegation_through_worker(real_executor_module, monkeypatch, stream_subgraphs=False)
assert record.status == RunStatus.success, f"run failed: {bridge.published}"
events = bridge.published
# No delegated frame of any mode reaches the parent stream...
for event, payload in events:
assert not (_collect_ids(payload) & _CHILD_MESSAGE_IDS), f"delegated messages leaked into event {event!r}: {payload}"
assert not [event for event, _payload in events if "|" in event]
# ...while the parent's own frames and the task_* progress contract
# (what the web frontend relies on instead of the flag) still hold.
bare_values = [payload for event, payload in events if event == "values"]
assert any(_PARENT_FINAL_ID in _collect_ids(payload) for payload in bare_values)
custom_types = [payload.get("type") for event, payload in events if event == "custom" and isinstance(payload, dict)]
assert "task_started" in custom_types and "task_completed" in custom_types

20
config.example.yaml
View File

@ -2124,9 +2124,25 @@ run_ownership:
# Authorization Configuration
# ============================================================================
# Fine-grained resource authorization (RBAC and beyond). Disabled by default;
# every authenticated user has access to all resources. Schema is present but
# inert; the built-in RBAC provider and runtime wiring arrive in subsequent phases.
# every authenticated user has access to all resources.
# See RFC: https://github.com/bytedance/deer-flow/issues/4063
#
# authorization:
# enabled: true
# fail_closed: true # block on provider error / unresolved identity
# default_role: user # applied when user_role is None; built-in RBAC requires this role below
# provider:
# use: deerflow.authz.rbac:RbacAuthorizationProvider
# config:
# # A known role with no `tools` policy is unrestricted for tools.
# # Define `tools` for every role whose tool access should be constrained.
# roles:
# admin:
# tools: {allow: "*"}
# user:
# tools: {allow: "*", deny: ["update_agent"]}
# guest:
# tools: {allow: ["web_search", "read_file"]}
authorization:
enabled: false

455
contracts/run_event_stream_contract.json Normal file
View File

@ -0,0 +1,455 @@
{
"version": 1,
"schema_dialect": "https://json-schema.org/draft/2020-12/schema",
"description": "The current DeerFlow run event stream contract. It freezes existing names and categories, describes producer payloads, and records compatibility rules without changing runtime behavior.",
"compatibility": {
"current_event_names": "frozen",
"consumer_rule": "Consumers must ignore unknown event types, unknown envelope fields, and unknown optional payload or metadata fields.",
"additive_changes": [
"add_event_type",
"add_optional_payload_field",
"add_optional_metadata_field",
"add_envelope_field"
],
"breaking_changes": [
"remove_event_type",
"rename_event_type",
"change_event_category",
"remove_required_field",
"change_required_field_type"
]
},
"legacy_event_aliases": [
{
"event_type": "ai_message",
"canonical_event_type": "llm.ai.response",
"status": "read-only compatibility",
"produced_by_current_runtime": false,
"compatibility_scope": "Category-based message projections and last-visible-AI store queries, including the /messages/page endpoint, recognize this historical name.",
"known_limitations": "The legacy /messages endpoint returns category=message rows but only enriches feedback for llm.ai.response.",
"notes": "New producers must use llm.ai.response."
}
],
"record_schema": {
"type": "object",
"required": [
"thread_id",
"run_id",
"seq",
"event_type",
"category",
"content",
"metadata",
"created_at"
],
"properties": {
"thread_id": {"type": "string"},
"run_id": {"type": "string"},
"seq": {"type": "integer", "minimum": 1},
"event_type": {"type": "string", "minLength": 1, "maxLength": 32},
"category": {"type": "string", "minLength": 1, "maxLength": 16},
"content": {"type": ["string", "object", "array", "number", "boolean", "null"]},
"metadata": {"type": "object"},
"created_at": {"type": "string", "format": "date-time"},
"user_id": {"type": ["string", "null"]}
},
"additionalProperties": true
},
"sequence": {
"scope": "thread_id",
"guarantee": "strictly increasing within a thread",
"jsonl_multi_process_limit": "JsonlRunEventStore only guarantees monotonic assignment within one process; use DbRunEventStore for shared multi-process writes."
},
"categories": {
"trace": "Execution evidence that is excluded from message projections.",
"message": "A candidate message projection. Server and frontend visibility filters still apply.",
"outputs": "Root graph completion output; not an authoritative run lifecycle status.",
"error": "Callback-observed run or chain failure evidence.",
"middleware": "Middleware state-change audit evidence.",
"context": "Identity-only evidence about effective hidden context.",
"subagent": "Subagent lifecycle and persisted step evidence.",
"workspace": "Run-scoped workspace and output file-change evidence."
},
"events": [
{
"event_type": "run.start",
"category": "trace",
"producer": "RunJournal.on_chain_start(parent_run_id=None)",
"content_schema": {
"type": "object",
"required": ["chain"],
"properties": {"chain": {"type": ["string", "null"]}},
"additionalProperties": true
},
"metadata_schema": {
"type": "object",
"required": ["caller"],
"properties": {"caller": {"type": "string"}},
"additionalProperties": true
}
},
{
"event_type": "run.end",
"category": "outputs",
"producer": "RunJournal.on_chain_end(parent_run_id=None)",
"content_schema": true,
"content_description": "Opaque root graph outputs as supplied by LangGraph. Values nested inside the output may have backend-dependent representations when they are not directly JSON serializable.",
"storage_semantics": {
"memory": "Retains the original Python container and nested values.",
"jsonl": "Persists JSON with json.dumps(default=str), so nested non-JSON values are restored as strings.",
"database": "Persists the top-level payload as JSON text with json.dumps(default=str), then restores the JSON container on read; nested non-JSON values remain strings."
},
"metadata_schema": {
"type": "object",
"required": ["status"],
"properties": {"status": {"const": "success"}},
"additionalProperties": true
}
},
{
"event_type": "run.error",
"category": "error",
"producer": "RunJournal.on_chain_error()",
"content_schema": {"type": "string"},
"metadata_schema": {
"type": "object",
"required": ["error_type"],
"properties": {"error_type": {"type": "string"}},
"additionalProperties": true
}
},
{
"event_type": "llm.human.input",
"category": "message",
"producer": "RunJournal.on_chat_model_start() for the first persisted lead-agent HumanMessage",
"content_schema": {
"type": "object",
"required": ["type", "content"],
"properties": {
"type": {"const": "human"},
"content": true,
"additional_kwargs": {"type": "object"}
},
"additionalProperties": true
},
"metadata_schema": {
"type": "object",
"required": ["caller"],
"properties": {"caller": {"const": "lead_agent"}},
"additionalProperties": true
}
},
{
"event_type": "llm.ai.response",
"category": "message",
"producer": "RunJournal.on_llm_end()",
"content_schema": {
"type": "object",
"required": ["type", "content", "tool_calls"],
"properties": {
"type": {"const": "ai"},
"content": true,
"tool_calls": {"type": "array"},
"additional_kwargs": {"type": "object"},
"response_metadata": {"type": "object"}
},
"additionalProperties": true
},
"metadata_schema": {
"type": "object",
"required": ["caller", "usage", "latency_ms", "llm_call_index"],
"properties": {
"caller": {"type": "string"},
"usage": {"type": "object"},
"latency_ms": {"type": ["integer", "null"], "minimum": 0},
"llm_call_index": {"type": "integer", "minimum": 1}
},
"additionalProperties": true
}
},
{
"event_type": "llm.tool.result",
"category": "message",
"producer": "RunJournal.on_tool_end() for ToolMessage or Command(update.messages[])",
"content_schema": {
"type": "object",
"required": ["type", "content"],
"properties": {
"type": {"type": "string"},
"content": true,
"tool_call_id": {"type": ["string", "null"]}
},
"additionalProperties": true
},
"metadata_schema": {"type": "object", "additionalProperties": true}
},
{
"event_type": "llm.error",
"category": "trace",
"producer": "RunJournal.on_llm_error()",
"content_schema": {"type": "string"},
"metadata_schema": {"type": "object", "additionalProperties": true}
},
{
"event_type": "context:memory",
"category": "context",
"producer": "RunJournal.record_memory_context() from DynamicContextMiddleware",
"content_schema": {
"type": "object",
"required": ["content_sha256"],
"properties": {
"content_sha256": {"type": "string", "pattern": "^[0-9a-f]{64}$"}
},
"additionalProperties": true
},
"metadata_schema": {"type": "object", "additionalProperties": true}
},
{
"event_type": "subagent.start",
"category": "subagent",
"producer": "subagent_run_event(task_started)",
"content_schema": {
"type": "object",
"required": ["task_id", "description"],
"properties": {
"task_id": {"type": "string", "minLength": 1},
"description": {"type": ["string", "null"]}
},
"additionalProperties": true
},
"metadata_schema": {
"type": "object",
"required": ["task_id"],
"properties": {"task_id": {"type": "string", "minLength": 1}},
"additionalProperties": true
}
},
{
"event_type": "subagent.step",
"category": "subagent",
"producer": "subagent_run_event(task_running)",
"content_schema": {
"type": "object",
"required": ["task_id", "message_index", "kind", "text", "truncated"],
"properties": {
"task_id": {"type": "string", "minLength": 1},
"message_index": {"type": "integer", "minimum": 0},
"kind": {"enum": ["ai", "tool"]},
"text": {"type": "string"},
"truncated": {"type": "boolean"},
"tool_calls": {
"type": "array",
"items": {
"type": "object",
"required": ["name", "args"],
"properties": {
"name": {"type": ["string", "null"]},
"args": true,
"args_truncated": {"type": "boolean"}
},
"additionalProperties": true
}
},
"tool_name": {"type": ["string", "null"]}
},
"allOf": [
{
"if": {"properties": {"kind": {"const": "ai"}}},
"then": {"required": ["tool_calls"]}
},
{
"if": {"properties": {"kind": {"const": "tool"}}},
"then": {"required": ["tool_name"]}
}
],
"additionalProperties": true
},
"metadata_schema": {
"type": "object",
"required": ["task_id", "message_index"],
"properties": {
"task_id": {"type": "string", "minLength": 1},
"message_index": {"type": "integer", "minimum": 0}
},
"additionalProperties": true
}
},
{
"event_type": "subagent.end",
"category": "subagent",
"producer": "subagent_run_event(task_completed|task_failed|task_cancelled|task_timed_out)",
"content_schema": {
"type": "object",
"required": ["task_id", "status"],
"properties": {
"task_id": {"type": "string", "minLength": 1},
"status": {"enum": ["completed", "failed", "cancelled", "timed_out"]},
"model_name": {"type": "string"},
"usage": {"type": "object"},
"result": {"type": "string"},
"result_truncated": {"type": "boolean"},
"error": {"type": "string"},
"error_truncated": {"type": "boolean"}
},
"additionalProperties": true
},
"metadata_schema": {
"type": "object",
"required": ["task_id"],
"properties": {"task_id": {"type": "string", "minLength": 1}},
"additionalProperties": true
}
},
{
"event_type": "workspace_changes",
"category": "workspace",
"producer": "workspace_changes.record_workspace_changes()",
"content_schema": {"type": "string"},
"metadata_schema": {
"type": "object",
"required": ["workspace_changes"],
"properties": {
"workspace_changes": {
"type": "object",
"required": ["version", "summary", "files", "limits"],
"properties": {
"version": {"const": 1},
"summary": {
"type": "object",
"required": ["created", "modified", "deleted", "symlink_created", "additions", "deletions", "truncated"],
"properties": {
"created": {"type": "integer", "minimum": 0},
"modified": {"type": "integer", "minimum": 0},
"deleted": {"type": "integer", "minimum": 0},
"symlink_created": {"type": "integer", "minimum": 0},
"additions": {"type": "integer", "minimum": 0},
"deletions": {"type": "integer", "minimum": 0},
"truncated": {"type": "boolean"}
},
"additionalProperties": true
},
"files": {
"type": "array",
"items": {
"type": "object",
"required": [
"path",
"root",
"status",
"binary",
"sensitive",
"size_before",
"size_after",
"sha256_before",
"sha256_after",
"diff",
"diff_truncated",
"diff_unavailable_reason",
"additions",
"deletions",
"symlink",
"symlink_target_before",
"symlink_target_after"
],
"properties": {
"path": {"type": "string"},
"root": {"type": "string"},
"status": {"enum": ["created", "modified", "deleted", "symlink_created"]},
"binary": {"type": "boolean"},
"sensitive": {"type": "boolean"},
"size_before": {"type": ["integer", "null"], "minimum": 0},
"size_after": {"type": ["integer", "null"], "minimum": 0},
"sha256_before": {"type": ["string", "null"]},
"sha256_after": {"type": ["string", "null"]},
"diff": {"type": "string"},
"diff_truncated": {"type": "boolean"},
"diff_unavailable_reason": {"type": ["string", "null"]},
"additions": {"type": "integer", "minimum": 0},
"deletions": {"type": "integer", "minimum": 0},
"symlink": {"type": "boolean"},
"symlink_target_before": {"type": ["string", "null"]},
"symlink_target_after": {"type": ["string", "null"]}
},
"additionalProperties": true
}
},
"limits": {
"type": "object",
"required": ["max_files", "max_scanned_files", "max_file_bytes_for_diff", "max_total_diff_bytes"],
"properties": {
"max_files": {"type": "integer", "minimum": 0},
"max_scanned_files": {"type": "integer", "minimum": 0},
"max_file_bytes_for_diff": {"type": "integer", "minimum": 0},
"max_total_diff_bytes": {"type": "integer", "minimum": 0}
},
"additionalProperties": true
}
},
"additionalProperties": true
}
},
"additionalProperties": true
}
}
],
"dynamic_event_patterns": [
{
"pattern": "middleware:{tag}",
"category": "middleware",
"producer": "RunJournal.record_middleware(tag, ...)",
"known_tags": ["guardrail", "safety_termination", "skill_activation", "skill_secrets"],
"event_type_schema": {
"type": "string",
"pattern": "^middleware:",
"minLength": 12,
"maxLength": 32
},
"tag_schema": {"type": "string", "minLength": 1, "maxLength": 21},
"content_schema": {
"type": "object",
"required": ["name", "hook", "action", "changes"],
"properties": {
"name": {"type": "string"},
"hook": {"type": "string"},
"action": {"type": "string"},
"changes": {"type": "object"}
},
"additionalProperties": true
},
"metadata_schema": {"type": "object", "additionalProperties": true}
}
],
"known_gaps": [
{
"id": "mixed-event-name-separators",
"status": "frozen for compatibility",
"notes": "Current dot, colon, and bare-word event names remain unchanged. Any future normalization requires a versioned migration or dual-write period."
},
{
"id": "tool-call-intent",
"status": "not first-class",
"notes": "Tool call requests are embedded in llm.ai.response content.tool_calls. llm.tool.result records returned messages, and a missing or timed-out result may leave no dedicated outcome event."
},
{
"id": "terminal-run-status",
"status": "split source of truth",
"notes": "run.end is only a root graph completion marker and always says success. RunRow.status is authoritative for success, error, interrupted, and timeout; worker loss may leave no terminal run event."
},
{
"id": "run-end-backend-serialization",
"status": "backend-dependent opaque payload",
"affected_event_types": ["run.end"],
"notes": "Memory retains nested Python values in root graph outputs, while JSONL and database persistence stringify nested values that are not directly JSON serializable. Consumers must not rely on backend-identical nested run.end content."
},
{
"id": "middleware-coverage",
"status": "partial",
"notes": "Loop detection and deferred-tool promotion do not currently emit middleware events."
},
{
"id": "run-scoped-observation-context",
"status": "manual wiring",
"notes": "Journal attribution, token accounting, and external tracing metadata are still attached manually at several LLM call sites."
}
]
}

40
docs/plans/2026-07-10-pluggable-authorization-implementation-notes.md
View File

@ -248,6 +248,46 @@ Phase 1 最低验证要求：
- **兼容性：** 只拒绝不符合 `AuthzRequest` / `filter_resources` 类型契约的运行时
输入；Phase 1A-2 仍未接入运行时，`authorization.enabled: false` 行为不变。
### 2026-07-22 — Phase 1B / 工具授权执行接入
- **背景：** Phase 1A 完成了 Principal 链路、RBAC provider 和 factory。
Phase 1B 将 provider 接入 Layer 1（组装时过滤）和 Layer 2（执行时拦截）。
- **决策：** `apply_tool_authorization()` 是 Layer 1 的统一入口，组合 provider
解析、Principal 构建和 `filter_tools_by_authorization` 过滤。disabled 时返回
原始工具和 `None`。
- **决策：** Layer 1 过滤在 `assemble_deferred_tools` 之前执行，覆盖三条路径：
lead agent（bootstrap + default）、subagent、embedded client。
- **决策：** Layer 2 通过 `GuardrailAuthorizationAdapter` 复用 `GuardrailMiddleware`，
authorization middleware 在显式 guardrail 之前（外层），两者独立运行。
- **决策：** Layer 1 和 Layer 2 尝试共享同一个 provider 实例（"resolve once per
build"）。subagent 通过 `_authz_provider` 属性传递。
- **决策：** Embedded client `_agent_config_key` 加入 `user_role` 和 `is_internal`
作为 cache key，角色变化时强制重建 agent。
- **兼容性：** `authorization.enabled: false` 时工具集合和执行决策完全不变。
- **延期：** Models/Skills/Sandbox 权限（Phase 2+）；route-level 迁移。
#### Phase 1B review 收口
- Layer 1 的候选集合必须包含本次 build 最终可能暴露给模型的全部业务工具；
`describe_skill` 和 memory tools 因此在授权过滤前加入，过滤后才进行 deferred
assembly。框架生成的 `tool_search` 仍是受已过滤 catalog 约束的基础设施工具。
- lead、bootstrap、native subagent 和 embedded client 都把 Layer 1 解析出的同一
provider 实例传给 Layer 2，禁止在 middleware 构建时再次解析 provider。
- `tool_search` 仅在当前 build 确实生成了 deferred catalog 时作为基础设施工具跳过
authorization adapter 的第二次 provider 调用；catalog 已由 Layer 1 过滤。显式配置的
guardrail 仍会检查它，没有 deferred setup 的普通同名工具也不获得豁免。
- 内置 RBAC provider 在解析时校验 `authorization.default_role` 属于已配置角色，配置
错误直接阻止 agent 构建，不再表现为难以诊断的空工具集合。
- `DeerFlowClient.stream()` 的调用方属于可信进程内边界，可通过关键字参数传入与
Gateway runtime context 相同的授权身份字段；这些字段同时进入真实执行 context。
agent cache key 使用完整 Principal（包括 user/channel/oauth/internal/attributes），
并深拷贝嵌套 attributes，防止调用方原地修改身份数据后复用旧工具集合。
- disabled 模式仍在 Layer 1 候选阶段包含 `describe_skill` / memory tools，但 deferred
assembly 后恢复原有顺序（业务工具、`tool_search`、late framework tools）。
- 回归测试必须经过真实 lead/bootstrap、subagent 和 embedded 组装函数，不能只测试
`filter_tools_by_authorization()` helper；同时断言被拒绝工具不在最终 bound tools 中，
且 Layer 2 收到的 provider 与 Layer 1 为同一对象。
### 新记录模板
```markdown

2
frontend/AGENTS.md
View File

@ -71,6 +71,8 @@ The frontend is a stateful chat application. Users create **threads** (conversat
5. TanStack Query manages server state; localStorage stores user settings
6. Components subscribe to thread state and render updates
Run duration is run-scoped UI metadata even though the compatibility field `additional_kwargs.turn_duration` is repeated on historical AI messages. `core/messages/run-duration.ts` folds those copies into one display anchored after the run's last visible message group. `MessageList` owns the temporary client-side duration for a just-completed live turn until authoritative history arrives. The duration is total run wall-clock time, not per-message reasoning time; reasoning disclosure and run activity/duration are rendered separately.
Composer drafts are tab-scoped browser state. `core/threads/composer-draft.ts` stores only text plus the selected slash-skill name in `sessionStorage`, keyed by user, agent, and logical conversation scope. New-chat pages pass the stable scope `"new"` because their runtime `threadId` is a fresh UUID on every reload; established conversations use their real thread ID. `InputBox` waits for enabled skills before restoring a skill chip, degrades a missing/disabled skill back to editable slash text, and clears the stored draft through `SendMessageOptions.onSent` only after the send passes the in-flight guard. Attachments, sidecar quotes, voice state, and polish undo state are not persisted.
Auth UI note: the login page's "keep me signed in" option submits only `remember_me` to the Gateway and may persist only the email address through `core/auth/remember-login.ts`. Passwords and tokens must never be stored in frontend storage; the `HttpOnly access_token` and readable `csrf_token` cookies remain Gateway-owned.

96
frontend/src/components/workspace/messages/message-list-item.tsx
View File

@ -10,7 +10,6 @@ import {
useCallback,
useMemo,
useState,
useEffect,
type ImgHTMLAttributes,
} from "react";
@ -24,6 +23,7 @@ import {
Reasoning,
ReasoningTrigger,
} from "@/components/ai-elements/reasoning";
import { Shimmer } from "@/components/ai-elements/shimmer";
import { Task, TaskTrigger } from "@/components/ai-elements/task";
import { Badge } from "@/components/ui/badge";
import {
@ -175,7 +175,6 @@ export function MessageListItem({
threadId,
artifactPaths = [],
showCopyButton = true,
turnStartTime,
}: {
className?: string;
message: Message;
@ -185,7 +184,6 @@ export function MessageListItem({
feedback?: FeedbackData | null;
runId?: string;
showCopyButton?: boolean;
turnStartTime?: number | null;
}) {
const isHuman = message.type === "human";
return (
@ -200,7 +198,6 @@ export function MessageListItem({
threadId={threadId}
artifactPaths={artifactPaths}
runId={runId}
turnStartTime={turnStartTime}
/>
{!isLoading && showCopyButton && (
<MessageToolbar
@ -275,8 +272,6 @@ function MessageImage({
);
}
const clientTurnDurations = new Map<string, number>();
function HumanMessageText({ content }: { content: string }) {
// `parseSlashSkillReference` is a pure regex gate (no data subscription), so
// the overwhelmingly common plain-text human message never subscribes to the
@ -320,7 +315,6 @@ function MessageContent_({
threadId,
artifactPaths,
runId,
turnStartTime,
}: {
className?: string;
message: Message;
@ -328,52 +322,18 @@ function MessageContent_({
threadId: string;
artifactPaths: readonly string[];
runId?: string;
turnStartTime?: number | null;
}) {
const { t } = useI18n();
const isHuman = message.type === "human";
const rawTurnDuration = message.additional_kwargs?.turn_duration as
| number
| undefined;
const [cachedDuration, setCachedDuration] = useState<number | undefined>(
() =>
message.id
? clientTurnDurations.get(`${threadId}:${message.id}`)
: undefined,
const getReasoningMessage = useCallback(
(isStreaming: boolean) =>
isStreaming ? (
<Shimmer duration={1}>{t.runDuration.reasoning}</Shimmer>
) : (
t.runDuration.reasoning
),
[t.runDuration.reasoning],
);
const turnDuration = rawTurnDuration ?? cachedDuration;
useEffect(() => {
if (rawTurnDuration !== undefined && message.id) {
clientTurnDurations.set(`${threadId}:${message.id}`, rawTurnDuration);
setCachedDuration(rawTurnDuration);
}
}, [rawTurnDuration, message.id, threadId]);
const handleDurationChange = useCallback(
(d: number | undefined) => {
if (d !== undefined && message.id) {
clientTurnDurations.set(`${threadId}:${message.id}`, d);
setCachedDuration(d);
}
},
[message.id, threadId],
);
useEffect(() => {
return () => {
for (const key of clientTurnDurations.keys()) {
if (key.startsWith(`${threadId}:`)) {
clientTurnDurations.delete(key);
}
}
};
}, [threadId]);
const [wasLoading, setWasLoading] = useState(isLoading);
useEffect(() => {
if (isLoading) setWasLoading(true);
}, [isLoading]);
const components = useMemo(
() => ({
img: (props: ImgHTMLAttributes<HTMLImageElement>) => (
@ -395,8 +355,11 @@ function MessageContent_({
const files = useMemo(() => {
const files = message.additional_kwargs?.files;
if (!Array.isArray(files) || files.length === 0) {
if (rawContent.includes("<uploaded_files>")) {
// If the content contains the <uploaded_files> tag, we return the parsed files from the content for backward compatibility.
if (
rawContent.includes("<current_uploads>") ||
rawContent.includes("<uploaded_files>")
) {
// If the content contains an upload context tag, we return the parsed files from the content for backward compatibility.
return parseUploadedFiles(rawContent);
}
return null;
@ -450,13 +413,8 @@ function MessageContent_({
if (!isHuman && reasoningContent && !rawContent) {
return (
<AIElementMessageContent className={className}>
<Reasoning
isStreaming={isLoading}
startTimeProp={turnStartTime}
duration={turnDuration}
onTurnDurationChange={handleDurationChange}
>
<ReasoningTrigger />
<Reasoning isStreaming={isLoading}>
<ReasoningTrigger getThinkingMessage={getReasoningMessage} />
<SafeReasoningContent>{reasoningContent}</SafeReasoningContent>
</Reasoning>
</AIElementMessageContent>
@ -495,20 +453,12 @@ function MessageContent_({
return (
<AIElementMessageContent className={className}>
{filesList}
{!isHuman &&
(!!reasoningContent || wasLoading || turnDuration !== undefined) && (
<Reasoning
isStreaming={isLoading}
startTimeProp={turnStartTime}
duration={turnDuration}
onTurnDurationChange={handleDurationChange}
>
<ReasoningTrigger hasContent={!!reasoningContent} />
{reasoningContent && (
<SafeReasoningContent>{reasoningContent}</SafeReasoningContent>
)}
</Reasoning>
)}
{reasoningContent && (
<Reasoning isStreaming={isLoading}>
<ReasoningTrigger getThinkingMessage={getReasoningMessage} />
<SafeReasoningContent>{reasoningContent}</SafeReasoningContent>
</Reasoning>
)}
<MarkdownContent
content={contentToDisplay}
isLoading={isLoading}

193
frontend/src/components/workspace/messages/message-list.tsx
View File

@ -15,6 +15,7 @@ import {
useRef,
useState,
type MouseEvent,
type ReactNode,
} from "react";
import { toast } from "sonner";
@ -23,10 +24,6 @@ import {
ConversationContent,
type ConversationProps,
} from "@/components/ai-elements/conversation";
import {
Reasoning,
ReasoningTrigger,
} from "@/components/ai-elements/reasoning";
import { Button } from "@/components/ui/button";
import type { FeedbackData } from "@/core/api/feedback";
import { extractArtifactsFromThread } from "@/core/artifacts/utils";
@ -38,6 +35,10 @@ import {
type HumanInputRequest,
type HumanInputResponse,
} from "@/core/messages/human-input";
import {
getMessageRunId,
getRunDurationDisplaysByGroupIndex,
} from "@/core/messages/run-duration";
import {
buildTokenDebugSteps,
type TokenDebugStep,
@ -89,6 +90,7 @@ import {
MessageTokenUsageDebugList,
MessageTokenUsageList,
} from "./message-token-usage";
import { RunActivity, RunDuration } from "./run-duration";
import { MessageListSkeleton } from "./skeleton";
import { SubtaskCard } from "./subtask-card";
@ -121,6 +123,16 @@ function sameMessageIdentity(previous: Message, next: Message) {
return previousKey !== null && previousKey === nextKey;
}
function sameRunDurationMetadata(previous: Message, next: Message) {
return (
getMessageRunId(previous) === getMessageRunId(next) &&
Object.is(
previous.additional_kwargs?.turn_duration,
next.additional_kwargs?.turn_duration,
)
);
}
function canReuseMessageGroup(
previous: ThreadMessageGroup | undefined,
next: ThreadMessageGroup,
@ -136,7 +148,8 @@ function canReuseMessageGroup(
return previous.messages.every(
(message, index) =>
next.messages[index] !== undefined &&
sameMessageIdentity(message, next.messages[index]),
sameMessageIdentity(message, next.messages[index]) &&
sameRunDurationMetadata(message, next.messages[index]),
);
}
@ -361,19 +374,50 @@ export function MessageList({
const sidecar = useMaybeSidecar();
const [selectionToolbar, setSelectionToolbar] =
useState<SelectionToolbarState | null>(null);
const [turnStartTime, setTurnStartTime] = useState<number | null>(null);
const messages = thread.messages;
const groupedMessages = useStableMessageGroups(messages, thread.isLoading);
const browserView = useMaybeBrowserView();
const pushBrowserFrame = browserView?.pushFrame;
const messageCount = messages.length;
// The backend exposes no live start timestamp, so a mid-run mount measures
// from mount until authoritative persisted turn_duration replaces it.
const [turnStartTime, setTurnStartTime] = useState<number | null>(() =>
thread.isLoading ? Date.now() : null,
);
const turnStartTimeRef = useRef(turnStartTime);
const [clientDurationsByGroupId, setClientDurationsByGroupId] = useState<
ReadonlyMap<string, number>
>(() => new Map());
const prevIsLoading = useRef(thread.isLoading);
useEffect(() => {
if (thread.isLoading && !prevIsLoading.current) {
setTurnStartTime(Date.now());
const now = Date.now();
turnStartTimeRef.current = now;
setTurnStartTime(now);
} else if (!thread.isLoading && prevIsLoading.current) {
const startTime = turnStartTimeRef.current;
const lastAssistantGroup = [...groupedMessages]
.reverse()
.find((group) => group.type !== "human" && group.id);
if (startTime !== null && lastAssistantGroup?.id && !thread.error) {
const duration = Math.max(
0,
Math.floor((Date.now() - startTime) / 1000),
);
const key = `${threadId}:${lastAssistantGroup.id}`;
setClientDurationsByGroupId((current) => {
const next = new Map(current);
next.set(key, duration);
return next;
});
}
turnStartTimeRef.current = null;
setTurnStartTime(null);
}
prevIsLoading.current = thread.isLoading;
}, [thread.isLoading]);
const messages = thread.messages;
const browserView = useMaybeBrowserView();
const pushBrowserFrame = browserView?.pushFrame;
const messageCount = messages.length;
}, [groupedMessages, thread.error, thread.isLoading, threadId]);
useEffect(() => {
// Only the primary chat surface drives the shared browser panel. The
// sidecar renders a different thread's messages against the same
@ -411,7 +455,6 @@ export function MessageList({
// repeatedly scan long history looking for the last browser frame.
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [messageCount, pushBrowserFrame, sidecarSurface]);
const groupedMessages = useStableMessageGroups(messages, thread.isLoading);
const [regeneratingMessageId, setRegeneratingMessageId] = useState<
string | null
>(null);
@ -438,6 +481,28 @@ export function MessageList({
const lastGroupIndex = groupedMessages.length - 1;
const turnUsageMessagesByGroupIndex =
getAssistantTurnUsageMessages(groupedMessages);
const runDurationDisplaysByGroupIndex = useMemo(
() => getRunDurationDisplaysByGroupIndex(groupedMessages),
[groupedMessages],
);
useEffect(() => {
setClientDurationsByGroupId((current) => {
let next: Map<string, number> | undefined;
runDurationDisplaysByGroupIndex.forEach((displays, groupIndex) => {
const groupId = groupedMessages[groupIndex]?.id;
if (displays.length === 0 || !groupId) {
return;
}
const key = `${threadId}:${groupId}`;
if (!current.has(key)) {
return;
}
next ??= new Map(current);
next.delete(key);
});
return next ?? current;
});
}, [groupedMessages, runDurationDisplaysByGroupIndex, threadId]);
const tokenDebugSteps = useMemo(
() =>
tokenUsageInlineMode === "step_debug"
@ -904,6 +969,47 @@ export function MessageList({
return <MessageListSkeleton />;
}
const withRunDuration = (
group: (typeof groupedMessages)[number],
groupIndex: number,
content: ReactNode,
) => {
const persistedDisplays = runDurationDisplaysByGroupIndex[groupIndex] ?? [];
const clientDuration =
!thread.error && group.id
? clientDurationsByGroupId.get(`${threadId}:${group.id}`)
: undefined;
const displays =
persistedDisplays.length > 0
? persistedDisplays
: clientDuration !== undefined
? [
{
runId: `client:${group.id}`,
durationSeconds: clientDuration,
},
]
: [];
if (!content && displays.length === 0) {
return null;
}
return (
<div
key={`duration-group:${group.id ?? groupIndex}`}
className="flex w-full flex-col gap-2"
>
{content}
{displays.map((display) => (
<RunDuration
key={display.runId}
durationSeconds={display.durationSeconds}
/>
))}
</div>
);
};
return (
<>
<Conversation
@ -924,9 +1030,10 @@ export function MessageList({
thread.isLoading && groupIndex === lastGroupIndex;
if (group.type === "human" || group.type === "assistant") {
return (
return withRunDuration(
group,
groupIndex,
<div
key={group.id}
data-assistant-turn={
group.type === "assistant" ? "" : undefined
}
@ -951,11 +1058,6 @@ export function MessageList({
: undefined
}
showCopyButton={group.type !== "assistant"}
turnStartTime={
groupIndex === groupedMessages.length - 1
? turnStartTime
: null
}
/>
);
@ -997,7 +1099,7 @@ export function MessageList({
branchableAssistantGroupIds.has(group.id),
group.id === latestAssistantGroupId,
)}
</div>
</div>,
);
} else if (group.type === "assistant:clarification") {
const message = group.messages[0];
@ -1014,8 +1116,10 @@ export function MessageList({
const pending = pendingHumanInputRequestIds.has(
humanInputRequest.request_id,
);
return (
<div key={group.id} className="w-full">
return withRunDuration(
group,
groupIndex,
<div className="w-full">
<HumanInputCard
answeredResponse={answeredResponse}
disabled={
@ -1042,13 +1146,15 @@ export function MessageList({
messages: group.messages,
turnUsageMessages,
})}
</div>
</div>,
);
}
if (hasContent(message)) {
return (
<div key={group.id} className="w-full">
return withRunDuration(
group,
groupIndex,
<div className="w-full">
<MarkdownContent
content={extractContentFromMessage(message)}
isLoading={thread.isLoading}
@ -1057,10 +1163,10 @@ export function MessageList({
messages: group.messages,
turnUsageMessages,
})}
</div>
</div>,
);
}
return null;
return withRunDuration(group, groupIndex, null);
} else if (group.type === "assistant:present-files") {
const files: string[] = [];
for (const message of group.messages) {
@ -1069,8 +1175,10 @@ export function MessageList({
files.push(...presentFiles);
}
}
return (
<div className="w-full" key={group.id}>
return withRunDuration(
group,
groupIndex,
<div className="w-full">
{group.messages[0] && hasContent(group.messages[0]) && (
<MarkdownContent
content={extractContentFromMessage(group.messages[0])}
@ -1083,7 +1191,7 @@ export function MessageList({
messages: group.messages,
turnUsageMessages,
})}
</div>
</div>,
);
} else if (group.type === "assistant:subagent") {
const tasks = new Set<Subtask>();
@ -1170,22 +1278,23 @@ export function MessageList({
);
}
}
return (
<div
key={"subtask-group-" + group.id}
className="relative z-1 flex flex-col gap-2"
>
return withRunDuration(
group,
groupIndex,
<div className="relative z-1 flex flex-col gap-2">
{results}
{renderTokenUsage({
messages: group.messages,
turnUsageMessages,
debugMessageIds: subagentDebugMessageIds,
})}
</div>
</div>,
);
}
return (
<div key={"group-" + group.id} className="w-full">
return withRunDuration(
group,
groupIndex,
<div className="w-full">
<MessageGroup
messages={group.messages}
isLoading={groupIsLoading}
@ -1201,14 +1310,12 @@ export function MessageList({
turnUsageMessages,
inlineDebug: false,
})}
</div>
</div>,
);
})}
{thread.isLoading && !hasActiveAssistantText && (
<div className="w-full">
<Reasoning isStreaming={true} startTimeProp={turnStartTime}>
<ReasoningTrigger hasContent={false} />
</Reasoning>
<RunActivity startTime={turnStartTime} />
</div>
)}
<div style={{ height: `${paddingBottom}px` }} />

59
frontend/src/components/workspace/messages/run-duration.tsx Normal file
View File

@ -0,0 +1,59 @@
"use client";
import { Clock3Icon } from "lucide-react";
import { useEffect, useState } from "react";
import { Shimmer } from "@/components/ai-elements/shimmer";
import { useI18n } from "@/core/i18n/hooks";
import { formatRunDuration } from "@/core/messages/run-duration";
export function RunActivity({ startTime }: { startTime: number | null }) {
const { t } = useI18n();
const [elapsed, setElapsed] = useState(0);
useEffect(() => {
if (startTime === null) {
setElapsed(0);
return;
}
const updateElapsed = () => {
setElapsed(Math.max(0, Math.floor((Date.now() - startTime) / 1000)));
};
updateElapsed();
const interval = setInterval(updateElapsed, 1000);
return () => clearInterval(interval);
}, [startTime]);
const formatted = formatRunDuration(elapsed, t.runDuration);
return (
<div
className="text-muted-foreground flex items-center gap-2 text-sm"
data-testid="run-activity"
>
<Clock3Icon className="size-4" />
<Shimmer duration={1}>{t.runDuration.working}</Shimmer>
{formatted && <span aria-hidden="true">({formatted})</span>}
</div>
);
}
export function RunDuration({ durationSeconds }: { durationSeconds: number }) {
const { t } = useI18n();
const formatted = formatRunDuration(durationSeconds, t.runDuration);
if (!formatted) {
return null;
}
return (
<div
className="text-muted-foreground flex items-center gap-2 text-sm"
data-testid="run-duration"
title={t.runDuration.description}
>
<Clock3Icon className="size-4" />
<span>{t.runDuration.completedIn(formatted)}</span>
</div>
);
}

13
frontend/src/core/i18n/locales/en-US.ts
View File

@ -72,6 +72,19 @@ export const enUS: Translations = {
showBrowser: "Open browser panel",
},
runDuration: {
reasoning: "Reasoning",
working: "Working…",
completedIn: (duration) => `Completed in ${duration}`,
description:
"Total task time, including model reasoning, tool calls, and waiting.",
lessThanSecond: "<1s",
hours: (value) => `${value}h`,
minutes: (value) => `${value}m`,
seconds: (value) => `${value}s`,
separator: " ",
},
// Home
home: {
docs: "Docs",

12
frontend/src/core/i18n/locales/types.ts
View File

@ -61,6 +61,18 @@ export interface Translations {
showBrowser: string;
};
runDuration: {
reasoning: string;
working: string;
completedIn: (duration: string) => string;
description: string;
lessThanSecond: string;
hours: (value: number) => string;
minutes: (value: number) => string;
seconds: (value: number) => string;
separator: string;
};
home: {
docs: string;
blog: string;

12
frontend/src/core/i18n/locales/zh-CN.ts
View File

@ -72,6 +72,18 @@ export const zhCN: Translations = {
showBrowser: "打开浏览器面板",
},
runDuration: {
reasoning: "思考过程",
working: "执行中…",
completedIn: (duration) => `本次任务耗时 ${duration}`,
description: "任务总耗时，包括模型推理、工具调用和等待时间。",
lessThanSecond: "不足 1 秒",
hours: (value) => `${value} 小时`,
minutes: (value) => `${value} 分`,
seconds: (value) => `${value} 秒`,
separator: " ",
},
// Home
home: {
docs: "文档",

104
frontend/src/core/messages/run-duration.ts Normal file
View File

@ -0,0 +1,104 @@
import type { Message } from "@langchain/langgraph-sdk";
import type { MessageGroup } from "./utils";
export interface RunDurationDisplay {
runId: string;
durationSeconds: number;
}
export interface RunDurationFormatter {
lessThanSecond: string;
hours: (value: number) => string;
minutes: (value: number) => string;
seconds: (value: number) => string;
separator: string;
}
type MessageWithRunId = Message & { run_id?: unknown };
export function getMessageRunId(message: Message): string | undefined {
const runId = (message as MessageWithRunId).run_id;
return typeof runId === "string" && runId.length > 0 ? runId : undefined;
}
function normalizeDuration(value: unknown): number | undefined {
if (typeof value !== "number" || !Number.isFinite(value) || value < 0) {
return undefined;
}
return Math.floor(value);
}
/**
* Locate the single UI position that owns each completed run's wall-clock
* duration. The backend keeps the value on every AI message for compatibility,
* but the UI treats it as run-scoped metadata and renders it after the last
* visible group belonging to that run.
*/
export function getRunDurationDisplaysByGroupIndex(
groups: MessageGroup[],
): RunDurationDisplay[][] {
const displays = groups.map(() => [] as RunDurationDisplay[]);
const durationByRunId = new Map<string, number>();
const lastGroupIndexByRunId = new Map<string, number>();
groups.forEach((group, groupIndex) => {
for (const message of group.messages) {
const runId = getMessageRunId(message);
if (!runId) {
continue;
}
lastGroupIndexByRunId.set(runId, groupIndex);
if (message.type !== "ai") {
continue;
}
const duration = normalizeDuration(
message.additional_kwargs?.turn_duration,
);
if (duration !== undefined) {
durationByRunId.set(runId, duration);
}
}
});
for (const [runId, durationSeconds] of durationByRunId) {
const groupIndex = lastGroupIndexByRunId.get(runId);
if (groupIndex !== undefined) {
displays[groupIndex]?.push({ runId, durationSeconds });
}
}
return displays;
}
export function formatRunDuration(
value: number,
formatter: RunDurationFormatter,
): string | null {
const duration = normalizeDuration(value);
if (duration === undefined) {
return null;
}
if (duration === 0) {
return formatter.lessThanSecond;
}
const hours = Math.floor(duration / 3600);
const minutes = Math.floor((duration % 3600) / 60);
const seconds = duration % 60;
const parts: string[] = [];
if (hours > 0) {
parts.push(formatter.hours(hours));
}
if (minutes > 0) {
parts.push(formatter.minutes(minutes));
}
if (seconds > 0) {
parts.push(formatter.seconds(seconds));
}
return parts.join(formatter.separator);
}

68
frontend/src/core/messages/utils.ts
View File

@ -92,13 +92,27 @@ export function getMessageGroups(messages: Message[]): MessageGroup[] {
if (lastGroup) {
lastGroup.messages.push(message);
} else {
// groups is empty (shouldn't happen — the outer for loop is guarded
// by `messages.length === 0 -> return []`), but keep the diagnostic
// just in case.
console.error(
"Unexpected tool message with no preceding group",
message,
);
// Leading orphan: `groups` is empty when this tool message
// arrives. Two paths reach here: (1) history pagination cuts by
// event seq, not turn boundaries, so the first loaded page begins
// mid-turn with a tool result whose AI tool-call sits on an
// unloaded older page (#4399); (2) the tool message is preceded
// only by hidden control messages. Open a processing group so it
// stays visible instead of being dropped with a per-render console
// error.
//
// Only case (1) self-heals — loading the older page re-groups the
// tool under its real turn. Case (2), and any truly orphaned tool
// with no AI antecedent, has no page to load: the group persists
// and renders as an empty ChainOfThought shell (convertToSteps
// emits steps only for `type === "ai"`). That empty shell is an
// accepted degradation — still a net win over dropping the result
// and firing console.error every render.
groups.push({
id: message.id,
type: "assistant:processing",
messages: [message],
});
}
}
}
@ -582,7 +596,10 @@ export interface FileInMessage {
*/
export function stripUploadedFilesTag(content: string): string {
return content
.replace(/<(uploaded_files|slash_skill_activation)>[\s\S]*?<\/\1>/g, "")
.replace(
/<(current_uploads|uploaded_files|slash_skill_activation)>[\s\S]*?<\/\1>/g,
"",
)
.trim();
}
@ -592,7 +609,8 @@ export function stripUploadedFilesTag(content: string): string {
*
* These markers are *not* user copy — they come from:
*
* - ``UploadsMiddleware`` → ``<uploaded_files>``
* - ``UploadsMiddleware`` → ``<current_uploads>`` (``<uploaded_files>``
* before #4174; still emitted by IM channels and present in history)
* - ``SkillActivationMiddleware`` → ``<slash_skill_activation>``
* - ``DynamicContextMiddleware`` → ``<system-reminder>`` (carrying
* ``<memory>`` / ``<current_date>`` inside)
@ -606,6 +624,7 @@ export function stripUploadedFilesTag(content: string): string {
* its ``hide_from_ui`` flag set.
*/
export const INTERNAL_MARKER_TAGS = [
"current_uploads",
"uploaded_files",
"slash_skill_activation",
"system-reminder",
@ -632,9 +651,32 @@ export function stripInternalMarkers(content: string): string {
return content.replace(INTERNAL_MARKER_RE, "").trim();
}
// The upload context block renders sizes as human-readable strings
// (uploads_middleware.py::_format_file_entry emits "<n> KB" / "<n> MB",
// mirroring formatBytes). Convert them back to bytes so the parsed
// FileInMessage.size honours its bytes contract and chips re-render at the
// original magnitude instead of e.g. treating "177.6 KB" as 177 bytes.
function parseHumanReadableSize(raw: string): number {
const match = /([\d.]+)\s*(B|KB|MB|GB|TB)?/i.exec(raw.trim());
if (!match) return 0;
const value = parseFloat(match[1] ?? "");
if (!Number.isFinite(value)) return 0;
const multipliers: Record<string, number> = {
B: 1,
KB: 1024,
MB: 1024 ** 2,
GB: 1024 ** 3,
TB: 1024 ** 4,
};
const unit = (match[2] ?? "B").toUpperCase();
return Math.round(value * (multipliers[unit] ?? 1));
}
export function parseUploadedFiles(content: string): FileInMessage[] {
// Match <uploaded_files>...</uploaded_files> tag
const uploadedFilesRegex = /<uploaded_files>([\s\S]*?)<\/uploaded_files>/;
// Match the upload context block; the tag name depends on backend version
// (<current_uploads> since #4174, <uploaded_files> before / on IM paths).
const uploadedFilesRegex =
/<(current_uploads|uploaded_files)>([\s\S]*?)<\/\1>/;
// eslint-disable-next-line @typescript-eslint/prefer-regexp-exec
const match = content.match(uploadedFilesRegex);
@ -642,7 +684,7 @@ export function parseUploadedFiles(content: string): FileInMessage[] {
return [];
}
const uploadedFilesContent = match[1];
const uploadedFilesContent = match[2];
// Check if it's "No files have been uploaded yet."
if (uploadedFilesContent?.includes("No files have been uploaded yet.")) {
@ -663,7 +705,7 @@ export function parseUploadedFiles(content: string): FileInMessage[] {
while ((fileMatch = fileRegex.exec(uploadedFilesContent ?? "")) !== null) {
files.push({
filename: fileMatch[1].trim(),
size: parseInt(fileMatch[2].trim(), 10) ?? 0,
size: parseHumanReadableSize(fileMatch[2]),
path: fileMatch[3].trim(),
});
}

37
frontend/src/core/threads/hooks.ts
View File

@ -18,6 +18,7 @@ import { getAPIClient } from "../api";
import { fetch } from "../api/fetcher";
import { getBackendBaseURL } from "../config";
import { useI18n } from "../i18n/hooks";
import { getMessageRunId } from "../messages/run-duration";
import { isHiddenFromUIMessage } from "../messages/utils";
import type { FileInMessage } from "../messages/utils";
import type { LocalSettings } from "../settings";
@ -319,8 +320,13 @@ export function mergeMessages(
optimisticMessages: Message[],
): Message[] {
const savedTurnDurations = new Map<string, number>();
const savedRunIds = new Map<string, string>();
for (const msg of historyMessages) {
const identity = messageIdentity(msg);
const runId = getMessageRunId(msg);
if (identity && runId) {
savedRunIds.set(identity, runId);
}
if (identity && msg.additional_kwargs?.turn_duration !== undefined) {
savedTurnDurations.set(
identity,
@ -415,17 +421,26 @@ export function mergeMessages(
return merged.map((message) => {
const identity = messageIdentity(message);
if (
identity &&
if (!identity) {
return message;
}
const shouldRestoreRunId =
savedRunIds.has(identity) && !getMessageRunId(message);
const shouldRestoreTurnDuration =
savedTurnDurations.has(identity) &&
message.additional_kwargs?.turn_duration === undefined
) {
message.additional_kwargs?.turn_duration === undefined;
if (shouldRestoreRunId || shouldRestoreTurnDuration) {
return {
...message,
additional_kwargs: {
...message.additional_kwargs,
turn_duration: savedTurnDurations.get(identity),
},
...(shouldRestoreRunId ? { run_id: savedRunIds.get(identity) } : {}),
...(shouldRestoreTurnDuration
? {
additional_kwargs: {
...message.additional_kwargs,
turn_duration: savedTurnDurations.get(identity),
},
}
: {}),
} as Message;
}
return message;
@ -1525,7 +1540,9 @@ export function useThreadStream({
},
{
threadId: threadId,
streamSubgraphs: true,
// No streamSubgraphs: subtask progress arrives via root-namespace
// custom events, while subgraph frames would leak a delegated
// subagent's values/messages into the thread view (#4399).
streamResumable: true,
config: {
recursion_limit: 1000,
@ -1631,7 +1648,7 @@ export function useThreadStream({
threadId,
checkpoint: prepared.checkpoint,
metadata: prepared.metadata,
streamSubgraphs: true,
// No streamSubgraphs — same contract as the main submit path (#4399).
streamResumable: true,
config: {
recursion_limit: 1000,

48
frontend/tests/e2e/thread-history.spec.ts
View File

@ -94,6 +94,54 @@ test.describe("Thread history", () => {
).toBeVisible({ timeout: 15_000 });
});
test("shows a completed run duration once after multi-step history", async ({
page,
}) => {
mockLangGraphAPI(page, {
threads: [
{
thread_id: MOCK_THREAD_ID,
title: "Multi-step duration",
updated_at: "2025-06-03T12:00:00Z",
messages: [
{
type: "human",
id: "msg-human-duration",
content: [{ type: "text", text: "Complete several steps" }],
},
{
type: "ai",
id: "msg-ai-duration-1",
content: "Intermediate result",
additional_kwargs: { turn_duration: 114 },
},
{
type: "ai",
id: "msg-ai-duration-2",
content: "Final result",
additional_kwargs: {
turn_duration: 114,
reasoning_content: "Final synthesis reasoning",
},
},
],
},
],
});
await page.goto(`/workspace/chats/${MOCK_THREAD_ID}`);
await expect(page.getByText("Final result")).toBeVisible({
timeout: 15_000,
});
await expect(page.getByTestId("run-duration")).toHaveCount(1);
await expect(page.getByText("Completed in 1m 54s")).toBeVisible();
await expect(
page.getByRole("button", { name: "Reasoning", exact: true }),
).toBeVisible();
await expect(page.getByText("Thought for 114 seconds")).toHaveCount(0);
});
test("input box recalls previous prompts with arrow keys", async ({
page,
}) => {

138
frontend/tests/unit/core/messages/run-duration.test.ts Normal file
View File

@ -0,0 +1,138 @@
import type { Message } from "@langchain/langgraph-sdk";
import { describe, expect, test } from "@rstest/core";
import { enUS } from "@/core/i18n/locales/en-US";
import { zhCN } from "@/core/i18n/locales/zh-CN";
import {
formatRunDuration,
getRunDurationDisplaysByGroupIndex,
} from "@/core/messages/run-duration";
import { getMessageGroups } from "@/core/messages/utils";
function message(
id: string,
type: Message["type"],
content: string,
runId?: string,
duration?: unknown,
): Message {
return {
id,
type,
content,
...(runId ? { run_id: runId } : {}),
...(type === "ai" && duration !== undefined
? { additional_kwargs: { turn_duration: duration } }
: {}),
} as Message;
}
describe("run duration display placement", () => {
test("shows one duration after the final visible group for a multi-step run", () => {
const groups = getMessageGroups([
message("human-1", "human", "Research this"),
{
...message("ai-tool-1", "ai", "", "run-1", 114),
tool_calls: [{ id: "call-1", name: "read_file", args: {} }],
} as Message,
{
...message("tool-1", "tool", "file contents", "run-1"),
tool_call_id: "call-1",
} as Message,
message("ai-middle", "ai", "Intermediate summary", "run-1", 114),
{
...message("ai-tool-2", "ai", "", "run-1", 114),
tool_calls: [{ id: "call-2", name: "write_todos", args: {} }],
} as Message,
{
...message("tool-2", "tool", "todos updated", "run-1"),
tool_call_id: "call-2",
} as Message,
message("ai-final", "ai", "Final answer", "run-1", 114),
]);
expect(
getRunDurationDisplaysByGroupIndex(groups).map((displays) =>
displays.map(({ runId, durationSeconds }) => ({
runId,
durationSeconds,
})),
),
).toEqual([[], [], [], [], [{ runId: "run-1", durationSeconds: 114 }]]);
});
test("keeps equal durations independent across runs", () => {
const groups = getMessageGroups([
message("human-1", "human", "First"),
message("ai-1", "ai", "First answer", "run-1", 114),
message("human-2", "human", "Second"),
message("ai-2", "ai", "Second answer", "run-2", 114),
]);
expect(getRunDurationDisplaysByGroupIndex(groups)).toEqual([
[],
[{ runId: "run-1", durationSeconds: 114 }],
[],
[{ runId: "run-2", durationSeconds: 114 }],
]);
});
test("carries an earlier AI duration to the run's final tool-bearing group", () => {
const groups = getMessageGroups([
message("human-1", "human", "Do it"),
message("ai-answer", "ai", "Initial answer", "run-1", 9),
{
...message("ai-tool", "ai", "", "run-1"),
tool_calls: [{ id: "call-1", name: "write_todos", args: {} }],
} as Message,
{
...message("tool-1", "tool", "done", "run-1"),
tool_call_id: "call-1",
} as Message,
]);
expect(getRunDurationDisplaysByGroupIndex(groups)).toEqual([
[],
[],
[{ runId: "run-1", durationSeconds: 9 }],
]);
});
test("keeps zero but ignores missing, negative, and non-finite durations", () => {
const groups = getMessageGroups([
message("ai-zero", "ai", "Zero", "run-zero", 0),
message("ai-missing", "ai", "Missing", "run-missing"),
message("ai-negative", "ai", "Negative", "run-negative", -1),
message("ai-infinite", "ai", "Infinite", "run-infinite", Infinity),
]);
expect(getRunDurationDisplaysByGroupIndex(groups)).toEqual([
[{ runId: "run-zero", durationSeconds: 0 }],
[],
[],
[],
]);
});
});
describe("run duration formatting", () => {
test("formats sub-second, second, minute, and hour durations in English", () => {
expect(formatRunDuration(0, enUS.runDuration)).toBe("<1s");
expect(formatRunDuration(59, enUS.runDuration)).toBe("59s");
expect(formatRunDuration(114, enUS.runDuration)).toBe("1m 54s");
expect(formatRunDuration(3723, enUS.runDuration)).toBe("1h 2m 3s");
});
test("formats durations in Chinese", () => {
expect(formatRunDuration(0, zhCN.runDuration)).toBe("不足 1 秒");
expect(formatRunDuration(114, zhCN.runDuration)).toBe("1 分 54 秒");
expect(formatRunDuration(3723, zhCN.runDuration)).toBe("1 小时 2 分 3 秒");
});
test("rejects invalid durations and floors fractional seconds", () => {
expect(formatRunDuration(-1, enUS.runDuration)).toBeNull();
expect(formatRunDuration(Infinity, enUS.runDuration)).toBeNull();
expect(formatRunDuration(Number.NaN, enUS.runDuration)).toBeNull();
expect(formatRunDuration(61.9, enUS.runDuration)).toBe("1m 1s");
});
});

142
frontend/tests/unit/core/messages/utils.test.ts
View File

@ -14,6 +14,8 @@ import {
hasContent,
hasReasoning,
isAssistantMessageGroupStreaming,
parseUploadedFiles,
stripInternalMarkers,
stripUploadedFilesTag,
} from "@/core/messages/utils";
@ -307,6 +309,47 @@ describe("human message internal context stripping", () => {
expect(getMessageCopyData(message)).toBe("Summarize this paper");
});
test("strips current_uploads context from copy data", () => {
// Mirrors the block UploadsMiddleware emits since #4174, including the
// trailing usage-guidance lines.
const message = {
id: "human-with-current-uploads",
type: "human",
content:
"<current_uploads>\nThe following files were uploaded in this message:\n\n- paper.docx (177.6 KB)\n Path: /mnt/user-data/uploads/paper.docx\n\nTo work with these files:\n- Use `grep` to search for keywords\n (e.g. `grep(pattern='revenue', path='/mnt/user-data/uploads/')`).\n</current_uploads>\n\nMake a slide deck from this",
} as Message;
expect(getMessageCopyData(message)).toBe("Make a slide deck from this");
});
test("parses uploaded files from a current_uploads block", () => {
const content =
"<current_uploads>\nThe following files were uploaded in this message:\n\n- paper.docx (177.6 KB)\n Path: /mnt/user-data/uploads/paper.docx\n Document outline (use `read_file` with line ranges to read sections):\n L1: Introduction\n- data.xlsx (12.0 KB)\n Path: /mnt/user-data/uploads/data.xlsx\n</current_uploads>\n\nSummarize";
// size is bytes (FileInMessage contract): the block's "177.6 KB" /
// "12.0 KB" are converted back from the human-readable form the backend
// emits, so formatBytes re-renders them at the original magnitude.
expect(parseUploadedFiles(content)).toEqual([
{
filename: "paper.docx",
size: Math.round(177.6 * 1024), // 181862
path: "/mnt/user-data/uploads/paper.docx",
},
{
filename: "data.xlsx",
size: 12 * 1024, // 12288
path: "/mnt/user-data/uploads/data.xlsx",
},
]);
});
test("stripInternalMarkers removes current_uploads blocks on export", () => {
const content =
"<current_uploads>\n- paper.docx (177.6 KB)\n Path: /mnt/user-data/uploads/paper.docx\n</current_uploads>\n\nExport me";
expect(stripInternalMarkers(content)).toBe("Export me");
});
test("strips slash skill activation context from display content", () => {
const content =
"<slash_skill_activation>\n<skill_content># Secret SKILL.md</skill_content>\n</slash_skill_activation>\nreal user task";
@ -724,6 +767,105 @@ describe("orphan tool messages", () => {
expect(t2?.type).toBe("tool");
});
test("opens a processing group for a leading orphan tool message", () => {
// History pagination cuts by event seq, not turn boundaries, so the first
// loaded page can begin mid-turn with tool results whose AI tool-call
// message sits on an unloaded older page (#4399). They must stay visible
// in a processing group, not be dropped with a console error per render.
const messages = [
{
id: "t-lead-1",
type: "tool",
name: "bash",
tool_call_id: "call-old-1",
content: "output from unloaded turn",
},
{
id: "t-lead-2",
type: "tool",
name: "bash",
tool_call_id: "call-old-2",
content: "second output",
},
{ id: "ai-1", type: "ai", content: "Done." },
] as Message[];
const groups = getMessageGroups(messages);
expect(groups.map((g) => g.type)).toEqual([
"assistant:processing",
"assistant",
]);
// Both leading orphans cluster into the same processing group.
expect(groups[0]?.messages.map((m) => m.id)).toEqual([
"t-lead-1",
"t-lead-2",
]);
});
test("leading orphan absorbs the following real AI turn into one processing group", () => {
// A leading orphan followed by a real tool-call turn must accumulate into
// a single processing group via the assistant:processing branch, not
// strand the orphan as a separate empty group beside the real turn.
const messages = [
{
id: "t-lead",
type: "tool",
name: "bash",
tool_call_id: "call-old",
content: "output from unloaded turn",
},
{
id: "ai-1",
type: "ai",
content: "running",
tool_calls: [{ id: "call-1", name: "bash", args: {} }],
},
{
id: "t-1",
type: "tool",
name: "bash",
tool_call_id: "call-1",
content: "output-1",
},
] as Message[];
const groups = getMessageGroups(messages);
// One processing group holds the orphan, the AI turn, and its tool result.
expect(groups.map((g) => g.type)).toEqual(["assistant:processing"]);
expect(groups[0]?.messages.map((m) => m.id)).toEqual([
"t-lead",
"ai-1",
"t-1",
]);
});
test("tool message preceded only by hidden messages gets a group", () => {
// messages is non-empty, but every earlier message is hidden from the UI —
// groups is still empty when the tool message arrives.
const messages = [
{
id: "hidden-1",
type: "human",
content:
"<slash_skill_activation>\n<skill_content># SKILL.md</skill_content>\n</slash_skill_activation>",
},
{
id: "t-1",
type: "tool",
name: "bash",
tool_call_id: "call-1",
content: "output",
},
] as Message[];
const groups = getMessageGroups(messages);
expect(groups.map((g) => g.type)).toEqual(["assistant:processing"]);
expect(groups[0]?.messages.map((m) => m.id)).toEqual(["t-1"]);
});
test("replayed tool with same tool_call_id is not lost (duplicate stream events)", () => {
// LangGraph subagent state restoration can replay tool-result events. The
// frontend log shows the same tool_call_id arriving twice. Both occurrences

33
frontend/tests/unit/core/threads/message-merge.test.ts
View File

@ -96,6 +96,39 @@ test("mergeMessages lets live thread messages replace overlapping history", () =
]);
});
test("mergeMessages preserves historical run metadata on a live checkpoint replacement", () => {
const persistedAi = {
id: "ai-1",
type: "ai",
content: "persisted",
additional_kwargs: { turn_duration: 114 },
} as Message;
const history = buildVisibleHistoryMessages(
[
{
run_id: "run-1",
content: persistedAi,
metadata: { caller: "lead_agent" },
created_at: "2026-07-21T00:00:00Z",
},
],
new Set(),
);
const checkpointAi = {
id: "ai-1",
type: "ai",
content: "live checkpoint",
} as Message;
expect(mergeMessages(history, [checkpointAi], [])).toEqual([
{
...checkpointAi,
run_id: "run-1",
additional_kwargs: { turn_duration: 114 },
},
]);
});
test("mergeMessages keeps a protected pre-compression input at its canonical position", () => {
const canonicalInput = {
id: "input-1",