From 10bd065615fff997231bbae4fcde8667335d1ad2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=9D=8E=E9=9D=96?= <82300920+LancerLee428@users.noreply.github.com> Date: Sun, 27 Sep 2026 11:09:41 +0800 Subject: [PATCH] feat(agents): persist tool artifacts as resolvable handles (#4676) (#4929) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(agents): persist tool artifacts as resolvable handles (#4676) MCP and sandbox tools return file paths, URLs, and task ids that are lost when context compaction summarizes the conversation. This adds a per-thread artifact handle registry so the model can reference artifacts by short deterministic handles (art_xxxxxxxx) that survive compaction and resolve to real references at the tool-call boundary. - ThreadState.tool_artifacts: reducer channel capped at 100 entries, same-handle latest-wins, survives summarization compaction - artifact_registry: deterministic handle derivation from (thread_id, tool_call_id, call_index); extraction precedence structured_content -> file/image blocks -> conservative text scan; HTML-escaped model-context rendering with char budget - ArtifactCaptureMiddleware: before_model hook captures lightweight metadata (never content) and tracks handle consumption - ArtifactResolutionMiddleware: rewrites handles in tool-call args to real references before execution; unknown handles untouched - DurableContextMiddleware: projects the registry into the hidden durable-context data block ([available]/[consumed] status) - config: new tool_artifacts section (all switches default on) - frontend: artifact badges on tool-call steps keyed by tool_call_id Fixes #4676 * fix(agents): harden artifact registry after review (#4676) Review findings on #4929: - capture/consumption updates emitted in the same before_model call are now concatenated instead of dict-merged; dict-merging clobbered the shared tool_artifacts key and permanently dropped fresh captures - handles embed a per-result ordinal so multiple references inside one tool result never collide (two paths in one text block previously collapsed into one handle) - structured_content extraction pulls concrete refs from known keys (file/path/url/task_id/..., string or list-of-string values) instead of storing a truncated JSON preview, falls back to an untruncated whole-object data entry, and no longer skips file/image content blocks - plain-string tool results (e.g. bash/write_file output) are scanned for sandbox paths / remote URLs too, gated by detect_refs_in_text - wire max_entries through configure_tool_artifact_max_entries at assembly (clamped 10..1000) and gate detect_refs_in_text scans - resolution requires both enabled and resolve_handles_in_args, matching its docs - add real ToolCallRequest.override coverage and the spec 14.4 integration cycle (capture -> inject -> resolve -> consumption) * fix(agents): address second-round artifact registry review (#4676) Follow-up findings on #4929: - drop generic `output`/`outputs` from structured ref keys: prose under those keys no longer becomes a bogus file entry (falls back to the whole-object data entry; path-shaped refs are still caught by the free-text scan) - strip closing quotes, brackets, and backticks from detected refs so markdown- and JSON-wrapped paths resolve to clean real references - remove the process-wide cap setter: the reducer enforces only an absolute ceiling (1000) and ArtifactCaptureMiddleware applies the configured max_entries per agent when emitting updates, so agents with different configs in one process never share or flap a cap - skip extraction for tool messages whose tool_call_id already has registry entries and memoize known-empty results (bounded FIFO), so steady-state capture cost drops to the new message tail instead of a full-history rescan per model call - correct the inverted ordering claim: capture is position-independent in the wrap chain because it is a before_model hook reading state messages; fixed the assembly comment, AGENTS.md item 15, and the ordering test docstring CI parity verified locally: ruff check + format --check, full offline suite, blocking-io suite, collect-only, uv lock --check, check_agent_guidance, frontend format/lint/typecheck/build/tests. * fix(agents): sliding-window cap and URI gates for artifact registry (#4676) Third-round review findings on #4929: - max_entries is now a sliding retention window: fresh captures are always registered, and when the projection exceeds the configured cap the middleware appends a trailing {"op": "trim_to", "keep": N} directive that the reducer applies by evicting the oldest entries. The previous remaining-budget emission froze the registry at the cap, permanently dropping post-compaction artifacts — the exact failure mode #4676 targets — and silently disabled the round-2 rescan optimizations at the point threads grew longest - _track_consumption memoizes settled tool-call args in a bounded FIFO alongside the existing empty-result memo, so historical AIMessage args are not regex-rescanned every model call - file/image content-block URLs must be http(s) or path-shaped; data:/blob: URIs (MCP embedded resources carrying base64 payloads) can no longer enter thread state, tool args, or crowd out the render budget; protocol-relative //host forms are rejected as well - structured-content ref values go through the same URI-shape gate as content blocks, closing the embedded-payload hole via known keys such as url/urls - tool_artifacts.enabled now gates the whole middleware including consumption tracking via an early return in before_model Docs updated to describe the sliding window, URI gates, enabled scope, and both bounded memos. * fix(agents): final evictions and pending-aware consumption for artifact registry (#4676) Fourth-round review findings on #4929: - the sliding window interacted with the seen_call_ids skip to resurrect evicted entries every round: once an entry was trimmed, its tool_call_id left the seen set while its ToolMessage stayed in context, so the next model call re-extracted it, re-registered it as newest, and evicted something else — a never-converging rotation. The empty-result memo is now a unified _processed_results memo that records every extraction regardless of outcome, making evictions final; steady state over a saturated thread is quiescent (pinned by a multi-round test asserting no updates and zero extractions) - the consumption scan built its handle map from pre-capture state, so a call referencing a handle captured in the same before_model call missed, got settled as quiet, and its consumption mark was lost forever. _track_consumption now receives the pending capture entries and resolves against state + pending; a call settles as quiet only after every referenced handle resolved, so transient misses retry next round instead of permanently suppressing the mark AGENTS.md entry 15 updated to describe both memos and the eviction-is-final semantics. * fix(artifacts): address occurrence identity, PII and URL review findings * fix(artifacts): enforce resolved policy checks and bounded registry handling --------- Co-authored-by: Willem Jiang --- README.md | 2 + .../deerflow/agents/lead_agent/agent.py | 1 + .../deerflow/agents/middlewares/AGENTS.md | 58 +- .../agents/middlewares/TOOL_ARTIFACTS.md | 21 + .../artifact_capture_middleware.py | 205 ++++++ .../artifact_resolution_middleware.py | 119 ++++ .../middlewares/durable_context_middleware.py | 20 +- .../tool_error_handling_middleware.py | 20 + .../harness/deerflow/agents/thread_state.py | 70 ++ .../harness/deerflow/config/app_config.py | 2 + .../deerflow/config/tool_artifact_config.py | 38 + .../harness/deerflow/extensions/ordering.py | 10 + .../deerflow/tools/artifact_registry.py | 390 ++++++++++ .../blocking_io/test_artifact_middlewares.py | 85 +++ .../replay/write_read_file.ultra.events.json | 60 ++ .../tests/test_artifact_capture_middleware.py | 392 ++++++++++ backend/tests/test_artifact_integration.py | 124 ++++ backend/tests/test_artifact_registry.py | 434 +++++++++++ .../test_artifact_resolution_middleware.py | 194 +++++ .../tests/test_artifact_review_regressions.py | 175 +++++ backend/tests/test_artifact_safety_review.py | 223 ++++++ .../tests/test_durable_context_middleware.py | 75 ++ backend/tests/test_extension_ordering.py | 12 + .../tests/test_middleware_release_policy.py | 2 +- backend/tests/test_subagent_executor.py | 14 + backend/tests/test_thread_state_reducers.py | 8 + .../test_tool_error_handling_middleware.py | 73 +- config.example.yaml | 21 + ...2026-08-19-mcp-artifact-handle-registry.md | 674 ++++++++++++++++++ frontend/src/AGENTS.md | 2 +- .../workspace/messages/message-group.tsx | 57 +- .../workspace/messages/message-list.tsx | 2 + frontend/src/core/threads/stream-state.ts | 63 +- frontend/src/core/threads/types.ts | 14 + .../messages/tool-call-details.dom.test.tsx | 37 +- .../incremental-stream-state.dom.test.tsx | 13 + .../tool-artifact-stream-state.test.ts | 72 ++ 37 files changed, 3744 insertions(+), 38 deletions(-) create mode 100644 backend/packages/harness/deerflow/agents/middlewares/TOOL_ARTIFACTS.md create mode 100644 backend/packages/harness/deerflow/agents/middlewares/artifact_capture_middleware.py create mode 100644 backend/packages/harness/deerflow/agents/middlewares/artifact_resolution_middleware.py create mode 100644 backend/packages/harness/deerflow/config/tool_artifact_config.py create mode 100644 backend/packages/harness/deerflow/tools/artifact_registry.py create mode 100644 backend/tests/blocking_io/test_artifact_middlewares.py create mode 100644 backend/tests/test_artifact_capture_middleware.py create mode 100644 backend/tests/test_artifact_integration.py create mode 100644 backend/tests/test_artifact_registry.py create mode 100644 backend/tests/test_artifact_resolution_middleware.py create mode 100644 backend/tests/test_artifact_review_regressions.py create mode 100644 backend/tests/test_artifact_safety_review.py create mode 100644 docs/plans/2026-08-19-mcp-artifact-handle-registry.md create mode 100644 frontend/tests/unit/core/threads/tool-artifact-stream-state.test.ts diff --git a/README.md b/README.md index da1d7aa30e..0ca092afe9 100644 --- a/README.md +++ b/README.md @@ -631,6 +631,8 @@ Text results retain their original representation, including large numeric IDs a Copy actions copy only the displayed preview. This is a frontend view of data already received by the browser, without an additional secret-redaction layer. +Tool-produced paths and URLs can be retained as short artifact handles across context compaction (`tool_artifacts` in `config.yaml`). Handles distinguish separate tool-result occurrences, even when a provider reuses call IDs. Detected file URLs preserve their query strings and fragments. When PII redaction is enabled, model-visible artifact labels follow that policy; internal references stay intact for tool argument resolution. The configured registry limit retains the newest artifacts, while checkpointed processing identities prevent evicted results from being recaptured after restart. Resolution runs before authorization and write-safety checks; unknown or expired handles return an error without executing the tool. Small unknown structured results may be retained as complete JSON up to 4096 UTF-8 bytes; empty or oversized payloads are skipped. Handles are agent-local: task arguments resolve parent handles to concrete references, and delegated reports must return concrete references rather than child-local handles. A truncated model projection reports how many handles are omitted. + DeerFlow supports configurable MCP servers and skills to extend its capabilities. For HTTP/SSE MCP servers, OAuth token flows are supported (`client_credentials`, `refresh_token`). Durable HTTP/SSE task status and cancellation calls select configured `user_auth` credentials using the persisted task owner, including after restart; per-request secrets are not retained for background calls. If a request-scoped credential overrides submit authentication, both credentials must authorize access to the same remote task. diff --git a/backend/packages/harness/deerflow/agents/lead_agent/agent.py b/backend/packages/harness/deerflow/agents/lead_agent/agent.py index ae29f80549..8fe6882af8 100644 --- a/backend/packages/harness/deerflow/agents/lead_agent/agent.py +++ b/backend/packages/harness/deerflow/agents/lead_agent/agent.py @@ -609,6 +609,7 @@ def build_middlewares( DurableContextMiddleware( skills_container_path=resolved_app_config.skills.container_path, skill_file_read_tool_names=resolved_app_config.summarization.skill_file_read_tool_names, + inject_tool_artifacts=resolved_app_config.tool_artifacts.enabled and resolved_app_config.tool_artifacts.inject_model_context, task_continuity_enabled=getattr(getattr(resolved_app_config, "task_continuity", None), "enabled", False) is True, pii_redaction_config=getattr(resolved_app_config, "pii_redaction", None), ) diff --git a/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md b/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md index b87fe04d23..0ed0ad57cf 100644 --- a/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md +++ b/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md @@ -75,7 +75,8 @@ strict providers reject. force eager sandbox acquisition. 8. **DanglingToolCallMiddleware** - Injects placeholder ToolMessages for AIMessage tool_calls that lack responses (e.g., user interruption), preserving raw provider tool-call payloads in `additional_kwargs["tool_calls"]`; malformed tool-call names and arguments are sanitized in the model-bound request so strict OpenAI-compatible providers do not reject the next request 9. **LLMErrorHandlingMiddleware** - Converts provider/model failures to recoverable assistant errors. Sync and async calls carry circuit-generation ownership; only the current owner may settle or release a half-open probe, so stale completions cannot affect a newer recovery attempt. Cancellation propagates unchanged without retry or failure accounting. -10. **Authorization / GuardrailMiddleware** - Up to two independent pre-tool-call gates run here. When `authorization.enabled`, the `AuthorizationProvider` instance already used for Layer 1 capability filtering is wrapped by `GuardrailAuthorizationAdapter` and reused for Layer 2 execution checks. A generated `tool_search` bypasses the adapter's second provider call only when the current build has a concrete deferred setup; its catalog was already filtered by Layer 1, and an ordinary same-named tool without that deferred setup receives no exemption. When `guardrails.enabled`, the explicitly configured `GuardrailProvider` is appended after authorization and still evaluates every call, including `tool_search`. Authorization therefore runs outermost and can deny before an external guardrail call; both use the existing middleware's fail-closed, audit, sync/async, and error-`ToolMessage` behavior. See the authorization RFC and [docs/GUARDRAILS.md](../../../../../docs/GUARDRAILS.md). +10. **ArtifactResolutionMiddleware** - See [tool artifacts](TOOL_ARTIFACTS.md). +11. **Authorization / GuardrailMiddleware** - Up to two independent pre-tool-call gates run here. When `authorization.enabled`, the `AuthorizationProvider` instance already used for Layer 1 capability filtering is wrapped by `GuardrailAuthorizationAdapter` and reused for Layer 2 execution checks. A generated `tool_search` bypasses the adapter's second provider call only when the current build has a concrete deferred setup; its catalog was already filtered by Layer 1, and an ordinary same-named tool without that deferred setup receives no exemption. When `guardrails.enabled`, the explicitly configured `GuardrailProvider` is appended after authorization and still evaluates every call, including `tool_search`. Authorization therefore runs outermost and can deny before an external guardrail call; both use the existing middleware's fail-closed, audit, sync/async, and error-`ToolMessage` behavior. See the authorization RFC and [docs/GUARDRAILS.md](../../../../../docs/GUARDRAILS.md). Every guardrail decision path publishes a neutral `deerflow.authz.outcome.AuthorizationOutcome` into the per-run runtime context, @@ -83,10 +84,11 @@ strict providers reject. `__authorization_outcome` key (so `build_run_config` strips caller-supplied forgeries). Consumers pop it; the publisher and the consumer share only that contract module. -11. **SandboxAuditMiddleware** - Audits sandboxed shell/file operations before tool execution; command classification is **defense-in-depth and audit, not a security boundary** (the sandbox is the isolation boundary). Command substitution is judged by *position*, not the presence of `$(`: **command position** (`$(curl url)`, `` `curl url` ``, the word after `|`/`&&`/`;`, an `eval`/`source` argument) executes fetched content and is blocked; **value position** (`x=$(curl url)`, `echo $(curl url)`, an argument, a `for` word list) only captures output and passes (#4611). So `_HIGH_RISK_COMMAND_POSITION_PATTERNS` is matched anchored against each sub-command from `_split_compound_command(split_pipes=True)`, never the whole string; pipe-spanning rules (`| sh`, `base64 -d | ...`) still use `_classify_command`'s whole-command Pass 1. `_COMMAND_POSITION_PREFIX` extends the anchor over leading assignments and exec wrappers (`FOO=1 $(curl url)`, `env`/`command`/`builtin`/`exec`/`nohup`/`time`/`sudo`/`doas`); its assignment branch requires whitespace before the substitution, which keeps `x=$(curl url)` in value position. Two contexts are deliberately **position-blind** (matched whole-command in Pass 1, since they execute their input anywhere, e.g. `xargs sh -c "$(curl url)"`): an `eval`/`source` argument, and an interpreter **code-string flag** — `-c` (shells, `python`), `-e` (`perl`/`ruby`/`node`), `-p` (`perl`/`node`), `-r` (`php`) — plus the here-string (`<<<`) reaching the same place via stdin. All three substitution spellings (`$(`, `<(`, `` ` ``) share one `_RISKY_SUBSTITUTION` opener. An unquoted newline splits like `;` (else `echo hi\n$(curl url)` evades the anchored rules). Heredoc bodies are data: `_split_compound_command` records headers (`<