From 14c9d44440780e63563e935046db8708e121a5b1 Mon Sep 17 00:00:00 2001 From: PeaceMaker-best Date: Tue, 15 Sep 2026 08:45:08 +0800 Subject: [PATCH] feat(runtime): persist tool-progress phase transitions (#5214) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(runtime): persist tool-progress phase transitions Record bounded warn, block, and recover decisions for lead and task subagent runs while preserving event-loop isolation, fail-open behavior, and concurrent transition order. * fix(runtime): trust server-owned tool progress attribution * fix(runtime): centralize trusted audit attribution * fix(runtime): preserve complete tool progress audit state * docs: trim tool progress guidance to pass size check * fix(runtime): fence subagent audit recorder loop * docs(readme): sync tool-progress event coverage Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> --------- Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: 嗜鵼 Co-authored-by: Willem Jiang --- README.md | 6 +- README_zh.md | 2 +- backend/app/gateway/services.py | 7 +- backend/docs/RUN_EVENT_STREAM.md | 37 +- .../deerflow/agents/middlewares/AGENTS.md | 4 +- .../agents/middlewares/audit_context.py | 27 +- .../middlewares/loop_detection_middleware.py | 18 +- .../middlewares/tool_progress_middleware.py | 202 ++++++++-- .../tool_promotion_audit_middleware.py | 20 +- .../agents/middlewares/tool_result_meta.py | 13 +- .../harness/deerflow/runtime/AGENTS.md | 18 + .../deerflow/runtime/events/catalog.py | 2 + .../harness/deerflow/runtime/journal.py | 96 +++-- .../harness/deerflow/runtime/runs/worker.py | 5 + .../harness/deerflow/subagents/AGENTS.md | 2 +- .../harness/deerflow/subagents/executor.py | 12 +- .../packages/harness/deerflow/tools/AGENTS.md | 2 +- .../deerflow/tools/builtins/task_tool.py | 4 + backend/tests/test_gateway_services.py | 22 ++ .../tests/test_loop_detection_middleware.py | 21 +- .../tests/test_run_event_stream_contract.py | 10 + backend/tests/test_run_journal.py | 132 +++++++ backend/tests/test_run_worker_rollback.py | 14 + backend/tests/test_subagent_executor.py | 34 ++ backend/tests/test_task_tool_core_logic.py | 13 + .../tests/test_tool_progress_middleware.py | 365 +++++++++++++++++- .../test_tool_promotion_audit_middleware.py | 33 ++ backend/tests/test_tool_result_meta.py | 2 + contracts/run_event_stream_contract.json | 4 +- 29 files changed, 1024 insertions(+), 103 deletions(-) diff --git a/README.md b/README.md index 3c9855df4..a6d364ef0 100644 --- a/README.md +++ b/README.md @@ -814,8 +814,10 @@ finalization) keep the existing completion-data behavior: they receive the zero-delivery receipt but do not overwrite RunStore completion fields with an empty snapshot. -The same run event history records loop-detection decisions and deferred MCP -tool promotions for both the lead agent and ordinary task subagents. Promotion +When `tool_progress.enabled` is true, the same run event history also records +result-quality guard phase changes. It records loop-detection decisions and +deferred MCP tool promotions for both the lead agent and ordinary task +subagents. Promotion events identify newly promoted deferred-tool names and whether routing metadata or `tool_search` selected them, without copying the search query, routing keywords, schemas, arguments, results, or catalog hash into the promotion event itself. diff --git a/README_zh.md b/README_zh.md index 0e6da345c..eef0d5b9f 100644 --- a/README_zh.md +++ b/README_zh.md @@ -562,7 +562,7 @@ logging: Gateway 的运行历史还会为每次运行记录一条终止时的 `run.delivery` 回执,包括零产出与崩溃恢复的运行。正常执行时,该回执会在持久化终止运行状态之前写入。孤儿恢复会先原子地认领过期租约,再幂等地回填回执,因此过期的恢复扫描不会覆盖仍在运行的详细交付事实。在事件存储中断期间,回执持久化保持尽力而为。对 checkpoint 预检失败(或在等待前序 finalization 时被取消)的运行,保持既有的完成数据行为:它们会收到零交付回执,但不会用空快照覆盖 RunStore 的完成字段。 -同一份运行事件历史还会为 lead agent 与普通 task subagent 记录 loop-detection 判定和延迟 MCP 工具晋升。晋升事件会标识新晋升的延迟工具名称,以及是路由元数据还是 `tool_search` 选中了它们,但不会把搜索查询、路由关键词、schema、参数、结果或目录哈希复制进晋升事件本身。 +当 `tool_progress.enabled` 为 true 时,同一份运行事件历史还会记录结果质量防护器的阶段变化。它也会为 lead agent 与普通 task subagent 记录 loop-detection 判定和延迟 MCP 工具晋升。晋升事件会标识新晋升的延迟工具名称,以及是路由元数据还是 `tool_search` 选中了它们,但不会把搜索查询、路由关键词、schema、参数、结果或目录哈希复制进晋升事件本身。 #### LangSmith 链路追踪 diff --git a/backend/app/gateway/services.py b/backend/app/gateway/services.py index 1defc5bee..ab4b1fe09 100644 --- a/backend/app/gateway/services.py +++ b/backend/app/gateway/services.py @@ -533,6 +533,11 @@ _SERVER_OWNED_RUNTIME_CONTEXT_KEYS: frozenset[str] = ( "is_internal", "authz_attributes", "channel_user_id", + "is_subagent", + "agent_id", + "__run_loop_detection_recorder", + "__run_tool_promotion_recorder", + "__run_tool_progress_recorder", "langgraph_auth_user", "langgraph_auth_user_id", } @@ -575,7 +580,7 @@ _CONTEXT_RUNTIME_ONLY_KEYS: frozenset[str] = frozenset({"github_token", "disable # credential that ``bash`` exports as ``GH_TOKEN``/``GITHUB_TOKEN``, and a copy # smuggled through ``body.config['configurable']`` would be written to the # checkpoint store. -_INTERNAL_ONLY_CONTEXT_KEYS: frozenset[str] = _CONTEXT_INTERNAL_CALLER_KEYS | _CONTEXT_RUNTIME_ONLY_KEYS +_INTERNAL_ONLY_CONTEXT_KEYS: frozenset[str] = _CONTEXT_INTERNAL_CALLER_KEYS | _CONTEXT_RUNTIME_ONLY_KEYS | _SERVER_OWNED_RUNTIME_CONTEXT_KEYS def strip_internal_context_keys(config: dict[str, Any]) -> None: diff --git a/backend/docs/RUN_EVENT_STREAM.md b/backend/docs/RUN_EVENT_STREAM.md index 31adf4803..adf766271 100644 --- a/backend/docs/RUN_EVENT_STREAM.md +++ b/backend/docs/RUN_EVENT_STREAM.md @@ -77,9 +77,9 @@ through run-event or specialized APIs: | `middleware:{tag}` | `middleware` | `record_middleware()` | Current middleware tags are `guardrail`, `loop_detection`, -`safety_termination`, `skill_activation`, `skill_secrets`, and -`tool_promotion`. The pattern is intentionally open so new middleware tags are -additive. Because the full event type is limited to 32 characters and +`safety_termination`, `skill_activation`, `skill_secrets`, `tool_promotion`, +and `tool_progress`. The pattern is intentionally open so new middleware tags +are additive. Because the full event type is limited to 32 characters and `middleware:` uses 11, a tag must contain 1-21 characters. `middleware:loop_detection` records transitions into the warned state (first @@ -95,6 +95,25 @@ subagent, and its agent id when applicable. Tool arguments, prompts, message content, tool results, and argument-derived hashes are not persisted in this event. +`middleware:tool_progress` records effective result-quality state transitions +from `ToolProgressMiddleware`: `warn` for ACTIVE → WARNED, `block` for a +transition to BLOCKED, `recover` for WARNED → ACTIVE after a useful result, +and `reset` when a later agent invocation (for example, goal continuation) +deliberately clears a prior WARNED/BLOCKED phase. +The bounded `changes` object contains the tool name, source and destination +phases, consecutive-problem count, normalized result status, error +classification and recovery advice, effective count threshold (null when no +count threshold fired), and +subagent attribution. Tool +arguments, prompts, message content, tool results, and content-derived hashes +are never copied into this event. Producer-supplied metadata is projected onto +fixed error/action vocabularies and a strict boolean-or-null before persistence, +so a custom tool cannot smuggle arbitrary values through its result stamp. +Recorder calls happen after the state lock is released, so a slow custom +recorder cannot stall unrelated tool-state updates. Recorder failures are +fail-open and do not change guard behavior. These events are emitted only when +`tool_progress.enabled` is true (the default is false). + `middleware:tool_promotion` records deferred MCP schemas newly promoted for the active catalog. `changes.source` distinguishes automatic `routing_hint` promotion from an explicit `tool_search`; the remaining fields contain sorted @@ -107,9 +126,10 @@ no names emit no event. Queries, routing keywords, catalog hashes, tool schemas and descriptions, arguments, and results are not copied into this middleware event. Other event types retain their existing payload contracts. -Ordinary task-tool subagents forward both loop-detection and tool-promotion -appends to the parent run loop through dedicated recorder context keys. The -loop-bound `RunJournal` itself never enters the isolated subagent loop. +Ordinary task-tool subagents forward loop-detection, tool-promotion, and +tool-progress appends to the parent run loop through dedicated recorder context +keys. The loop-bound `RunJournal` itself never enters the isolated subagent +loop. ### Opaque Run Outputs @@ -206,7 +226,8 @@ be used by new producers. - Nested non-JSON values in `run.end.content` have backend-dependent representations: memory retains Python values, while JSONL and database stores read them back as strings. -- Durable batch subagent loop detection and deferred-tool promotion do not - emit middleware events because those runs have no parent run journal. +- Durable batch subagent loop detection, deferred-tool promotion, and + tool-progress transitions do not emit middleware events because those runs + have no parent run journal. - Journal attribution, token accounting, and external tracing metadata still depend on manual instrumentation at several LLM call sites. diff --git a/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md b/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md index 1b7dabeb0..b847d5dca 100644 --- a/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md +++ b/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md @@ -73,7 +73,7 @@ ordinary compaction or a model call. contract module. 10. **SandboxAuditMiddleware** - Audits sandboxed shell/file operations before tool execution; command classification is **defense-in-depth and audit, not a security boundary** (the sandbox is the isolation boundary). Command substitution is judged by *position*, not the presence of `$(`: **command position** (`$(curl url)`, `` `curl url` ``, the word after `|`/`&&`/`;`, an `eval`/`source` argument) executes fetched content and is blocked; **value position** (`x=$(curl url)`, `echo $(curl url)`, an argument, a `for` word list) only captures output and passes (#4611). So `_HIGH_RISK_COMMAND_POSITION_PATTERNS` is matched anchored against each sub-command from `_split_compound_command(split_pipes=True)`, never the whole string; pipe-spanning rules (`| sh`, `base64 -d | ...`) still use `_classify_command`'s whole-command Pass 1. `_COMMAND_POSITION_PREFIX` extends the anchor over leading assignments and exec wrappers (`FOO=1 $(curl url)`, `env`/`command`/`builtin`/`exec`/`nohup`/`time`/`sudo`/`doas`); its assignment branch requires whitespace before the substitution, which keeps `x=$(curl url)` in value position. Two contexts are deliberately **position-blind** (matched whole-command in Pass 1, since they execute their input anywhere, e.g. `xargs sh -c "$(curl url)"`): an `eval`/`source` argument, and an interpreter **code-string flag** — `-c` (shells, `python`), `-e` (`perl`/`ruby`/`node`), `-p` (`perl`/`node`), `-r` (`php`) — plus the here-string (`<<<`) reaching the same place via stdin. All three substitution spellings (`$(`, `<(`, `` ` ``) share one `_RISKY_SUBSTITUTION` opener. An unquoted newline splits like `;` (else `echo hi\n$(curl url)` evades the anchored rules). Heredoc bodies are data: `_split_compound_command` records headers (`<