diff --git a/README.md b/README.md index 546f8b7da4..ca18c0600b 100644 --- a/README.md +++ b/README.md @@ -2926,6 +2926,11 @@ independently shaped summaries from other runners still contribute evidence. Pytest sibling summaries retain warning counts and the optional human-readable duration suffix printed for runs lasting at least a minute, such as `(0:01:05)`. +`tests_passed` evidence pairs each command with its own result, even when a +provider reuses a tool-call id in a later response. Each run stays a separate +entry in execution order, so a later failing run is decisive and an earlier +run's output is never credited to another command. + For Maven and Gradle test acceptance, a `BUILD SUCCESS` / `BUILD SUCCESSFUL` banner alone remains `UNVERIFIED`: builds can succeed without running tests. Complete `Tests run: N, Failures: F, Errors: E, Skipped: S` or `N tests completed` diff --git a/backend/packages/harness/deerflow/subagents/AGENTS.md b/backend/packages/harness/deerflow/subagents/AGENTS.md index 3a2e84e91e..99ce3091e3 100644 --- a/backend/packages/harness/deerflow/subagents/AGENTS.md +++ b/backend/packages/harness/deerflow/subagents/AGENTS.md @@ -97,7 +97,7 @@ Native worker reasoning settings come from YAML or managed definitions, with exp **Concurrency and total delegation cap**: Ordinary `task` concurrency is resolved once as the minimum of the per-run request, the startup-frozen `subagent_runtime.max_running`, and the schema safety ceiling (1-64), then shared by the lead prompt and `SubagentLimitMiddleware`. Hot reloads must not make either layer advertise more capacity than the already-created process controller; a changed startup-only value takes effect only after restart. The same middleware separately enforces `subagents.max_total_per_run` (default 6, config schema 1-50, runtime override `max_total_subagents` clamped to the same range) against current-run entries in the durable delegation ledger, so a long lead-agent run cannot bypass concurrency limits by launching repeated legal-sized batches at each planning checkpoint, but historical delegations from previous runs in the same thread do not consume the new run's budget. Explicit `batch_task` work does not consume or relax that ordinary-run ledger: its persisted total/live/running limits live under `subagent_batches`. Gateway `run_agent()` and embedded `DeerFlowClient.stream()` both provide a per-invocation `run_id` in runtime context; `DeerFlowClient.stream()` also tags its input `HumanMessage` with that same id so durable-context capture can identify the current request boundary. Gateway resume paths may not append a new `HumanMessage`, so the worker also exposes the pre-run checkpoint's message ids in runtime context; durable-context capture uses that as the current-run boundary and never re-tags older task calls as the resumed run. When no delegation slots remain, task calls are stripped, provider raw tool-call metadata is synced, `finish_reason` is forced to `stop`, and a visible "subagent delegation limit" note is appended so the agent can synthesize already-collected results. Default subagent timeout `subagents.timeout_seconds=1800` (30 min) and built-in `general-purpose` `max_turns=150`. **Flow**: `task()` → `SubagentExecutor` → shared slot → polling/SSE. Explicit `batch_task()` → durable rows → lease-based batch service (`subagents/batch_service.py`, Gateway or direct runtime) → same executor/slots → bounded result and owner-scoped API/JSONL. Batch mode is explicit, never inferred from prompt size. Queue rejection/timeout releases the lease without an item attempt; execution failure/expired leases consume one. User cancellation terminalizes nonterminal items and clears leases, fencing stale workers. Resolve background cancellation under `_background_tasks_lock`, then call `Future.cancel()` outside it because callbacks may reacquire the lock. `task_started` includes the effective model. `SubagentTokenCollector` publishes cumulative usage after each LLM response; `task_running` updates cards without double-counting. Terminal ToolMessage metadata (`subagent_model_name`, `subagent_token_usage`) and `subagent.end` preserve model/usage after reload; absent usage stays absent. Resolve one `AppConfig` snapshot (explicit or `get_app_config()`) for assembly, deferred setup, and receipt harvesting; `verification.receipts_enabled=false` holds on both construction paths. Harvest terminal receipts before `try_set_terminal` under the result lock so polling cannot see terminal status without metadata. Each values chunk updates the latest harvest state and publishes receipts before cooperative cancellation. Cancellation/failure scans current ToolMessages; completion uses the bounded ledger snapshot attached to returned assistant text, including max-turn partials. Missing/malformed completed snapshots yield no receipts, preventing omitted receipts from validating citations. Cancellation/timeouts retain the latest streamed evidence. **Report contract (RFC #4651 PR3)**: `report_contract.py` owns the prompt-layer text that makes Layer 1 receipt verification non-inert. `SubagentExecutor._build_initial_state` appends `build_report_contract_section(receipts_enabled=...)` to every subagent's consolidated `SystemMessage` — built-in and custom alike — requiring `[rN tool_name]` citations (from the Tool receipts ledger) for action claims, verifiable handles (absolute path, URL, ID, HTTP status) for deliverables, and explicit reporting of failures; the citation clause follows `verification.receipts_enabled`, and the citation example derives from the single-owner `format_citation`/`receipt_id` so prompt text cannot drift from the verifier. The `task` tool hands lead-supplied `acceptance_criteria` to the `SubagentExecutor` constructor, which appends them via `render_acceptance_criteria_block(...)` to the task `HumanMessage` (stripped, capped at 20 items × 500 chars, each entry neutralized) — the untrusted channel `InputSanitizationMiddleware` escapes and boundary-frames, matching their model-supplied provenance. The subagent's `SystemMessage` never carries criterion text; it gets only the framework-owned `build_acceptance_criteria_system_note(...)` pointer naming the list's location and authority, so natural-language injection inside a criterion cannot gain system-channel priority over framework instructions. Deterministic leaf checking is a separate layer. -**Acceptance checklist (RFC #4651 PR4)**: `acceptance_checks.py` checks lead-supplied `acceptance_criteria` in code on the `task` tool's completed branch (offloaded via `asyncio.to_thread`, failure-isolated). Decidable leaves: `file: exists|non-empty` and `file_written:` read through `read_current_file_content` scoped to the shared thread workspace (`workspace_path`/`outputs_path`; virtual `/mnt/user-data/...` prefixes and workspace-relative spellings normalized first) — the read itself uses the sandbox-native virtual form because the local read validator and provider mount tables resolve virtual paths, not host paths; a remote provider's `"Error: ..."` return string (remote providers return instead of raise for missing files) is normalized to a failed check, never evaluated as content — provider-typed via `is_local_sandbox`, so a genuine `Error:`-prefixed file on the local sandbox stays valid content; a UnicodeDecodeError on a text read marks a binary deliverable (PDF/image) as existing and non-empty instead of dropping the verdict; reads are byte-bounded — the size is established first (`os.stat` on the validated host path locally, so the host-bash-disabled configuration needs no shell; a metadata-only `stat`/`realpath` probe in a fresh `env -i` shell on remote providers — absolute-path utilities and a marker env that routes AIO off its persistent shell, so a completed subagent's poisoned session state cannot steer it; `stat` never opens content, so a FIFO cannot block the parent for the provider's idle timeout; the file must stat as a regular non-symlink file, and containment is canonicalized — the file's realpath must stay under the mount root's realpath, which is exactly what the provider's own read path resolves (e2b and Tenki realize `/mnt/user-data` as a symlink to the home dir by default), so a final-component symlink is rejected outright and an intermediate dir-link escape under a sane root still lands outside the canonical root), leaves above `_FILE_CONTENT_READ_CAP_BYTES` answer from the size alone — `file_written` only with an added bounded one-byte open probe (a mode-000 file stats fine while any open raises EACCES), and smaller files run the full read, and an unestablishable size degrades to UNVERIFIED rather than an unbounded fallback read; out-of-scope paths degrade to UNVERIFIED, never misjudge — on the local sandbox the scope decision canonicalizes with realpath, so a workspace symlink into uploads cannot satisfy a scoped leaf with upload content. `tests_passed:` anchors to a matching bash execution (newest match wins) with `status=success` and a test-summary stamp computed from the complete recorded tool output before tail truncation — each harvested execution carries a `shell_persistent` provenance stamp — the producing sandbox's `persistent_shell_sessions` flag (AIO's legacy exec path) resolved from the state that carried the evidence, never the parent runtime, which has no `sandbox` key when the parent delegated before touching one — and a persistent stamp (or an unidentifiable or undeclared one — a custom provider that never declared its session semantics is unknown, not fresh-shell; unknown provenance fails closed) degrades the leaf to UNVERIFIED instead, because any earlier call in the shared session could have mutated the state the clean-looking run executed in and a fresh controlled session (RFC §6 verifier) would be needed to prove otherwise, harvested by the executor (`_harvest_bash_executions`, only when criteria were delegated) from the same stamped `ToolMessage`s the receipt layer reads, accumulated per streamed chunk (merged by `tool_call_id`, newest-capped) so subagent summarization compacting earlier messages cannot erase a recorded execution, with over-cap commands carrying `command_truncated` so the matcher degrades to UNVERIFIED instead of proving a match on a suffix-less prefix; the recorded status is the actual shell exit status parsed from the output's `Exit Code: N` / `Command exited with code N` marker (a nonzero bash exit returns ordinary text that `deerflow_tool_meta` still reports as success; every provider appends a marker on nonzero exit — local/e2b/opensandbox/tenki/boxlite textually, aio via the SDK's structured exit_code — and timeouts append `Exit Code: 124` (local process-group kill, AIO `hard_timeout`, e2b/tenki SDK deadlines, boxlite bridge deadline) while signal kills parse as signed markers (`Exit Code: -9` records error); `_truncate_bash_output` preserves a trailing exit marker inside its budget, with a 32-char floor for smaller configured limits and the remote `Command exited with code N` form accepted only as the whole trimmed output; the matched marker text travels on the entry as `status_marker`), falling back to the meta status only when no marker exists; matching is shell-structure aware (operator-separated segments — a physical newline separates with `;` semantics — or with the continuation operator the next line opens with (`cmd1\n&& cmd2` is `&&`; `cmd1\n|| cmd2` is `||`, which after a successful first command skips the rest while exiting 0, so flattening it to `;` would record a run that never happened) —, so a multi-line script's trailing `echo`/`seq` lines are never merged into the matched segment's arguments, status, or output attribution —, executable identity — directional: a bare criterion executable accepts any path spelling of the name, while an explicitly path-spelled criterion requires a path-spelled execution of the same normalized executable path (spelling judged on the raw token: `./pytest` names the project-local file and normpath collapsing `./` must not demote it to a PATH lookup; a `..` component on either side is unprovable outright), so `/tmp/fake/pytest` cannot certify `/opt/project/.venv/bin/pytest` —, ordered argument subsequence whose env-assignment prefix must equal the criterion's exactly as an effective name → final-value mapping — extra, missing, or differently-valued assignments degrade the match, as does any reordering of a repeated name (`CI=0 CI=1` vs `CI=1 CI=0` are last-wins opposite environments), since no variable is provably inert across repositories, and any assignment or argumented `export`/`unset` in a preceding segment is state pollution, and any span token carrying a runtime expansion (`$VAR`/`$( )`/backticks) or an extra token carrying glob metacharacters (crafted option-looking filenames narrow invisibly) is likewise unprovable), so a command that merely mentions the criterion string (`echo '12 passed'; # pytest x.py`) cannot anchor the leaf, and control flow is preserved — the matching span must end at the command's last segment with provable execution (`&&` needs recorded success, `||` needs recorded failure, pipelines inside the span and backgrounding are never provable), and the criterion's own connectors are preserved — an expected `&&` executed as `;` (`cd missing; pytest x` for `cd missing && pytest x`) lets a failed preceding step be bypassed, so it degrades; only the stricter direction (criterion `;` executed as `&&` with recorded success) survives), so a short-circuited segment (`false && pytest x; echo '3 passed'`) degrades to UNVERIFIED instead of a false hold; the summary shape is evaluated only when the output is attributable to the matched segment (every preceding segment provably silent by invocation form — only shape-free `cd dir` and pure assignments qualify; `pushd`/`umask`/`ulimit`, any `export`/`unset` (including an invalid identifier, whose bash error text is subagent-chosen), and any `source`/`.`, whose `*/bin/activate` path shape says nothing about what a crafted script prints, are not; the `cd` print channel is closed too — CDPATH makes `cd` print the subagent-chosen resolved path and the pass shapes match as substrings, so a `cd` argument carrying a summary shape or runtime expansion is non-silent, and any `CDPATH=` assignment degrades the match as state pollution), Go `[no test files]` / `[no tests to run]` lines are package-local: other packages' passes survive; all-zero stays UNVERIFIED, as does a `-v` record whose per-case lines are only `--- SKIP`; `0 passed` / `Ran 0 tests` veto, argument matching treats tokens consumed by negating options (`--ignore`/`--deselect`/`--exclude`/`--skip` families) as ineligible evidence (`pytest --ignore tests/security tests` cannot affirm the excluded tests) and a target negated anywhere in the command degrades the match (`pytest tests/security tests/unit --ignore tests/security` cannot certify the excluded target), and a bare criterion with no positional selection target (bare `pytest`, `make test`) treats ANY negating option as a different selection, and accepts extra executed flags only when provably selection-preserving (verbosity/output/xdist/coverage forms; `-k`/`-m`/`--lf`/`--collect-only`/`-c`/`-p` and friends degrade to UNVERIFIED — `pytest -k smoke tests/security` cannot certify the full selection; extra positionals widen only after a path-scoped criterion — scoping reads positional targets by option arity, so a path embedded in an option (`--basetemp=/tmp/p`) never counts as a target, and an option of unknown arity (absent from the value-taking table, no glued `=`) followed by a path-like token (`--rootdir /tmp/project`) fails closed, so `python -m unittest pkg.OneTest` cannot certify bare `python -m unittest`), and criterion text is neutralized with `neutralize_untrusted_tags` before storage/rendering so a blocked tag in a criterion can never reach the lead-visible checklist section. Other criteria stay UNVERIFIED. The verdict (`subagent_acceptance_verdict` in `additional_kwargs`, validated on read) feeds the delegation ledger's `acceptance:` segment and the model-visible checklist section appended to the result text; the gateway strips caller-forged verdicts from both ledger entries and message metadata like the citation verdict. Leaf read errors catch only `OSError`/`SandboxError` — unexpected errors propagate to the task-tool-level isolation so the blocking-IO gate keeps teeth (`tests/blocking_io/test_task_tool_acceptance_checklist.py`). Known accepted boundaries (pinned by `TestKnownBoundaries` — conscious, do not re-raise): a bare criterion executable trusts PATH/filesystem spelling (a subagent-crafted same-name script is textually indistinguishable; spelling an absolute criterion path restricts evidence to that normalized path); relative criterion targets resolve in the wrapper's cwd, so `cd` prefixes must stay in scope (relative without `..`, or absolute under the thread data roots / virtual prefix) while a symlink inside an allowed root pointing out is a filesystem-layer concern; runner semantics are trusted (a Makefile swallowing failures, a runner exiting 0 on failure) — Layer 2 is execution evidence only, claim correctness belongs to the PR5 judge / RFC §6 re-execution; evidence is bounded (500-char command, 1000-char diagnostic tail, fixed-size test-summary stamp); command truncation degrades to UNVERIFIED rather than proving. Legacy entries without a summary stamp use their retained tail. +**Acceptance checklist (RFC #4651 PR4)**: `acceptance_checks.py` checks lead-supplied `acceptance_criteria` in code on the `task` tool's completed branch (offloaded via `asyncio.to_thread`, failure-isolated). Decidable leaves: `file: exists|non-empty` and `file_written:` read through `read_current_file_content` scoped to the shared thread workspace (`workspace_path`/`outputs_path`; virtual `/mnt/user-data/...` prefixes and workspace-relative spellings normalized first) — the read itself uses the sandbox-native virtual form because the local read validator and provider mount tables resolve virtual paths, not host paths; a remote provider's `"Error: ..."` return string (remote providers return instead of raise for missing files) is normalized to a failed check, never evaluated as content — provider-typed via `is_local_sandbox`, so a genuine `Error:`-prefixed file on the local sandbox stays valid content; a UnicodeDecodeError on a text read marks a binary deliverable (PDF/image) as existing and non-empty instead of dropping the verdict; reads are byte-bounded — the size is established first (`os.stat` on the validated host path locally, so the host-bash-disabled configuration needs no shell; a metadata-only `stat`/`realpath` probe in a fresh `env -i` shell on remote providers — absolute-path utilities and a marker env that routes AIO off its persistent shell, so a completed subagent's poisoned session state cannot steer it; `stat` never opens content, so a FIFO cannot block the parent for the provider's idle timeout; the file must stat as a regular non-symlink file, and containment is canonicalized — the file's realpath must stay under the mount root's realpath, which is exactly what the provider's own read path resolves (e2b and Tenki realize `/mnt/user-data` as a symlink to the home dir by default), so a final-component symlink is rejected outright and an intermediate dir-link escape under a sane root still lands outside the canonical root), leaves above `_FILE_CONTENT_READ_CAP_BYTES` answer from the size alone — `file_written` only with an added bounded one-byte open probe (a mode-000 file stats fine while any open raises EACCES), and smaller files run the full read, and an unestablishable size degrades to UNVERIFIED rather than an unbounded fallback read; out-of-scope paths degrade to UNVERIFIED, never misjudge — on the local sandbox the scope decision canonicalizes with realpath, so a workspace symlink into uploads cannot satisfy a scoped leaf with upload content. `tests_passed:` anchors to a matching bash execution (newest match wins) with `status=success` and a test-summary stamp computed from the complete recorded tool output before tail truncation — each harvested execution carries a `shell_persistent` provenance stamp — the producing sandbox's `persistent_shell_sessions` flag (AIO's legacy exec path) resolved from the state that carried the evidence, never the parent runtime, which has no `sandbox` key when the parent delegated before touching one — and a persistent stamp (or an unidentifiable or undeclared one — a custom provider that never declared its session semantics is unknown, not fresh-shell; unknown provenance fails closed) degrades the leaf to UNVERIFIED instead, because any earlier call in the shared session could have mutated the state the clean-looking run executed in and a fresh controlled session (RFC §6 verifier) would be needed to prove otherwise, harvested by the executor (`_harvest_bash_executions`, only when criteria were delegated) from the same stamped `ToolMessage`s the receipt layer reads, accumulated per streamed chunk (merged by result message ID, newest-capped) so subagent summarization compacting earlier messages cannot erase a recorded execution, with over-cap commands carrying `command_truncated` so the matcher degrades to UNVERIFIED instead of proving a match on a suffix-less prefix; the recorded status is the actual shell exit status parsed from the output's `Exit Code: N` / `Command exited with code N` marker (a nonzero bash exit returns ordinary text that `deerflow_tool_meta` still reports as success; every provider appends a marker on nonzero exit — local/e2b/opensandbox/tenki/boxlite textually, aio via the SDK's structured exit_code — and timeouts append `Exit Code: 124` (local process-group kill, AIO `hard_timeout`, e2b/tenki SDK deadlines, boxlite bridge deadline) while signal kills parse as signed markers (`Exit Code: -9` records error); `_truncate_bash_output` preserves a trailing exit marker inside its budget, with a 32-char floor for smaller configured limits and the remote `Command exited with code N` form accepted only as the whole trimmed output; the matched marker text travels on the entry as `status_marker`), falling back to the meta status only when no marker exists; matching is shell-structure aware (operator-separated segments — a physical newline separates with `;` semantics — or with the continuation operator the next line opens with (`cmd1\n&& cmd2` is `&&`; `cmd1\n|| cmd2` is `||`, which after a successful first command skips the rest while exiting 0, so flattening it to `;` would record a run that never happened) —, so a multi-line script's trailing `echo`/`seq` lines are never merged into the matched segment's arguments, status, or output attribution —, executable identity — directional: a bare criterion executable accepts any path spelling of the name, while an explicitly path-spelled criterion requires a path-spelled execution of the same normalized executable path (spelling judged on the raw token: `./pytest` names the project-local file and normpath collapsing `./` must not demote it to a PATH lookup; a `..` component on either side is unprovable outright), so `/tmp/fake/pytest` cannot certify `/opt/project/.venv/bin/pytest` —, ordered argument subsequence whose env-assignment prefix must equal the criterion's exactly as an effective name → final-value mapping — extra, missing, or differently-valued assignments degrade the match, as does any reordering of a repeated name (`CI=0 CI=1` vs `CI=1 CI=0` are last-wins opposite environments), since no variable is provably inert across repositories, and any assignment or argumented `export`/`unset` in a preceding segment is state pollution, and any span token carrying a runtime expansion (`$VAR`/`$( )`/backticks) or an extra token carrying glob metacharacters (crafted option-looking filenames narrow invisibly) is likewise unprovable), so a command that merely mentions the criterion string (`echo '12 passed'; # pytest x.py`) cannot anchor the leaf, and control flow is preserved — the matching span must end at the command's last segment with provable execution (`&&` needs recorded success, `||` needs recorded failure, pipelines inside the span and backgrounding are never provable), and the criterion's own connectors are preserved — an expected `&&` executed as `;` (`cd missing; pytest x` for `cd missing && pytest x`) lets a failed preceding step be bypassed, so it degrades; only the stricter direction (criterion `;` executed as `&&` with recorded success) survives), so a short-circuited segment (`false && pytest x; echo '3 passed'`) degrades to UNVERIFIED instead of a false hold; the summary shape is evaluated only when the output is attributable to the matched segment (every preceding segment provably silent by invocation form — only shape-free `cd dir` and pure assignments qualify; `pushd`/`umask`/`ulimit`, any `export`/`unset` (including an invalid identifier, whose bash error text is subagent-chosen), and any `source`/`.`, whose `*/bin/activate` path shape says nothing about what a crafted script prints, are not; the `cd` print channel is closed too — CDPATH makes `cd` print the subagent-chosen resolved path and the pass shapes match as substrings, so a `cd` argument carrying a summary shape or runtime expansion is non-silent, and any `CDPATH=` assignment degrades the match as state pollution), Go `[no test files]` / `[no tests to run]` lines are package-local: other packages' passes survive; all-zero stays UNVERIFIED, as does a `-v` record whose per-case lines are only `--- SKIP`; `0 passed` / `Ran 0 tests` veto, argument matching treats tokens consumed by negating options (`--ignore`/`--deselect`/`--exclude`/`--skip` families) as ineligible evidence (`pytest --ignore tests/security tests` cannot affirm the excluded tests) and a target negated anywhere in the command degrades the match (`pytest tests/security tests/unit --ignore tests/security` cannot certify the excluded target), and a bare criterion with no positional selection target (bare `pytest`, `make test`) treats ANY negating option as a different selection, and accepts extra executed flags only when provably selection-preserving (verbosity/output/xdist/coverage forms; `-k`/`-m`/`--lf`/`--collect-only`/`-c`/`-p` and friends degrade to UNVERIFIED — `pytest -k smoke tests/security` cannot certify the full selection; extra positionals widen only after a path-scoped criterion — scoping reads positional targets by option arity, so a path embedded in an option (`--basetemp=/tmp/p`) never counts as a target, and an option of unknown arity (absent from the value-taking table, no glued `=`) followed by a path-like token (`--rootdir /tmp/project`) fails closed, so `python -m unittest pkg.OneTest` cannot certify bare `python -m unittest`), and criterion text is neutralized with `neutralize_untrusted_tags` before storage/rendering so a blocked tag in a criterion can never reach the lead-visible checklist section. Other criteria stay UNVERIFIED. The verdict (`subagent_acceptance_verdict` in `additional_kwargs`, validated on read) feeds the delegation ledger's `acceptance:` segment and the model-visible checklist section appended to the result text; the gateway strips caller-forged verdicts from both ledger entries and message metadata like the citation verdict. Leaf read errors catch only `OSError`/`SandboxError` — unexpected errors propagate to the task-tool-level isolation so the blocking-IO gate keeps teeth (`tests/blocking_io/test_task_tool_acceptance_checklist.py`). Known accepted boundaries (pinned by `TestKnownBoundaries` — conscious, do not re-raise): a bare criterion executable trusts PATH/filesystem spelling (a subagent-crafted same-name script is textually indistinguishable; spelling an absolute criterion path restricts evidence to that normalized path); relative criterion targets resolve in the wrapper's cwd, so `cd` prefixes must stay in scope (relative without `..`, or absolute under the thread data roots / virtual prefix) while a symlink inside an allowed root pointing out is a filesystem-layer concern; runner semantics are trusted (a Makefile swallowing failures, a runner exiting 0 on failure) — Layer 2 is execution evidence only, claim correctness belongs to the PR5 judge / RFC §6 re-execution; evidence is bounded (500-char command, 1000-char diagnostic tail, fixed-size test-summary stamp); command truncation degrades to UNVERIFIED rather than proving. Legacy entries without a summary stamp use their retained tail. **Acceptance checklist path portability**: Paths are host-independent and raw `..` fails closed. Drive/UNC absolutes retain their class across `ntpath` normalization and cannot cross the root; drive-relative, shell-dependent provider/PSDrive, and POSIX-rooted `cd` on Windows are unprovable. Selection overlap separates pytest node IDs, normalizes safe `.`/duplicate separators, rejects `..`, and applies Windows casing to drive, UNC, PSDrive, or Windows-context paths; provider-qualified drive/UNC roots remain paths until a later node-ID split. POSIX paths and all node IDs keep their required case sensitivity. Ambiguous trailing-dot/space or 8.3 components, different volume IDs, cross-family pairs, and absolute/relative PSDrive pairs fail closed without execution/filesystem provenance; only same-volume, same-root-form paths compare lexically. Without shell provenance, raw commands reject backslashes; cmd `%VAR%`/`!VAR!`, `^`, `#`; Bash braces/tilde; and PowerShell splatting, typographic quotes, or unquoted parentheses. A lone `%` remains eligible. POSIX-only markers must short-circuit on Windows because pytest evaluates them at import. **JVM test evidence**: Build-success banners do not establish `tests_passed`. @@ -119,11 +119,14 @@ to their tail. Complete Node counters outrank loose pass/fail log text; independ sibling runner summaries still count. Preserve pytest warnings/duration suffixes and Go per-case PASS/FAIL/SKIP records when filtering sibling evidence; an all-skipped Go package must not gain proof merely from its `ok` banner. Full-output stamps are cached per execution -by tool-call ID and unchanged content; changed content is reclassified. The cache +by result message ID and unchanged content; changed content is reclassified. The cache is invocation-local and pruned to the newest 20 retained executions, so values chunks do not repeatedly run full-output regexes or retain compacted logs. Bounded diagnostic tails drop their first partial physical line when slicing cuts a line; otherwise a long test label can lose its reporter prefix. +Providers may reuse call ids: pair results via `pair_tool_call_results` and +merge chunks by result message ID (`tests/test_acceptance_evidence_reused_call_ids.py`). +Calls without an id pair with nothing, so they record no evidence (UNVERIFIED). **Events**: `task_started`, `task_running`, `task_completed`/`task_failed`/`task_timed_out` **Handled LLM failures**: `LLMErrorHandlingMiddleware` deliberately converts provider/model exceptions into an `AIMessage` so the graph can end cleanly, stamping `additional_kwargs.deerflow_error_fallback=true` plus error metadata. Clean graph termination does not imply subagent success: `SubagentExecutor` inspects the last assistant message at terminalization and maps a marked fallback to `SubagentStatus.FAILED`, which then emits `task_failed` and the existing structured `subagent_error`. Only the marker is authoritative — error-looking assistant prose without it remains a normal completed result, so neither the executor nor frontend parses display text as a status protocol. diff --git a/backend/packages/harness/deerflow/subagents/executor.py b/backend/packages/harness/deerflow/subagents/executor.py index 59b823a4c9..7326ded25b 100644 --- a/backend/packages/harness/deerflow/subagents/executor.py +++ b/backend/packages/harness/deerflow/subagents/executor.py @@ -155,7 +155,8 @@ class SubagentResult: ``persistent_shell_sessions`` flag resolved from the state that carried the evidence (``None`` when unidentifiable — the matcher fails closed on it). - Accumulated per chunk (merged by ``tool_call_id``) so + Accumulated per chunk (merged by result message id, ``tool_call_id`` + only for entries without one) so summarization compacting earlier messages cannot erase a recorded execution. ``None`` when the delegation carried no acceptance criteria, the run ended before streaming, or harvesting failed; @@ -213,8 +214,9 @@ class SubagentResult: def update_bash_executions(self, executions: list[dict[str, Any]] | None) -> None: """Merge bash evidence from the latest yielded state while still running. - Entries merge by ``tool_call_id`` in first-seen order and are capped to - the newest ``_BASH_EVIDENCE_MAX_ENTRIES`` — unlike a terminal + Entries merge by their result message (``tool_call_id`` when it has no + id, since providers can reuse call ids across responses) in first-seen + order and are capped to the newest ``_BASH_EVIDENCE_MAX_ENTRIES`` — unlike a terminal ``final_state`` scan, accumulation survives summarization compacting earlier AI/ToolMessages out of the streamed history. ``None`` leaves the field untouched (no evidence this chunk); an empty list still @@ -227,9 +229,9 @@ class SubagentResult: with self._state_lock: if self.status.is_terminal: return - merged = {str(entry.get("tool_call_id")): entry for entry in (self.bash_executions or [])} + merged = {_bash_evidence_key(entry): entry for entry in (self.bash_executions or [])} for execution in executions: - merged[str(execution.get("tool_call_id"))] = dict(execution) + merged[_bash_evidence_key(execution)] = dict(execution) self.bash_executions = list(merged.values())[-_BASH_EVIDENCE_MAX_ENTRIES:] def snapshot_tool_receipts(self) -> list[dict[str, Any]] | None: @@ -531,10 +533,22 @@ def _harvest_shell_persistence(final_state: Any) -> bool | None: return None +def _bash_evidence_key(execution: dict[str, Any]) -> tuple[str, str]: + """Identify one recorded execution across values chunks. + + A provider may reuse a call id in a later response, so the result message + id names the execution; entries without one keep the call-id identity. + """ + result_id = execution.get("result_id") + if isinstance(result_id, str) and result_id: + return ("result", result_id) + return ("call", str(execution.get("tool_call_id"))) + + def _harvest_bash_executions( final_state: Any, *, - summary_cache: dict[str, tuple[str, str]] | None = None, + summary_cache: dict[tuple[str, str], tuple[str, str]] | None = None, ) -> list[dict[str, Any]] | None: """Harvest bounded bash command/output evidence from one streamed state. @@ -564,48 +578,47 @@ def _harvest_bash_executions( summary_cache.clear() return None try: + from deerflow.agents.middlewares.tool_call_args import pair_tool_call_results from deerflow.agents.middlewares.tool_result_meta import TOOL_META_KEY from deerflow.subagents.acceptance_checks import summarize_test_output - commands: dict[str, tuple[str, str, bool]] = {} - for message in messages: - if not isinstance(message, AIMessage): - continue - for tool_call in message.tool_calls or []: - name = str(tool_call.get("name") or "") - if name not in _BASH_EVIDENCE_TOOL_NAMES: - continue - tool_call_id = str(tool_call.get("id") or "") - args = tool_call.get("args") - command = args.get("command") if isinstance(args, dict) else None - command = command if isinstance(command, str) else "" - # A truncated command loses its suffix; the matcher must not - # treat shell-structure analysis of the prefix as proof (a - # selection-changing suffix like ``-k smoke`` could be cut). - commands[tool_call_id] = (name, command[:_BASH_EVIDENCE_COMMAND_CHARS], len(command) > _BASH_EVIDENCE_COMMAND_CHARS) - if not commands: + # Tool-call ids are unique per response, not per conversation: pair each + # result with the call of its own response, so a reused id never labels + # one command's output with another command. + bash_calls = [occurrence for occurrence in pair_tool_call_results(messages) if occurrence.name in _BASH_EVIDENCE_TOOL_NAMES] + if not bash_calls: if summary_cache is not None: summary_cache.clear() return [] + positions = {id(message): index for index, message in enumerate(messages)} + answered = sorted((occurrence for occurrence in bash_calls if occurrence.result is not None), key=lambda occurrence: positions[id(occurrence.result)]) executions: list[dict[str, Any]] = [] # Only retained executions need full-output parsing. The per-run cache # survives values chunks, but keeps at most the same newest 20 outputs. - tool_messages = [message for message in messages if isinstance(message, ToolMessage) and str(message.tool_call_id or "") in commands][-_BASH_EVIDENCE_MAX_ENTRIES:] - next_cache: dict[str, tuple[str, str]] = {} - for message in tool_messages: - tool_call_id = str(message.tool_call_id or "") - entry = commands[tool_call_id] - name, command, command_truncated = entry + next_cache: dict[tuple[str, str], tuple[str, str]] = {} + for occurrence in answered[-_BASH_EVIDENCE_MAX_ENTRIES:]: + message = occurrence.result + tool_call_id = occurrence.call_id + name = occurrence.name + command = occurrence.args.get("command") + command = command if isinstance(command, str) else "" + # A truncated command loses its suffix; the matcher must not + # treat shell-structure analysis of the prefix as proof (a + # selection-changing suffix like ``-k smoke`` could be cut). + command_truncated = len(command) > _BASH_EVIDENCE_COMMAND_CHARS + command = command[:_BASH_EVIDENCE_COMMAND_CHARS] meta = (message.additional_kwargs or {}).get(TOOL_META_KEY) or {} meta_status = str(meta.get("status") or getattr(message, "status", "success") or "success") content = message.content if isinstance(message.content, str) else json.dumps(message.content, sort_keys=True, default=str) status, status_marker = _bash_evidence_status(content, meta_status) - cached = summary_cache.get(tool_call_id) if summary_cache is not None else None + result_id = message.id if isinstance(message.id, str) and message.id else None + cache_key = _bash_evidence_key({"result_id": result_id, "tool_call_id": tool_call_id}) + cached = summary_cache.get(cache_key) if summary_cache is not None else None if cached is not None and (cached[0] is content or cached[0] == content): test_summary = cached[1] else: test_summary = summarize_test_output(content) - next_cache[tool_call_id] = (content, test_summary) + next_cache[cache_key] = (content, test_summary) output_tail = content[-_BASH_EVIDENCE_OUTPUT_TAIL_CHARS:] if len(content) > _BASH_EVIDENCE_OUTPUT_TAIL_CHARS and content[-_BASH_EVIDENCE_OUTPUT_TAIL_CHARS - 1] != "\n": # A clipped test name/log line can look like a summary after @@ -614,6 +627,7 @@ def _harvest_bash_executions( executions.append( { "tool_call_id": tool_call_id, + "result_id": result_id, "tool_name": name, "command": command, "command_truncated": command_truncated, @@ -1802,7 +1816,7 @@ class SubagentExecutor: return None return _harvest_tool_receipts(final_state, prefer_citing_turn=prefer_citing_turn) - bash_summary_cache: dict[str, tuple[str, str]] = {} + bash_summary_cache: dict[tuple[str, str], tuple[str, str]] = {} def current_bash_executions() -> list[dict[str, Any]] | None: # RFC #4651 PR4: evidence for tests_passed acceptance leaves. diff --git a/backend/tests/test_acceptance_evidence_reused_call_ids.py b/backend/tests/test_acceptance_evidence_reused_call_ids.py new file mode 100644 index 0000000000..88cece465c --- /dev/null +++ b/backend/tests/test_acceptance_evidence_reused_call_ids.py @@ -0,0 +1,175 @@ +"""tests_passed evidence must follow each tool call, even when a provider reuses its id. + +Tool-call ids are unique per assistant response, not per conversation: some +providers number calls per response, so two turns can both issue ``call_0``. +""" + +import importlib.util +import sys +from pathlib import Path + +import pytest +from langchain_core.language_models.fake_chat_models import GenericFakeChatModel +from langchain_core.messages import AIMessage, HumanMessage, ToolMessage +from langchain_core.tools import StructuredTool + +from deerflow.config.app_config import AppConfig +from deerflow.subagents.acceptance_checks import check_acceptance_criteria +from deerflow.subagents.config import SubagentConfig + +FAILED_RUN = "===== 1 failed, 2 passed in 0.10s =====\nExit Code: 1" +PASSED_RUN = "===== 3 passed in 0.10s =====" + + +@pytest.fixture +def executor_module(monkeypatch, tmp_path): + # conftest installs an executor stub. Load the production module separately + # so the real subagent graph streams, harvests and merges the evidence. + path = Path(__file__).parents[1] / "packages/harness/deerflow/subagents/executor.py" + spec = importlib.util.spec_from_file_location("_reused_call_id_executor", path) + module = importlib.util.module_from_spec(spec) + monkeypatch.setitem(sys.modules, spec.name, module) + spec.loader.exec_module(module) + monkeypatch.setenv("DEER_FLOW_HOME", str(tmp_path)) + # The scripted bash tool has no sandbox; declare fresh-shell provenance so + # the verdict depends only on which run the evidence attributes. + monkeypatch.setattr(module, "_harvest_shell_persistence", lambda state: False) + yield module + module._shutdown_isolated_subagent_loop() + + +class ScriptedModel(GenericFakeChatModel): + def bind_tools(self, tools, **kwargs): + return self + + +def _bash_call(call_id: str, command: str) -> AIMessage: + return AIMessage(content="", tool_calls=[{"name": "bash", "id": call_id, "args": {"command": command}}]) + + +@pytest.mark.asyncio +async def test_latest_failing_run_under_a_reused_call_id_fails_tests_passed(executor_module, monkeypatch): + outputs = iter([FAILED_RUN, PASSED_RUN, FAILED_RUN]) + commands = [] + + def bash(command: str) -> str: + """Run a shell command.""" + commands.append(command) + return next(outputs) + + # The provider numbers calls per response: the third run reuses call_0. + responses = [ + _bash_call("call_0", "pytest tests"), + _bash_call("call_1", "pytest tests -q"), + _bash_call("call_0", "pytest tests"), + AIMessage(content="Tests were run."), + ] + model = ScriptedModel(messages=iter(responses)) + monkeypatch.setattr(executor_module, "create_chat_model", lambda **kwargs: model) + app_config = AppConfig.model_validate( + { + "models": [{"name": "offline", "use": "langchain_openai:ChatOpenAI", "model": "offline"}], + "sandbox": {"use": "deerflow.sandbox.local:LocalSandboxProvider"}, + "summarization": {"enabled": False}, + } + ) + executor = executor_module.SubagentExecutor( + config=SubagentConfig(name="tester", description="Offline test runner", skills=[]), + tools=[StructuredTool.from_function(bash, name="bash")], + parent_model="offline", + app_config=app_config, + thread_id="reused-call-id-thread", + user_id="reused-call-id-user", + acceptance_criteria=["tests_passed:pytest tests"], + ) + + result = await executor._aexecute("Run the test suite") + + assert result.status == executor_module.SubagentStatus.COMPLETED, result.error + assert commands == ["pytest tests", "pytest tests -q", "pytest tests"] + # Every run stays a separate entry, in execution order. + assert [(entry["tool_call_id"], entry["status"]) for entry in result.bash_executions] == [("call_0", "error"), ("call_1", "success"), ("call_0", "error")] + verdict = check_acceptance_criteria(["tests_passed:pytest tests"], bash_executions=result.bash_executions) + # The decisive latest run failed, so the leaf is a checked negative. + assert (verdict["leaves"][0]["checked"], verdict["all_hold"]) == (True, False), verdict + + +def test_reused_call_id_never_takes_another_commands_output(executor_module): + # A cancelled or timed-out subagent leaves its last call unanswered. Its + # reused id must not relabel the earlier subset run as the full suite. + messages = [ + HumanMessage(content="Run the suite"), + _bash_call("call_0", "pytest tests/unit"), + ToolMessage(content=PASSED_RUN, tool_call_id="call_0"), + _bash_call("call_0", "pytest tests"), + ] + + executions = executor_module._harvest_bash_executions({"messages": messages}) + + assert [entry["command"] for entry in executions] == ["pytest tests/unit"] + verdict = check_acceptance_criteria(["tests_passed:pytest tests"], bash_executions=executions) + assert verdict["all_hold"] is False, verdict + + +def test_merge_keeps_each_occurrence_of_a_reused_call_id(executor_module): + first = executor_module._harvest_bash_executions({"messages": [_bash_call("call_0", "pytest tests"), ToolMessage(content=FAILED_RUN, tool_call_id="call_0", id="result-1")]}) + later = executor_module._harvest_bash_executions( + { + "messages": [ + _bash_call("call_0", "pytest tests"), + ToolMessage(content=FAILED_RUN, tool_call_id="call_0", id="result-1"), + _bash_call("call_1", "pytest tests -q"), + ToolMessage(content=PASSED_RUN, tool_call_id="call_1", id="result-2"), + _bash_call("call_0", "pytest tests"), + ToolMessage(content=FAILED_RUN, tool_call_id="call_0", id="result-3"), + ] + } + ) + result = executor_module.SubagentResult(task_id="task", trace_id="trace", status=executor_module.SubagentStatus.RUNNING) + + result.update_bash_executions(first) + result.update_bash_executions(later) + + # The reused id's later run is appended, not written back over the first. + assert [(entry["tool_call_id"], entry["status"]) for entry in result.bash_executions] == [("call_0", "error"), ("call_1", "success"), ("call_0", "error")] + verdict = check_acceptance_criteria(["tests_passed:pytest tests"], bash_executions=result.bash_executions) + assert (verdict["leaves"][0]["checked"], verdict["all_hold"]) == (True, False), verdict + + +def test_summary_cache_keeps_one_slot_per_run_of_a_reused_call_id(executor_module, monkeypatch): + from unittest.mock import Mock + + from deerflow.subagents import acceptance_checks + + classify = Mock(wraps=acceptance_checks.summarize_test_output) + monkeypatch.setattr(acceptance_checks, "summarize_test_output", classify) + messages = [ + _bash_call("call_0", "pytest tests"), + ToolMessage(content=FAILED_RUN, tool_call_id="call_0", id="result-1"), + _bash_call("call_0", "pytest tests"), + ToolMessage(content=PASSED_RUN, tool_call_id="call_0", id="result-2"), + ] + cache = {} + + for _ in range(3): + executor_module._harvest_bash_executions({"messages": messages}, summary_cache=cache) + + # Each run is classified once; values chunks reuse both cached summaries. + assert classify.call_count == 2 + assert list(cache) == [("result", "result-1"), ("result", "result-2")] + + +@pytest.mark.parametrize("call_id", [None, ""]) +def test_a_call_without_an_id_records_no_evidence(executor_module, call_id): + # Pairing needs a call id. Without one the owner of a result is unknown, + # so the harvest records nothing and the leaf stays UNVERIFIED. + messages = [ + AIMessage(content="", tool_calls=[{"name": "bash", "id": call_id, "args": {"command": "pytest tests"}}]), + ToolMessage(content=PASSED_RUN, tool_call_id=call_id or ""), + ] + + executions = executor_module._harvest_bash_executions({"messages": messages}) + + assert executions == [] + verdict = check_acceptance_criteria(["tests_passed:pytest tests"], bash_executions=executions) + assert (verdict["leaves"][0]["checked"], verdict["all_hold"]) == (False, False), verdict diff --git a/backend/tests/test_node_test_acceptance.py b/backend/tests/test_node_test_acceptance.py index c60171a4d6..b77a49150d 100644 --- a/backend/tests/test_node_test_acceptance.py +++ b/backend/tests/test_node_test_acceptance.py @@ -355,7 +355,7 @@ def test_summary_cache_is_bounded_and_tracks_compaction(executor_module, monkeyp assert len(entries) == len(cache) == 20 assert classify.call_count == 20 executor_module._harvest_bash_executions({"messages": [ai, tools[-1]]}, summary_cache=cache) - assert list(cache) == ["call-24"] + assert list(cache) == [("call", "call-24")] assert classify.call_count == 20 executor_module._harvest_bash_executions({"messages": [AIMessage(content="compacted")]}, summary_cache=cache) assert cache == {}