From fb2830537a921cf6bd410eeffb74e9a10a0a8d30 Mon Sep 17 00:00:00 2001 From: LingYi Date: Fri, 9 Oct 2026 17:46:35 +0800 Subject: [PATCH] fix(agents): avoid executor starvation in file-gate waits (#6511) * fix(agents): avoid executor starvation in file-gate waits Keep async file-gate waiters out of the executor needed by the holder. Preserve sync and cross-loop exclusion, and drain started file work on cancellation. * test(agents): cover mixed file-gate callers Document why release wakes every async waiter. Cover a synchronous tool holder and asynchronous waiter through the middleware entry points. * test(agents): cover sync waiters released by async file tools Signed-off-by: LingYi --------- Signed-off-by: LingYi Co-authored-by: Willem Jiang --- README.md | 2 + .../deerflow/agents/middlewares/AGENTS.md | 6 +- .../read_before_write_middleware.py | 82 ++++- .../test_read_before_write_cancellation.py | 75 ++-- .../test_read_before_write_coordination.py | 339 ++++++++++++++++++ 5 files changed, 439 insertions(+), 65 deletions(-) create mode 100644 backend/tests/test_read_before_write_coordination.py diff --git a/README.md b/README.md index 4dc5db1d69..00d634c969 100644 --- a/README.md +++ b/README.md @@ -2086,6 +2086,8 @@ Each task gets its own execution environment with a full filesystem view — ski The read-before-write gate ties each read mark to that `read_file` call's result, including custom tools returning multi-message `Command` updates. An unrelated result cannot authorize a write after a failed read or hide a successful read. +Concurrent reads and writes to the same file share a gate across synchronous and asynchronous tool calls. Async callers waiting for that gate do not occupy worker threads needed to finish the current read or write. Cancelling a waiting call leaves the current operation running; a call that already started file inspection still waits for that work to finish before releasing its gate. + The built-in `grep` tool searches either one text file or all matching text files below a directory, so an agent can search an uploaded document directly without first broadening the request to the entire uploads directory. E2B's `glob` filter preserves spaces, quotes, and dollar signs in filename patterns, while wildcard matching and root-relative directory scoping remain unchanged. diff --git a/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md b/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md index 2af0ee0179..be2ade1c24 100644 --- a/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md +++ b/backend/packages/harness/deerflow/agents/middlewares/AGENTS.md @@ -1,5 +1,9 @@ ### Middleware Chain +Async gate waiters own separate cross-loop futures. Cancel only their signal; +notify outside the state lock. `_await_off_thread` drains started I/O across +repeated cancellation. + Table synopses count nonblank CSV/TSV logical records, excluding the header. Preserve the recognition sample and 5,000,000 UTF-8-byte guard. Count incrementally; on parsing or field-limit errors, report an undetermined count rather than a @@ -85,7 +89,7 @@ No match: skip inspection, log ID. Tests: `test_read_mark_tool_call_correlation. forgeries). Consumers pop it; the publisher and the consumer share only that contract module. 12. **SandboxAuditMiddleware** - Audits sandboxed shell/file operations before tool execution; command classification is **defense-in-depth and audit, not a security boundary** (the sandbox is the isolation boundary). Command substitution is judged by *position*, not the presence of `$(`: **command position** (`$(curl url)`, `` `curl url` ``, the word after `|`/`&&`/`;`, an `eval`/`source` argument) executes fetched content and is blocked; **value position** (`x=$(curl url)`, `echo $(curl url)`, an argument, a `for` word list) only captures output and passes (#4611). So `_HIGH_RISK_COMMAND_POSITION_PATTERNS` is matched anchored against each sub-command from `_split_compound_command(split_pipes=True)`, never the whole string; pipe-spanning rules (`| sh`, `base64 -d | ...`) still use `_classify_command`'s whole-command Pass 1. `_COMMAND_POSITION_PREFIX` extends the anchor over leading assignments and exec wrappers (`FOO=1 $(curl url)`, `env`/`command`/`builtin`/`exec`/`nohup`/`time`/`sudo`/`doas`); its assignment branch requires whitespace before the substitution, which keeps `x=$(curl url)` in value position. Two contexts are deliberately **position-blind** (matched whole-command in Pass 1, since they execute their input anywhere, e.g. `xargs sh -c "$(curl url)"`): an `eval`/`source` argument, and an interpreter **code-string flag** — `-c` (shells, `python`), `-e` (`perl`/`ruby`/`node`), `-p` (`perl`/`node`), `-r` (`php`) — plus the here-string (`<<<`) reaching the same place via stdin. All three substitution spellings (`$(`, `<(`, `` ` ``) share one `_RISKY_SUBSTITUTION` opener. An unquoted newline splits like `;` (else `echo hi\n$(curl url)` evades the anchored rules). Heredoc bodies are data: `_split_compound_command` records headers (`<