* fix(persistence): wait out a peer's Postgres bootstrap past command_timeout
The bootstrap advisory lock was taken with a blocking pg_advisory_lock on
the app engine, whose asyncpg command_timeout (30s by default) also bounds
that statement. A second Gateway instance therefore failed startup with
TimeoutError whenever another instance's migration outlasted the timeout.
Poll the non-blocking pg_try_advisory_lock instead: each attempt returns
immediately and stays bounded by command_timeout, while the wait lasts as
long as the holder's migration. The wait is logged once.
* docs(changelog): reference #6306 in the bootstrap lock entry
* docs(persistence): describe polled bootstrap lock in the migrations guide
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(harness): normalize local HTTP host case in skillscan
_LOCAL_HTTP_HOSTS holds lowercase names and every membership test
compared the extracted host verbatim, so an endpoint written with any
uppercase letter (e.g. http://LOCALHOST:8080) was classified as an
external endpoint: non-root deployments failed with EACCES while root
deployments silently wrote plans outside the workspace.
Normalize the host to lowercase in _http_host and in the
network-cleartext/local scan branch so all three call sites agree.
Fixes#6278
* test(skillscan): cover local host case and external endpoint controls
* fix(skillscan): parse IPv6 hosts and detect metadata case variants
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): serialize lifecycle mutations under per-channel lock
* fix(channels): invalidate stale config snapshots across lifecycle mutations under per-channel lock
* fix(projects): convert shelf documents outside the SQLite write lock
Lazy conversion ran pymupdf/markitdown inside the BEGIN IMMEDIATE
transaction that serializes the derived/converted.md publish against
trash and purge. On SQLite that lock is database-wide, so every unrelated
writer (run status, thread metadata, the scheduler) waited for the whole
conversion and failed with "database is locked" past busy_timeout (30s).
Conversion now writes into .staging/ outside any transaction. The
document row lock is taken only to revalidate the live row and atomically
rename the output into place (publish_under_live_lock, renamed from
convert_under_live_lock); a row trashed or purged meanwhile discards the
staged output, so nothing is published after purge. A read-only probe
skips converting rows that are already gone, a failed conversion is
still revalidated so a purged row reports content_missing, and the
conversion is drained on cancellation so its staging file is always
removed.
* docs(changelog): reference #6305 in the shelf conversion lock entry
* fix(projects): share one conversion across concurrent first reads
A first-read conversion holds a bounded file-IO pool thread for its
whole duration. With the conversion moved out of the row lock,
concurrent first reads of one document each converted on their own
thread, so a burst of reads could starve unrelated offloaded file work.
Route the conversion through a single-flight keyed by (user, document,
project): the first reader leads and the rest wait on its result
without holding a pool thread. The map holds concurrent.futures.Future
values under a threading.Lock because the read tool also runs on the
isolated subagent loop. Followers wait through asyncio.shield, so a
cancelled follower cannot cancel the shared future, and a leader that
is cancelled or raises hands the work to a follower instead of failing
it.
* fix(skills): stop reporting an env path, shebang or flag as an env dump
* fix(skills): report shell env dumps only at a command position
`shell-env-dump` matched the word `env` anywhere it was not preceded by a path, dot, dash or word character, so ordinary argument text (`echo env`) and comment text (`# env`, `# export -p`) were still reported as environment dumps. Anchor the match to a shell command position -- the start of a line or right after a `;`, `&`, `|`, `(`, `)` or `{` separator, optionally behind `NAME=value` assignment prefixes -- and add regression cases for argument and comment text.
* fix(skills): ignore an env dump inside a comment or heredoc body
`shell-env-dump` matches `env`, `printenv` and `export -p` at a command
position, but the text it sees is still raw shell. A `;` inside a comment
(`# documentation; env is only an example`) and a bare `env` line inside
heredoc data both sit at a command position on paper while the shell runs
neither, so both were reported as environment dumps.
Reduce the text to shell code before matching: a quote-aware pass strips
comments (only a `#` at a word start opens one) and blanks heredoc bodies,
covering `<<-` tab stripping and quoted delimiters, while keeping one output
line per input line so reported line numbers stay correct. Real commands
after a separator (`true; env`) and on the line that opens a heredoc
(`env <<EOF`) still report.
* fix(skills): report env dumps after a keyword and stop masking ${#...}
The command-position anchor for `shell-env-dump` accepted only a line start or a
`;`, `&`, `|`, `(`, `)` or `{` separator, so real dumps were dropped: after a
reserved word that introduces a command (`if true; then env; fi`,
`while true; do printenv; done`), after `exec` (`exec env`) and inside a backtick
substitution (``V=`env` ``). Accept those positions too, with a guard so a path
component (`./then env`) does not arm the anchor, and add the backtick to the
separator set.
A `{` directly before a `#` is not a word start -- bash reads `${#HOME}` as the
length operator and `}#` as part of a word, while a brace group needs the space
of `{ # ...`. The comment heuristic no longer treats `{` and `}` as comment
openers, so `n=${#HOME}; env` keeps its `; env` and is reported again, while
`${env}` (a variable read, not a command) stays silent.
---------
Co-authored-by: sxh313 <sxh313@users.noreply.github.com>
* fix(agents): keep malformed todo statuses recoverable
* docs(agents): fit todo guidance within inherited size limit
* docs(agents): clarify todo attribution guidance
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(browser): drain live cleanup before releasing session
* test(browser): cover cancellation precedence during cleanup
* style(browser): separate cancellation regressions
* fix(gateway): scope per-run message and event reads to the run's data identity
start_run stamps run-event rows with the raw trusted owner id, but
GET /runs/{rid}/messages and /runs/{rid}/events called the event store with
user_id=AUTO, which resolves to the internal caller's make_safe_user_id form.
On the database event store, owners whose id contains characters outside
[A-Za-z0-9_-] (IM channel ids such as "feishu:owner-777") therefore got an
empty list. The regenerate source-run scan had the same mismatch and fell
back or returned 409.
Pass _run_scope_user_id() explicitly, as the thread message routes already
do (#5448), and add user_id to RunEventStore.list_messages_by_run() so every
backend accepts it like list_messages()/list_events(). Browser and API
sessions keep their per-user filter.
* fix(gateway): scope workspace-changes and archive reads to the run's data identity
The same raw-owner mismatch reached two more run-scoped routes. On the SQL
stores, GET /runs/{rid}/workspace-changes reported no changes because its
event lookup used user_id=AUTO, and GET /runs/{rid}/artifacts/archive
answered 404 because both its run-row lookup and its run.delivery event
lookup did.
Pass _run_scope_user_id() to both, and let get_workspace_changes_response()
forward an explicit user_id to the event store.
* docs(changelog): reference #6282 in the run-scope read fix entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(runtime): keep renewing the lease while a staged terminal run finalizes
With an event store, run_agent stages the terminal status in memory and
commits it to the run store only after the journal flush, delivery receipt,
workspace scan and duration checkpoint. _renew_leases selected runs by local
status, so it stopped renewing as soon as the status was staged. A
finalization longer than lease_seconds + grace_seconds let a peer, or this
worker's own reconciler, claim the still-active row and record the
successful run as an orphan error.
The worker now marks RunRecord.terminal_commit_pending until the deferred
commit is attempted, and lease renewal treats it like the existing
scheduled-goal barrier: keep renewing, release on a confirmed own commit,
and fence when a peer claims the row. A fenced run stops renewing.
* docs(changelog): reference #6263 for the staged terminal lease renewal fix
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): report secret assignments whose key is quoted
The line-oriented secret sweep required the credential name to be followed
directly by ':' or '=', so a JSON config ('{"api_key": "..."}') never matched
while the same binding in YAML, .env, .ini or shell did. Allow one optional
quote between the key and the separator.
* fix(skills): match secret assignments whose key has a separator prefix
The line-oriented sweep anchored the credential word with `\b`, which cannot
land after `_` because `_` is a word character. So `access_token`,
`client_secret` and `MY_API_KEY` — the common spellings in OAuth and
service-account config — matched nothing, and `_SECRET_TOKEN_PATTERNS` does
not backstop JWT- or `GOCSPX-`-style values, so such a package produced no
finding at all.
Guard the word with "not preceded by an alphanumeric" instead: a separator
(`_`, `.`, `-`) may now introduce it, while a run of letters (`tokenizer`,
`secretive`, `credentials_file`) still stays quiet.
---------
Co-authored-by: sxh313 <sxh313@users.noreply.github.com>
* fix: include completed child usage in lead budget checks
* test: require completed usage middleware in lead budget regressions
* docs: fit middleware guidance within the size budget
Condense the subagent, loop, and token budget guard descriptions while retaining their limits, lifecycle rules, and completed child usage ordering.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
is_eval_fixture_path returned on the first 'evals' segment, so a path whose
'evals/fixtures/' came after a plain 'evals' segment read as a non-fixture.
Move the 'fixtures' test into the loop condition so every segment is checked.
Co-authored-by: sxh313 <sxh313@users.noreply.github.com>
* fix(channels): follow Buzz threads mapped in the connection repository
With channel_connections enabled, ChannelManager stores a bound author's
thread mapping only in the connection repository, but Buzz's thread-follow
gate read only the JSON ChannelStore. engaged_thread was therefore always
False for authors bound with /connect, and every unmentioned reply in a
thread the bot was already answering was silently dropped.
Buzz now resolves the connection identity before the mention gate and looks
the thread up through lookup_thread_id, a helper extracted from
ChannelManager._lookup_thread_id that both sides now share, so the gate and
the manager can no longer disagree about which store owns a mapping.
* fix(channels): drop the manager's own JSON-store read in the skill whitelist
_resolve_available_skill_names fell back to reading the JSON ChannelStore
when the caller's lookup_thread_id found nothing. For a bound message that
lookup is repo-only, so a legacy unbound mapping for the same chat decided
which custom agent's skill whitelist applied. The parameter is now required
and the caller passes "" when nothing is mapped.
Also pin the Buzz thread-follow fix against the real SQL connection
repository (bound through a real /connect) instead of a fake.
* docs(changelog): reference #6232 for the Buzz thread-follow fix
* fix(channels): log Buzz drops of unmentioned replies with no engaged thread
The thread-follow gate was the one drop in _handle_chat_event without a
log line. Log it at DEBUG with the thread root and whether the author was
bound, matching the allowlist drop, so a report that the bot stopped
following replies can be told apart from a lost bind or missing mapping.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Path(client).name == "docker" misses the Windows CLI (docker.EXE), so
every real-compose test routed through the standalone-Compose wrapper,
whose shift strips the compose subcommand; the root CLI then rejected
the script's --env-file probe with "unknown flag: --env-file". Compare
the stem instead so the CLI is recognized on Windows too.
Root cause identified by @xihongshichaojidan8 in #6259.
Fixes#6261
Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com>
* feat(extensions): add full Custom Agent team collaboration plugin
* fix(extensions): register agent teams in the capability catalog
* fix(agent-teams): recover incomplete teams and preserve request capacity
* fix(agent-teams): preserve clarification waits and active handoff lineage
* fix(agent-teams): align mention and input text limits
* fix(gateway): read uvicorn's WEB_CONCURRENCY in the remaining worker-count readers
* fix(gateway): name the worker spelling an operator can act on
The browser refusal and the dedupe warning both took the count but printed
GATEWAY_WORKERS, so on the launches that pass uvicorn no --workers the advice
silences the gate while the extra processes keep starting. Resolve which
variable set the count and print that, and let an unparsable GATEWAY_WORKERS
fall through to WEB_CONCURRENCY instead of reporting a single worker.
Also stop the comments from claiming the readers are equivalent: they share
the two spellings and the blank-means-unset rule, but channel_connections.py
turns an unparsable value into 0 and refuses while these stay inert.
* docs: state the single-worker requirement under both spellings
The browser tool group and the inbound dedupe warning now refuse or warn on
WEB_CONCURRENCY as well, so guidance that names only GATEWAY_WORKERS=1 is
incomplete for the shipped Dockerfile and serve.sh launches, where uvicorn
takes the process count from WEB_CONCURRENCY.
* test(features): isolate the browser capability surface from the shell's worker count
browser_capability() asks for the Gateway worker count, which this change makes
readable from WEB_CONCURRENCY, so an exported count in the invoking shell turned
test_features_reports_browser_control_enabled_when_configured_and_runtime_available
into a failure that has nothing to do with the router.
* fix(gateway): read uvicorn's WEB_CONCURRENCY in the multi-worker safety gates
The SQLite/scheduler gates in deps.py derived the worker count from
GATEWAY_WORKERS alone, but backend/Dockerfile and scripts/serve.sh launch
uvicorn without --workers, and uvicorn then takes the count from
WEB_CONCURRENCY. A Gateway scaled that way starts N processes with every
gate inert, which is exactly what the gates exist to prevent.
Read both variables, preferring GATEWAY_WORKERS, treat a blank value as
unset the way the compose command's ${GATEWAY_WORKERS:-1} does, and name
the variable that actually set the count in the refusal text. The channel
connection gate already reads both (channel_connections.py).
* fix(gateway): fall through to the fallback worker count
A non-numeric GATEWAY_WORKERS made the shared reader return one worker
immediately, so on the launches that pass no --workers (backend/Dockerfile,
scripts/serve.sh, backend/Makefile gateway) an operator's
GATEWAY_WORKERS=abc plus WEB_CONCURRENCY=4 started four processes while every
gate stayed inert. Keep looking at the remaining spellings and report a single
worker only after all of them have been tried.
The gate file also needs to be independent of the invoking shell: with
WEB_CONCURRENCY exported, the pre-existing
test_gate_treats_invalid_env_as_single_worker already failed on this branch,
because the reader now consults that variable too.
* feat(runtime): add P0 per-LLM-call telemetry and summarization no-op counter
Measurement only; no runtime behavior changes.
- RunJournal enriches llm.ai.response / llm.error metadata with
langchain_run_id, caller_category, provider, model, stop_reason, status,
token counts and rendered request size (chars). Unavailable values are None.
- Summarization middleware emits a middleware:summarize event per summarizer
call and counts outputs identical to the prior summary_text. The call is
never skipped.
- All telemetry fails soft.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* fix(runtime): register summarize middleware tag, document telemetry contract
- Add MIDDLEWARE_SUMMARIZE_TAG to the middleware event catalog and the
contract's known_tags; emit the middleware:summarize event with it.
- Document the optional per-call telemetry metadata on llm.ai.response and
llm.error in contracts/run_event_stream_contract.json.
- Measure request size by walking string leaves instead of json.dumps on the
event-loop thread.
- Initialise summarization call/no-op counters in __init__.
- Move RunJournal telemetry docs to the runtime guide.
- Add tests: llm.error without a start, replayed on_llm_end, and a DB-store
metadata round trip.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* fix(runtime): refresh token counters on usage replay
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
#6229 gave the scheduler its own integer knobs, but they accept YAML
booleans: recursion_limit: true loads as 1, so every scheduler-launched
run dies at its first LangGraph super-step - the same hole the
app-config guard already closes for Gateway runs (#6171), now on the
same-named field. max_concurrent_runs: true serializes the scheduler
and poll_interval_seconds: true polls five times a second.
Mirror the app-config boolean guard: a before-validator rejects booleans
with a field-specific error across all six integer fields, keeping
integers and numeric strings as-is. lease_seconds and
queue_timeout_seconds already reject true via their higher ge floors,
but with a misleading "should be greater than or equal to" message -
the guard gives every field the same boolean-specific error.
Signed-off-by: yetuge <2219677952@qq.com>
YAML booleans coerce into the ge=1 integer backstops that bound every
subagent and ACP invocation: timeout_seconds: true became 1 (the executor's
future.result(timeout=...) then killed every subagent run after one second)
and max_turns: true became 1 (the LangGraph recursion budget collapsed to a
single super-step). The ACP agent timeout mirrors subagents.timeout_seconds
and shared the hole. false was already rejected by ge=1, so only true
slipped through - silently. Mirror the app-config boolean guard: reject
bools before coercion in SubagentOverrideConfig, CustomSubagentConfig and
SubagentsAppConfig (timeout_seconds/max_turns/max_total_per_run) and
ACPAgentConfig (timeout_seconds), keeping None, integers, and numeric
strings as-is.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(memory): treat facts deleted during unlocked DeerMem scans as removed
load(), reload(), and the full rebuild_index() scan list agent fact
files without the storage locks. A delete committed between the listing
and opening a file raised MemoryStorageCorruption for intact data, which
surfaced as HTTP 500 from the memory API and an empty memory block in
the prompt. A listed fact that no longer exists is now skipped; an entry
that is still present but unreadable is still corruption.
* docs(changelog): reference #6255 for the DeerMem vanished-fact fix
* docs(memory): note lock coverage and heuristic limits of the vanished-fact skip
* fix(skillscan): report secrets bound by unpacking and container literals
The AST walk behind secret-env-assignment claims "precision, never less
coverage", but it only read a single assignment target and a bare string
value, so a credential moved into `host, api_key = endpoint, "..."` or
into `api_key = ["..."]` stopped being reported even though the pre-#5648
line sweep still reported the same bytes.
Pair unpacking targets with values positionally (fixed head and tail only,
so a runtime-sized slice names nothing) and assert on each literal a
container or conditional holds, in source order.
* fix(skillscan): distinguish mapping labels from credentials
---------
Co-authored-by: sxh313 <sxh313@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The channel runtime config store's atomic 0o600-temp + replace + chmod
cannot land a POSIX mode on Windows, where chmod is a no-op
approximation. Skip the two POSIX-bound tests and guard the trailing
mode assertion of the chmod-failure test, keeping its platform-
independent log-behavior coverage everywhere.
Fixes#6256
Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com>
Four of the eight packages/harness/deerflow citations in this file name modules
that are not in the tree: sandbox/local.py, mcp/manager.py, skills/loader.py,
and tools/__init__ (missing its extension). The LocalSandboxProvider class lives
under sandbox/local/, the MCP and skills subsystems are packages rather than
single manager/loader modules, and get_available_tools is re-exported by
tools/__init__.py.
* refactor(config): converge the boolean guards into one shared helper
Willem-bd's #6171 review follow-up: a tiny shared helper lets the next numeric field get the guard in one line and unifies the messages. Consolidates 14 validator copies across 9 config files into deerflow/config/_boolean_guards.reject_boolean(value, info, kind=...); the bare per-file messages gain their field name prefix, command_timeout keeps its distinct wording.
Signed-off-by: yetuge <2219677952@qq.com>
* refactor(config): converge stream_bridge_config and pin the guard wording
Review follow-ups on #6226: convert the last surviving per-file copy (stream_bridge_config) so deerflow/config is fully converged, make kind required keyword-only so a float field cannot silently inherit integer wording, and pin the unified field-prefixed messages with direct tests.
Signed-off-by: yetuge <2219677952@qq.com>
* test(config): cast the boolean-guard test probe to ValidationInfo
The _info probe was annotated -> object, so a type checker flags every call
site that passes its result to reject_boolean's info: ValidationInfo
parameter (runtime was fine since the helper only reads .field_name).
Return cast(ValidationInfo, _Probe(field_name)) to keep the probe honest
for both.
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* fix(memory): sign DeerMem reload before reading the document
reload() read the document before computing its cache signature, so a
write committed in between cached the old document under the new
signature and later load() calls served stale memory until the next
write. Compute the signature first, matching load().
* test(memory): cover markdown storage and agent scope in reload race test
* docs(changelog): reference #6238 for the DeerMem reload fix
