Why this change:
In Lark broker mode the shim forwards piped stdin to the broker only
after EOF. AIO's /v1/bash transport keeps a subprocess stdin pipe open
for writes, so a bare `lark-cli ...` never saw EOF and the shim blocked
forever on input that would never arrive. An earlier brace-group/eval
approach fixed the hang but changed top-level parsing (aliases/extglob)
or let a trailing backslash consume the appended delimiter.
What changed:
- _run_bash_exec prefixes the original command with `exec < /dev/null`
on its own line, giving the fresh, per-invocation session immediate
EOF on default stdin. Explicit pipes/heredocs/files still override
fd0. The plain prefix keeps top-level parsing and appends no
delimiter a trailing backslash can consume.
- The persistent /v1/shell PTY transport keeps commands unchanged; the
shim already treats TTY stdin as no input.
- The shim drains non-TTY stdin under grace/tail idle budgets
(DEERFLOW_LARK_BROKER_STDIN_{GRACE,TAIL}_SECONDS, default 2s, valid
(0, 600]); an idle pipe exits 124 without contacting the broker, and
read errors fail loudly instead of becoming empty input.
- Broker logs each exec as argc/exit/elapsed only (never argument
values) and warns on concurrency-cap rejections.
Problem solved:
Lark commands without explicit stdin hung indefinitely in broker mode,
stalled producers could execute partial writes, and broker logs risked
capturing argument values. Docs updated: README, harness AGENTS.md,
docker/lark-cli-broker README.
* fix(cli): exit non-zero when a headless run ends in an LLM error fallback
LLMErrorHandlingMiddleware turns provider failures (e.g. an expired
credential returning 401) into an AIMessage flagged
`deerflow_error_fallback` instead of raising. The Gateway run status and
the subagent executor already treat that flag as a failure. #5963 gave
`deerflow --print` / `--json` an error boundary for raised exceptions,
but a flagged fallback never raises, so both modes still printed the
fallback text and exited 0.
Both headless modes now follow #5963's failure contract when the run's
final AI message carries the flag: `--print` keeps the fallback text on
stdout and adds an `Error:` line on stderr; `--json` appends the terminal
`{"type": "error"}` record. Both exit 1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* refactor(cli): share final-answer accumulation with the embedded client
* docs: scope headless accumulator guidance to TUI
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(authz): enforce skill authorization at assembly and slash-activation (#4063 Phase 3)
Phase 3 / Skills — the second of three resource-type PRs (Models, Skills,
Sandbox). The RBAC provider already maps "skill" → config key "skills"
(rbac.py _RESOURCE_POLICY_KEYS), so no schema change is needed.
Layer 1 (assembly-time, mirrors Phase 1B):
- filter_available_skills_by_authorization() in skill_filter.py filters the
skill-name allowlist (set[str] | None) by the provider's "skill" policy.
In lead agent, runs after _available_skill_names() so the catalog
(describe_skill) and SkillActivationMiddleware share one filtered set.
In subagent executor _load_skills(), filters config.skills before disk load.
- filter_resources_by_authorization() in enforcement.py — generic version of
filter_tools_by_authorization for any resource with a name attribute.
Layer 2 (slash-activation):
- No new middleware code needed. SkillActivationMiddleware._resolve_activation
already checks "reference.name not in self._available_skills". Since Layer 1
filters the allowlist, denied skills cannot be slash-activated.
authorization.enabled: false is a complete no-op. available_skills=None (no
agent allowlist) + enabled: resolves all skill names from config and filters.
12 new tests + 243 existing authz/skills tests pass.
* test(subagents): wait on the chained future instead of polling done() in caller-loop teardown test
* feat(authz): enforce action-scoped skill:activate at runtime and reuse the catalog loader for filter candidates
* feat(authz): arm skill:activate on the subagent chain, unify the client skill-surface user, and record the Layer 2 reversal in the design notes
* chore(authz): align the subagent filter's fallback user with the chain's bucket convention
* feat(authz): gate autonomous skill loading on skill:activate, close the bootstrap assembly leak, and reuse one provider per subagent assembly
Review round 4 (willem-bd, CHANGES_REQUESTED):
- [P1] Autonomous load path: describe_skill filters its match set on the
action-scoped skill:activate decision (all-denied is indistinguishable
from "no match"), and ToolErrorHandlingMiddleware stamps skill_context
entries only when the action allows it — a denied SKILL.md read gets a
skill_context_denied marker instead, so durable context, allowed-tools
policy, and autonomous secret bindings never activate a denied skill.
File visibility itself stays Layer 1 (sandbox-projection semantics).
- [P1] Bootstrap assembly: build_middlewares/apply_prompt_template now
receive the authorization-filtered set instead of the raw
_BOOTSTRAP_SKILL_NAMES, and an empty skill_names index is preserved
when deferred discovery is on rather than round-tripping through None
into the legacy full-metadata prompt that re-advertises a denied
skill (deferred-off keeps the legacy rendering; the filtered empty
allowlist blanks it on denial).
- [P2] SubagentExecutor resolves ResolvedSkillAuthorization once per
assembly (_resolve_skill_authorization, memoized per executor) and
shares it across the Layer 1 filter (authorization=), the activation
middleware, and both new gates — uncached provider factories can no
longer hand the layers different instances/policy snapshots.
All gates share one helper (skill_filter.skill_activation_allowed) so
slash, describe, and skill-file-load cannot drift; provider errors keep
the configured fail-closed/fail-open policy. skill_authorization is
threaded through _build_runtime_middlewares into both runtime builders.
Tests: describe gating (deny/fail-closed/fail-open/disabled), read
stamping (denied marker vs entry) + extract_skills skip without warning,
bootstrap three-way param (allow/deny x deferred on/off), lead + subagent
chain wiring for the stamp gate, and the distinct-instance provider
factory repro. Each fix is mutation-verified (reverting it fails its
test). Full A/B suite run: no new failures vs HEAD.
* feat(authz): re-authorize persisted skill_context before reuse and route async paths through aauthorize
Review round 5 (willem-bd):
- [P1] Persisted entries: skill_context survives across runs, so both
consumers now re-check the action-scoped skill:activate decision before
reuse. SkillToolPolicyMiddleware._active_skills_for_paths skips a
denied skill (same skip as a disabled/unallowlisted one; all-denied
keeps the existing fail-closed-to-builtins semantics) and
_in_context_secret_sources binds no secrets for a denied entry.
skill_authorization is threaded into SkillToolPolicyMiddleware on both
chains.
- [P2] Async provider API: skill_activation_allowed_async mirrors the sync
helper through await provider.aauthorize() (same pattern as
authorize_sandbox_execution_async). Async hooks precompute an
activation_decisions map on the event loop (slash target + persisted
entry names, one batch of awaits) and hand it to the worker-thread
handler, which consults the map instead of the sync API:
SkillActivationMiddleware.awrap_model_call (activation + secret
binding), ToolErrorHandlingMiddleware.awrap_tool_call (_amaybe_stamp
awaits the decision before stamping), and SkillToolPolicyMiddleware's
awrap_model_call/awrap_tool_call (entry names + slash source name).
describe_skill becomes a StructuredTool with both a sync function
(invoke -> authorize) and a coroutine (ainvoke -> aauthorize), so async
agent execution never routes a loop-affine provider through the sync
API while sync callers keep their semantics.
Tests: allow-read-then-deny-next-run regressions for secret binding and
allowed-tools (partial deny does not poison, all-denied fails closed to
builtins), and an _AsyncOnlyProvider (sync authorize always fails,
aauthorize works) driving four async-path assertions (activation, stamp
allow/deny, describe, tool policy). All six fixes are mutation-verified;
full A/B suite run against HEAD shows no new failures.
* fix(authz): authorize skill activation by declared Skill.name, not directory basename
ShenAC-SAC's P2: bundled skills may declare a name that differs from their
directory (skills/public/vercel-deploy-claimable declares name: vercel-deploy),
but the runtime gates derived authorization targets from the container path.
- New deerflow/skills/container_registry.py: build_container_path_registry()
+ canonical_skill_name() — the one path-to-Skill.name resolution shared by
the activation, tool-policy, and read-stamping middlewares.
- ToolErrorHandlingMiddleware resolves the read path through the (user-scoped,
via new user_id kwarg threaded from both runtime builders) registry before
the skill:activate decision, sync and async (registry lookup in a worker
thread, aauthorize() on the loop). RBAC allow by the declared name now
activates; allow by the directory name does not.
- SkillToolPolicyMiddleware async hooks canonicalize the policy paths first
(three-phase: thread registry -> loop aauthorize -> thread filter reusing
the registry; the cached per-model-step decision still short-circuits),
so the decision map is keyed by the names _active_skills_for_paths checks —
no more map miss falling back to the sync provider API from the thread.
_entry_names removed: persisted entry["name"] is path-derived by
construction; entries are resolved by path instead.
- SkillActivationMiddleware collects slash names (already canonical) plus
persisted entry *paths*, canonicalized off-loop via
_canonical_names_for_paths().
Regressions (test_skills_authorization.py 45->50): sync stamp target, RBAC
allow declared-name/deny directory-name both directions, async stamp via
_AsyncOnlyProvider, composed async slash -> tool-policy keeps the skill's
tools with zero sync fallback, persisted-entry secret binding survives the
name mismatch. All three mutations (stamp target, policy map key, entry
canonicalization) fail their tests.
* fix(authz): close the sync-fallback class on skill activation paths
willem-bd's two P2s on the canonical-name fix, plus a fourth instance of
the same class found while sweeping the membership:
- SkillToolPolicyMiddleware: a transient prepass registry failure is now
preserved (_REGISTRY_LOAD_FAILED marker) — the worker-side filter applies
its own fail-closed treatment instead of silently retrying storage,
which recovered skills absent from the decision map and fell back to
the synchronous provider API from the thread (fail-open retaining a
denied skill's allowed-tools).
- SkillActivationMiddleware: an empty entry-path list no longer pays a
full skill-tree scan on ordinary authorization-enabled async model
steps (previously a passive path with zero registry I/O).
- 4th instance (reproduced with the PR fixtures, then fixed): the entry
prepass failure on the secret-binding path let the consumer-side
_load_skill_registry_by_path silently recover and resolve entry names
outside the decision map — the same sync fallback, fail-open binding
the aauthorize-denied skill's declared secret. The fix goes one step
further than failure propagation: the prepass hands down the registry
*snapshot* the decision map was keyed by, and entry sources resolve
against that snapshot, so a map miss is impossible by construction
(this also closes the narrower window where both loads succeed but
disagree because storage changed mid-step). The slash source is
deliberately unaffected: it does not consult the decision map
(validated once at activation) and keeps its fresh per-call load.
- Class membership swept: skill_activation_allowed() sync call sites are
exactly four (stamping sync path and describe's sync implementation are
legitimately sync); the only two async-reachable fallback points (the
_activation_allowed helpers) are both construction-guarded now.
Structural hardening (typed async-batch decision object where a map
miss is fail-closed by construction) recorded in the plan notes as a
follow-up, anchored at those two helpers.
- _RegistryArg type alias replaces the degenerate dict|object|None union
in the policy middleware signatures.
Tests (test_skills_authorization.py 50->54): transient prepass failure ->
builtins-only with exactly one storage attempt and zero provider calls;
ordinary async step -> zero registry scans; two-call scenario -> entry
binding zeroed while the slash binding survives; entries resolve against
the prepass snapshot under a mid-step rename. Mutations verified red:
marker reverted to None, over-broad slash kill, snapshot bypass.
* fix(authz): correct secret-binding reference points and gate the rendered skill reminder
Review round 9 (willem-bd) plus construction-audit follow-ups on the same
class — every async divergence between two loads of the same data now has
an explicit, pinned reference point:
- Slash secret source resolves from a FRESH post-activation registry: the
prepass snapshot is taken before the aauthorize awaits, so binding the
slash source from it would activate NEW_KEY content while injecting
OLD_KEY when a skill's declared secrets change in that window. The
'one scan per step' bonus that motivated sharing the snapshot is
withdrawn — consistency wins. The two secret sources resolve
independently: a transient fresh-load failure zeroes only the slash
binding; entries keep binding from their own snapshot.
- Same-skill dominance: when a skill is both the slash source (fresh) and
a persisted entry (snapshot), only the activation-era declaration binds
— unioning both eras injected a secret the skill no longer declares.
Aligns with the tool-policy rule (slash activation dominates for the
run); different-skill entries still stack additively.
- The rendered 'Active skills' reminder (DurableContextMiddleware) no
longer advertises skills the provider denies — surface consistency with
describe_skill. Publication/consumption split: the activation
middleware publishes per-step path-keyed skill:activate decisions into
the run context (async: reused from the prepass snapshot, zero extra
I/O; sync: one scan + authorize() only when entries exist — passive
steps stay scan-free), the renderer consumes them and filters its
render copy (state untouched; unpublished = permissive, matching the
other run-context carriers). New carrier key listed in
REDACTED_CONTEXT_KEYS with a redaction regression.
- Wiring regressions on both chains for everything added above (user_id
to the stamp gate, skill_authorization + token pairing + activation-
before-durable ordering to the durable renderer): each mutation
verified red (M4-M13 across this change set).
- Tests 54->65: slash post-activation lookup, source independence,
dominance, sync-chain sync-API path, render filter (async/sync/
disabled with load-count assertions), chain wiring x2, redaction
listing. Plan notes record rounds 9-2..6 including the reference-point
post-mortem and the audit axes exhausted.
* fix(authz): anchor same-skill dominance on the slash identity, not binding results
Review round 10 (willem-bd) plus a self-audit extension the new checklist
items caught in the first fix itself:
- R10 (P2): the same-skill exclusion was derived from successfully BOUND
slash sources. When the prepass snapshots required-secrets [OLD_KEY]
and the operator strips the declaration during the await window, the
slash source binds nothing (required_secrets gate) — the exclusion set
stayed empty and the stale entry view kept injecting OLD_KEY for a
skill that now declares no secrets. The exclusion now derives from the
AUTHENTICATED slash activation identity (run-context source path,
owner-token checked): the identity's declared name resolves from the
fresh registry first, then the entry snapshot, and joins the exclusion
set regardless of binding success.
- Self-audit (value->gate->downstream on the fix itself): the declared
name can also change mid-run (same path, foo->bar). A name-anchored
exclusion then holds only the new name while the snapshot entry
resolves the same path to the old one — both eras bound again. The
PATH is the stable identity shared by the activation and every earlier
read: entries at the slash identity's path are now excluded before
registry resolution (exclude_paths), independent of what either
version calls the skill; the name-based set remains for cross-path
same-name shadowing.
Tests 65->67: no-secrets-era dominance (ACTIVE secrets None under the
OLD_KEY-to-empty interleaving), path-anchored dominance across a mid-run
rename (only the activation-era key binds). Mutations verified red:
identity-derived exclusion disabled (M14), path exclusion disabled
(M16).
* test(authz): pass explicit config to lead middleware wiring test
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): drain agent and user-profile writes across cancellation
A client that disconnects mid-request cancels the handler task. On the four
persistent-write endpoints of /api/agents - create, update, delete, and the
USER.md write - a bare asyncio.to_thread then either cancels a still-queued
worker (the write silently never happens) or detaches from a running one and
drops its failure, because the handler's except never runs.
Route the four writes through await_drained like the managed-subagent (#6023),
managed-model (#6024), and custom-skill (#6078) mutations, logging a lost
worker failure with the exception type only before the drain consumes it.
Expected domain errors (AgentExistsError on create) re-raise unlogged; the
caller still maps them to a 409 while connected. Reads stay bare: abandoning
them loses nothing.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(gateway): address review nits on the agents write drain
Make expected_errors a positional-only parameter before *args so the
ParamSpec signature is checker-clean under PEP 612 (the one call site with
extra worker arguments passes () explicitly), acknowledge that
non-cancelled failures log twice like artifacts.py's drained commit does,
and drop the unused request stub helper from the tests.
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* fix(agents): anchor agent-name validation against a trailing newline
Two copies of the shared ^[A-Za-z0-9-]+$ grammar still used re.match,
whose $ also matches before a final newline, while every other copy uses
fullmatch. POST /api/agents {"name": "reviewer\n"} therefore passed the
router guard and died inside the strict file store as an HTTP 500, and
DeerMem accepted the same name as an agents/{name}/facts directory.
* test(agents): state which trailing-newline params the regression actually covered
.match only accepted "reviewer\n" on main; "reviewer\n\n" and "reviewer \n"
were already rejected, so the docstrings should not read as if every param in
these parametrizations was broken before the fix.
* docs: replace nonexistent async client API in harness docs
DeerFlowClient has no async methods: client.astream() and client.ainvoke()
do not exist, so the "Create Your First Harness" tutorial and the harness
integration guide fail with AttributeError on their first call. Overrides
were also shown as a nested config={"configurable": {...}} dict, which
stream()/chat() swallow via **kwargs and silently ignore.
Switch the examples to the shipping API (sync stream()/chat(), flat kwarg
overrides, agent_name as a constructor arg) in the EN and ZH copies of both
pages. backend/docs/STREAMING.md is left alone: its astream mentions are
design discussion and LangGraph internals, not user-facing examples.
Assisted-by: Claude Code / claude-opus-5
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Anton Dziatkovskii <194927794+tonydzi@users.noreply.github.com>
* docs: serialize StreamEvent before StreamingResponse, use real event names
Review follow-up on the two points raised in #5843.
P2 — the streaming examples handed a generator of `StreamEvent` dataclasses
straight to Starlette. `StreamingResponse` calls `.encode()` on any chunk that
is not `str`/`bytes`, so the first event raised
`AttributeError: 'StreamEvent' object has no attribute 'encode'`. Both
occurrences (the `run_agent` snippet and the inline FastAPI route, which shipped
the dataclass `repr()` through an f-string) now serialize to SSE frames built
from `event.type` / `event.data`.
P3 — the tutorial showed mapping literals with `messages` and `thread_state`,
neither of which exists. The real contract is
`StreamEventType = Literal["values", "messages-tuple", "custom", "end"]`
(backend/packages/harness/deerflow/client.py:128) and the stream yields
dataclasses, so the section now shows `StreamEvent(...)` with the real names
plus a consuming loop.
The zh tutorial had no equivalent section at all; added for parity.
Assisted-by: Claude Code / claude-opus-5
Machine: A-Mac16-2019-PaloAlto
Account: tonydzi
Operator: robot:pr-reply-daily
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: drop the nonexistent load_config() from every harness snippet
Review follow-up on #5843 (@willem-bd): the rewritten tutorial still opened
with `from deerflow.config import load_config`, which raises ImportError
before `client.stream(...)` is ever reached — `deerflow/config/__init__.py`
exports `get_app_config` and friends, and the string `load_config` does not
appear anywhere under `backend/packages/harness/deerflow/` (the only near
match is `load_configured_extension_middlewares`).
The call was also unnecessary: `DeerFlowClient.__init__` already resolves
configuration itself (client.py:220-222 — `reload_app_config(config_path)`
when a path is given, otherwise `get_app_config()`, which auto-loads from
the default path or DEER_FLOW_CONFIG_PATH).
Fixed the class rather than the one snippet you flagged: the same dead
import appeared 10 times across 4 pages (EN + ZH tutorial and integration
guide). Every one is gone. The "configuration in embedded mode" section now
shows the API that exists — `DeerFlowClient(config_path="...")` — instead of
`load_config(config_path="...")`.
Also from the review:
- the `end` example showed only `total_tokens`; `stream()` always yields
input/output/total (client.py:920), and the snippet right below prints the
whole usage dict, so a reader saw keys the docs omitted. Both EN and ZH now
show the full payload, and the `values` example visibly elides `artifacts`
and `summary_text` instead of pretending they are absent.
- the two SSE comments on the ZH integration page stayed English while the
example above them was translated; translated.
Verified statically, not executed: `grep` for the symbol across the harness
package (zero hits) and a read of `DeerFlowClient.__init__`. I did not stand
up a backend to run the tutorial end to end.
Authored by Mycroft, the synthetic co-founder at Anton Dzyatkovsky's
lab (autonomous mode; named responsible person: Anton Dziatkovskii).
Assisted-by: Claude Code / Opus 5 (review response, implementation)
Machine: MacBook-Anton
Account: a
Operator: mycroft (autonomous routine github-thread-watch)
Signed-off-by: tonydzi <194927794+tonydzi@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: fix native artifact SSE encoding and custom config advice
---------
Signed-off-by: Anton Dziatkovskii <194927794+tonydzi@users.noreply.github.com>
Signed-off-by: tonydzi <194927794+tonydzi@users.noreply.github.com>
Co-authored-by: Anton Dziatkovskii <194927794+tonydzi@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(agent): drop script guidance from the lead prompt when no bash tool is bound
The workspace section told the agent how to write scripts and commands
("prefer relative paths", "avoid hardcoding /mnt/user-data/... inside
generated scripts") even when no tool could run them. With the default
LocalSandboxProvider host bash is off and the bash tool is removed, so
agents wrote helper scripts nothing could execute, then stalled or
computed by hand.
apply_prompt_template() takes bash_available (default True, so other
callers keep today's text). The lead agent (both assembly branches) and
the embedded client pass whether their authorized tools, bound or
deferred, include bash. Without it:
- the two script bullets become one line: no bash tool is bound, work
out results directly and write them with write_file;
- the subagent section no longer shows direct bash examples, and marks
the bash subagent unavailable, since it inherits the lead's tool groups;
- the ACP hint no longer suggests `bash cp`.
* fix(goal): show the goal evaluator the user's answer to a Human Input Card
The web UI sends a Human Input Card answer as a hidden human message
(hide_from_ui with a human_input_response) and shows it on the card.
format_visible_conversation() skipped every hidden message, so the
evaluator saw the question but never the answer. Following the user's
answer then read as the assistant guessing, and the goal stood down with
needs_user_input although the user had answered.
A hidden human message whose human_input_response passes
read_human_input_response() now adds a line
"User (Human Input Card answer): <value>" from the structured value; the
question is already on the card's own line. Other hidden messages, such
as goal continuations, stay out. Over the 12,000-char cap the latest
answer the cut drops is kept at the top next to the latest request, and
is never taken for the request itself. Without card answers the cut is
unchanged.
* fix(agent): keep the subagent examples and bash hint in line with the bound tools
Review follow-up for the lead prompt without bash.
- The last delegation example follows the tools. With a bash subagent on
offer it is unchanged. A lead with bash and no bash subagent keeps
"Run a routine test, build, or git command directly." without the Bash
subagent sentence. A lead without bash gets a file-work example instead.
- The unavailable-bash description no longer points at the sandbox
provider. Since #2253 it renders only when the bash subagent is
registered and the lead binds no bash, where the sandbox is not the cause.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(gateway): reject non-ASCII tokens instead of raising in constant-time compares
hmac.compare_digest raises TypeError when a str operand holds non-ASCII
characters. Starlette decodes header bytes as latin-1 and query values as
UTF-8, so a single 0xE9 byte in a CSRF token, GitHub webhook signature,
internal auth token, or OIDC state turned the expected 403/401 into a 500.
Route the five client-facing comparisons through a new
app.gateway.utils.constant_time_equals helper that compares the UTF-8
encodings ("surrogatepass" so even a lone surrogate cannot raise). ASCII
results are unchanged, and no bypass was possible before the fix.
* fix(provisioner): reject non-ASCII API keys instead of raising; drive OIDC test over HTTP
The standalone provisioner compared X-API-Key with secrets.compare_digest on
str operands, so a non-ASCII header raised TypeError (500) the same way the
Gateway sites did. It cannot import app.gateway, so encode inline.
The OIDC state regression test now mounts the auth router and sends the
percent-encoded state through a real signed state cookie instead of stubbing
get_state_cookie and calling the handler directly.
* docs(changelog): reference #6076
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
test_refresh_skew_rejects_negative_and_boolean omitted the required
token_url, so McpOAuthConfig(refresh_skew_seconds=-1) failed validation
on the missing field alone -- deleting the ge=0 bound and the boolean
validator would leave the test green. Pass token_url (the same URL the
refresh_skew_seconds=0 boundary test uses) so only the guards can
satisfy pytest.raises; verified by stripping both guards from
extensions_config.py, which now fails exactly the two parametrized
cases, and restoring them, which returns the suite to 42 passed.
Also move SandboxOwnershipConfig up beside the other config imports
and drop the redundant in-test `import pytest` (review nits on #6026).
Generated-by: ZCode (GLM, coding agent)
Co-authored-by: liwenjie200543 <liwenjie200543@users.noreply.github.com>
* fix(agents): treat a null max_total_subagents override as unset
An API caller sending "max_total_subagents": null in the run context
failed that run with a TypeError: the key was present, so
dict.get(key, default) returned None and building SubagentLimitMiddleware
compared None with an int.
Resolve the per-run delegation cap through one helper,
effective_total_subagents_per_run, which treats None as "use
subagents.max_total_per_run" and clamps to 1-50. The lead agent's
middleware and assembly paths, DeerFlowClient, and the system prompt all
use it, so the extension host policy and release policy now report the
enforced cap instead of an out-of-range request.
* docs(changelog): reference #6088 in the null max_total_subagents entry
* fix(sandbox): expose DEERFLOW_USER_ID to bash commands
A skill's SKILL.md can trigger scripts, but those scripts run as bash
subprocesses inside the sandbox and cannot reach the Gateway auth context,
so get_effective_user_id() is unavailable to them and user-scoped skills
are impossible to write. The maintainer settled the direction on #3919:
inject the id as DEERFLOW_USER_ID.
Every bash command now begins with 'export DEERFLOW_USER_ID=<resolved>; '
(or 'unset' when the resolved id is unusable), mirroring the existing
DEERFLOW_CHANNEL_USER_ID channel-identity prefix (#3914). The id rides the
command string rather than execute_command(env=...), because a non-empty env
switches AioSandbox to the bash.exec API reserved for request-scoped secrets.
The earlier attempt (#4029) was closed for making every command carry the
new export without updating the exact-command assertions in
tests/test_channel_user_id_env.py. Those assertions are updated here, and the
composition of the two prefixes is pinned by a dedicated test.
* fix(sandbox): fix the DEERFLOW_USER_ID unit test and cover review findings
Follow-up to the review on this PR.
Unit test (CI `backend-unit-tests (2)`): the change moved the command
contract but left the exact-command assertion in
tests/test_skill_request_scoped_secrets.py::test_local_bash_forwards_env_and_timeout
expecting the bare `echo hi`. The local validation run used a `-k` filter that
did not select that file, so the breakage was invisible there. The assertion
now pins the composed command, and `_is_windows` is patched explicitly so it no
longer depends on the host OS.
[P2] Windows local sandbox: `export` is invalid under the PowerShell/cmd.exe
fallback, but skipping the prefix outright left the promise unmet on Windows.
The id is published through `execute_command(env=...)` there instead:
LocalSandbox layers `env` into the per-process environment and, unlike
AioSandbox, has no persistent session that could carry a stale value. It is
deliberately kept out of `injected_env` so the identifier does not enter the
secret-redaction set.
Docstring: `_user_identity_prefix` now carries the same "informational, not
authenticated identity" caveat the channel variable has in
backend/app/channels/AGENTS.md. The value is authorization-grade at the Gateway
boundary, but any bash command can overwrite its own environment, so skills
must not treat the shell variable as proof of identity.
Docs: the en/zh sandbox.mdx identity sections and the channels/AGENTS.md
identity paragraph now describe the second, outermost prefix, its ordering
against the channel prefix, and the Windows behaviour.
* fix(goal): record a failed evaluator call as a stand-down
When the goal evaluator call raised (a provider error, or an answer that
was not valid JSON), _prepare_goal_continuation_input() logged a warning
and returned without writing anything. The goal stayed active with no
last_evaluation, or with an earlier run's verdict and run id, unlike the
other paths that stop the goal loop.
Record it like the no_durable_end_of_turn stand-down: blocker run_failed,
stand_down_reason evaluator_failed, the current run id, and a reason that
names only the exception type. Nothing is recorded if the run was aborted
meanwhile. A failure before the evaluator is called keeps its behavior and
now logs "Could not prepare goal evaluation".
* docs(gateway): fit the evaluator-failure note in the AGENTS.md budget
Main grew backend/app/gateway/AGENTS.md to 49,006 bytes, and the longer
note pushed the merged file past the 49,152-byte hard limit that
tests/test_agent_guidance_check.py enforces. Say it in one clause.
* docs(gateway): keep main's AGENTS.md line after the merge
The merge from main kept this branch's version of the goal paragraph,
which restored the Human Input Card wording #6068 replaced and put the
file at 49,194 bytes, over the 49,152-byte limit the agent-guidance
check enforces. Take main's line. It already says every non-satisfied
evaluation is persisted with last_evaluation, which this fix makes true
for a failed evaluator call.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(subagents): retain completed model usage on cancellation
* docs(subagents): fit guidance within chain budget
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
---------
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(goal): show the goal evaluator the tool calls and results
The evaluator read only the text of user and assistant messages and was told
to fail closed without explicit evidence, so it stood down with
missing_evidence on most completed file tasks: the tool calls and tool
results that the web UI shows never reached it.
Include the assistant's tool calls and the tools' results inside the
existing message window, shortened (200 characters per argument value, 600
per step, line breaks kept), and tell the evaluator to treat tool results as
data, that a successful tool call alone does not meet a goal, and that a
guessed input means needs_user_input. The no-progress and final-clear
signatures are unchanged.
* fix(goal): tighten the evaluator's tool evidence after review
- Pair each tool result with the latest call of its id before it. Some
providers reuse ids such as call_0 across turns, and the window-wide
map labelled an earlier read with a later write's name.
- Keep model-supplied tool names on one line in calls and result labels,
and escape U+0085, U+2028 and U+2029 in arguments and results, which
json.dumps(ensure_ascii=False) leaves raw, so no tool step can start a
new evidence line.
- Leave out the results of a hidden assistant message's calls. The web UI
renders a result only under its call, so neither shows there either.
A clarification prompt stays: the UI shows it as its own card.
- Over the 12,000-char cap, keep whole lines from the end instead of
slicing mid-line. Keep the latest user message that the cut drops at
the top, whole when it and the lines after it fit, else shortened to
2,000 chars. If the next assistant message back does not fit whole,
keep its end in the room left. A marker counts the lines left out.
- Docs: the Gateway AGENTS.md and a worker docstring no longer say the
evaluator never sees tool messages.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* docs(backend): point the stream-mode translation anchor at stream_modes.py
* docs(backend): use one anchor spelling within the stream-mode table row
The row named the translator by its backend-relative path and its call
site by the harness-relative short form, so the same symbol was written
two ways two rows apart.
* fix(events): read a non-filename run ID as an unknown run on JSONL
The run-scoped routes (events, messages, workspace-changes) pass the URL's
run_id straight to the event store. With run_events.backend: jsonl, an ID
that cannot be a filename (run.1, a%20b, %2e%2e) raised ValueError and
surfaced as a 500, while the memory and database stores return an empty
result.
JSONL reads and deletes now treat such an ID as an unknown run through one
helper, _existing_run_file(), which returns before building any path.
Writes still reject it.
* docs(changelog): reference #6070 for the JSONL run ID fix
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): drain managed model saves on cancellation
* test(gateway): pin managed model save cancellation drain
* fix(gateway): log failed drained managed model saves
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): keep guidance within hard size limit
---------
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(persistence): reject SQLite file: URIs instead of opening them as filenames
resolve_sqlite_conn_str passed a file: connection string through
unchanged, but LangGraph's SQLite checkpointer and Store factories
(SqliteSaver, SqliteStore, AsyncSqliteSaver, AsyncSqliteStore) connect
without uri=True, so SQLite treated the whole URI as a literal filename:
file:checkpoints.db?mode=rwc created a file with that exact name in the
working directory, file::memory:?cache=shared persisted to disk, and a
file:/// URI failed to open. The readiness probe did open URIs with
uri=True, so it checked a different file than the runtime used and
reported in-memory URIs as not_configured.
Reject file: URIs in resolve_sqlite_conn_str, the one helper all four
factories go through, with an error that names
checkpointer.connection_string. The probe drops its URI parsing and
reports a rejected connection string as unreachable.
* docs(changelog): reference #6069
* test(persistence): pin the case-sensitive file: URI check
* fix(channels): decode Slack entity escapes in inbound message text
Slack sends a user-typed &, < and > as &, < and > in event text, so raw <...> always marks a control sequence. The Slack channel passed that text to the agent as-is, so code, comparisons and names like R&D reached the agent as 'a < b', 'R&D'. Decode the three entities after bot-mention stripping (amp last, so an escaped entity is decoded once) and keep mentions and links untouched.
* test(channels): cover Slack entities inside link labels
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
---------
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): render the bundled interpreter repair path without backslash escapes
The repair hint used {sys.executable!r}, whose repr() escapes Windows
backslashes - users saw 'D:\college\...\python.exe' in the
copy-paste guidance, and
test_stale_bundled_interpreter_can_be_repaired_without_reinstall fails
3/3 on Windows because the raw path never appears in the detail. The
!r was written for the surrounding quotes; on POSIX paths that is its
only effect, which is why CI never saw it.
Quote the raw path explicitly instead. On Windows the three test params
now pass (22/22 in the file); POSIX output is unchanged apart from the
quoting coming from the literal instead of repr().
* fix(gateway): json-escape the interpreter command in the repair hint
Review follow-up (P2): the raw Windows path in the hint cannot be
pasted into the server's JSON ('\U' is not a JSON escape), so the
prescribed repair could not be saved. Show the command value via
json.dumps(sys.executable) - pasteable JSON on every platform - and
keep the raw path in a parenthetical for terminal use, per review.
* fix(helm): plumb lark-cli runtime images into the provisioner
The provisioner, docker-compose, and the root README all support
`LARK_CLI_INIT_IMAGE` (Pattern A) and `LARK_CLI_BROKER_IMAGE` (Pattern B),
but the chart never exposed them, so a Helm operator following the README
had no value to set and silently got a sandbox without a `lark-cli`
runtime (`lark-cli: command not found` at chat time).
- values.yaml: `provisioner.larkCliInitImage` / `larkCliBrokerImage`, both
empty by default, so the rendered Pod spec is unchanged for existing
releases.
- provisioner-deployment.yaml: render each env var only when non-empty.
The provisioner keys off `bool(env)`, so omitting the variable keeps
"feature off" identical to the pre-change chart.
- chart README: new "Sandbox lark-cli runtime (optional)" section; root
README points Helm users at the two keys.
- backend/tests/test_helm_provisioner_lark_cli_images.py: render
regression test (red before the template change).
* docs(helm): drop duplicated lark-cli callout in chart README
Review on #5835 caught the "An image here does not authenticate anyone
by itself" blockquote pasted twice verbatim in the new chart README
section. Remove the second copy and keep the first as the section's
closing note. Docs-only; the rendered chart is unchanged.
* docs(zh): add the missing External Chat Message Roles section to README_zh
Adds the 'External Chat Message Roles' section to the Chinese README,
mirroring the section present in the English README under the Security
Notice. Same position (between Gateway Admin Is Equivalent to Code
Execution and Deployment Defaults), same content semantics, translated
rather than paraphrased.
Covers the HTTP 400 rejection of client-supplied system/developer
messages, the no-rewrite-of-existing-checkpoints note, and the full
local-verification PoC instructions (poc_external_system_message_injection.py,
--expect vulnerable / --expect blocked). The referenced script exists at
backend/tests/poc_external_system_message_injection.py.
The English TOC does not list this subsection either, so the zh TOC is
left unchanged to stay consistent (same approach as #5409).
* Fix minor language inconsistencies in README_zh.md
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(config): upgrade the config.yaml the Gateway loads
scripts/config-upgrade.sh resolved DEER_FLOW_CONFIG_PATH, then
backend/config.yaml, then <checkout>/config.yaml. make dev exports
DEER_FLOW_PROJECT_ROOT=<checkout>, so the Gateway reads the checkout copy
first: with both files present the script upgraded a file nothing loads
and left the live config outdated while reporting success. It also
ignored DEER_FLOW_PROJECT_ROOT and a DEER_FLOW_CONFIG_PATH set in .env,
and fell back to another file when DEER_FLOW_CONFIG_PATH named a
missing one.
Ask the harness resolver (AppConfig.resolve_config_path) instead of
copying its order, defaulting DEER_FLOW_PROJECT_ROOT to the checkout as
serve.sh does. A missing DEER_FLOW_CONFIG_PATH or invalid project root
now fails with the Gateway's error; no config anywhere still creates
<checkout>/config.yaml from the example.
* docs(changelog): reference #5991
* test(config): pin config-upgrade seeding a missing config from the example
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
path_matches backs the glob tool and grep's glob= filter for every sandbox provider. It relied on PurePosixPath.match, which matches from the right and treats ** as a single-segment * on Python 3.12+, so only a leading **/ was recursive: src/**/*.py missed src/top.py and anything two or more levels deep, and docs/*.md also matched vendor/docs/b.md.
Match patterns that contain a / segment by segment from the search root, with ** spanning zero or more directories. Patterns without / keep matching the basename at any depth.
* docs(helm): fix the external PostgreSQL example in the chart README
The example set postgresql.external.host/port/database/username/password,
but the chart has never read those keys; it only reads
postgresql.external.databaseUrl or existingSecret. With the documented
values the chart creates no postgres Secret and the gateway gets no
DATABASE_URL, while config.yaml still references $DATABASE_URL, so the
config fails to load. Show databaseUrl, with existingSecret as the
alternative.
* docs(helm): clarify external database URL encoding
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
---------
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
On Windows, Path.write_text and the text-mode append translate the
fixtures' newlines to CRLF on disk, so read_config_with_signature
correctly returns b'...info\r\n' while the tests assert the byte-exact
b'...info\n' - three tests fail on Windows (CI's Linux never sees the
translation). The product behavior is correct; the fixtures must write
the exact bytes they assert.
write_bytes / binary append for the three byte-sensitive tests: the
file now passes 10/10 on Windows.
* fix(community): close the per-call Firecrawl client's async HTTP pool
The firecrawl tools build a fresh AsyncFirecrawlApp for every web_search
and web_fetch call. The migration to the async client (#5786) kept the
per-call construction, but AsyncFirecrawlApp eagerly creates an
httpx.AsyncClient-backed pool in its constructor and exposes no public
teardown, so each tool call abandons one unclosed pool. The sibling
providers either scope their httpx clients with 'async with' or close
them in a finally (tavily); firecrawl now does the same via a
best-effort teardown that tolerates SDK shapes without the v2 client.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(firecrawl): make per-call pool teardown resilient
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
* test(firecrawl): skip the real-SDK teardown test when the private layout differs
The real-SDK test hard-asserted the locked firecrawl-py 4.23.0 private
layout (_v2_client.async_http_client._client.is_closed), so an
environment resolving another version in the declared >=1.15.0 range
died with AttributeError instead of skipping — while the helper (and
test_teardown_survives_a_client_without_v2_attributes) deliberately
treat a missing or reshaped layout as a no-op. Guard on the same
attributes and skip when they are absent.
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): unwrap Overwrite-wrapped sandbox state in tools and middlewares
* fix(changelog): restore database entry and add #4380 reference link, simplify task_tool unwrap
* fix(subagents): preserve Overwrite wrapper when task_tool forwards sandbox state to executor
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(integrations): keep Lark runtime status consistent
Probe sandbox runtime readiness for every Lark mutation response outside the credential lock. Expose explicit sandbox_runtime_probed state so the frontend can distinguish conservative fallbacks without inferring semantics from runtime detail text, while cancelling stale reads before updating the cache.
* style(frontend): format Lark cache test
* fix(integrations): snapshot Lark credential status under the lock
Why this change is needed:
PR #5994 moved the post-mutation status read outside the per-user
credential lock so the new sandbox runtime probe would not extend the
critical section. As flagged in review, that lets a concurrent app
switch or auth flow land between the committed mutation and the status
read; in remote-provisioner mode the runtime probe can spend seconds
before probe_lark_auth runs, so the raced switch can clear the new
tokens and make a completed login report success=false, or mix an old
app_id with a newer auth state.
What changed:
- add _LarkCredentialSnapshot (app config + auth probe) and capture it
in the same _lark_credential_lock section that commits the mutation
- thread the snapshot through _get_lark_mutation_status into
get_lark_integration_status(credential_snapshot=...); only the
credential-independent runtime probe still runs after the lock
- apply to complete_lark_auth, complete_lark_config, and
set_lark_app_credentials; install keeps the one-shot status build
Problem solved:
Mutation responses stay tied to the just-committed credential
generation: a raced app switch can no longer flip a completed
device-code login to success=false or blend a stale app_id with newer
auth state, while the runtime probe stays off the lock.
* fix(integrations): address Lark status review follow-ups
- install_lark_integration: snapshot credential-derived status fields
under _lark_credential_lock so the install response cannot mix newer
credential state read across the unlocked runtime probe
- useInstallLarkIntegration: route the cache write through
cacheLarkMutationStatus so install cancels in-flight reads and keeps
probed runtime fields under frontend/backend version skew
- README: soften the UI claim - the status API exposes
sandbox_runtime_probed and the mutation cache honors it
* test(integrations): tighten Lark status fixtures and guidance docs
- cache.test: use a distinct mutation runtime mode in the unprobed-merge
case so the assertions prove the cached runtime fields win
- e2e mock: include broker in the sandbox_runtime_mode union cast so the
fixture cannot drift from LarkSandboxRuntimeMode
- AGENTS.md (gateway + skills): mention sandbox_runtime_probed alongside
the existing runtime readiness fields, keeping the gateway guidance
file under its byte budget
* fix(integrations): harden Lark mutation cache against skew and races
- cacheLarkMutationStatus: await cancelQueries before writing so an
in-flight GET whose fetch already resolved cannot clobber the fresher
mutation status, matching the existing threads/archive pattern
- self-heal version skew: an unprobed (pre-field backend) mutation
response now triggers a status refetch instead of risking the
conservative runtime fallback lingering until the next focus/remount
- api: normalize a missing sandbox_runtime_probed to false at the
boundary so the declared-non-optional type matches skewed runtimes
- tests: pin the skew contract (missing-field cases for cache and api)
and add a useInstallLarkIntegration regression test proving the
install cache write goes through cacheLarkMutationStatus
- README: describe the probed flag as cross-version semantics instead of
claiming the same-version API surfaces both states
- lark_cli: note that install's credential snapshot runs the auth probe
subprocess while holding the credential lock
* refactor(integrations): derive sandbox_runtime_probed inside the resolver
_resolve_sandbox_runtime_readiness now returns a fourth probed element
next to the branch that produces the unevaluated remote fallback, so the
'was readiness actually evaluated' invariant can no longer drift from the
fallback logic it mirrors. Callers and resolver-level tests updated to
pin probed per branch.
* fix(doctor): check the config file the Gateway actually loads
make doctor always inspected <checkout>/config.yaml and ignored
DEER_FLOW_CONFIG_PATH and DEER_FLOW_PROJECT_ROOT. A missing override
that stops the Gateway from starting still reported "config.yaml found"
and "config.yaml loadable", and a valid override pointing elsewhere got
the wrong file checked.
Doctor now resolves the path through the harness's own resolver,
AppConfig.resolve_config_path, after defaulting DEER_FLOW_PROJECT_ROOT
to the checkout as scripts/serve.sh does for make dev. An override the
Gateway would reject fails "config.yaml found" with the Gateway's error
and the config checks skip; an unimportable harness is reported rather
than raised.
* docs(changelog): reference #5987
* fix(doctor): hand the config-location variables over like serve.sh
Review follow-up. serve.sh sources the checkout .env over the shell and
then replaces an unset or empty DEER_FLOW_PROJECT_ROOT with the
checkout. Doctor loaded .env shell-first and only defaulted an unset
root, so a shell DEER_FLOW_CONFIG_PATH / DEER_FLOW_PROJECT_ROOT that
differed from .env, or an empty root, made it check a different file
than the Gateway loads.
For those two variables the .env value now overrides the shell (other
keys keep doctor's shell-first loading), and an empty root becomes the
checkout, before the harness resolver runs.
* fix(doctor): report any harness import failure and expand ~ like source
Review follow-up.
- The harness-import guard caught only ImportError, so a harness that is
present but broken differently (TypeError/SyntaxError while the module
body runs) still crashed doctor. Catch Exception, as
check_config_loadable does, and name the exception type in the report.
- serve.sh's `source .env` expands an unquoted leading `~` in
DEER_FLOW_CONFIG_PATH / DEER_FLOW_PROJECT_ROOT and keeps a quoted one
literal; dotenv_values expands neither. Expand the unquoted form so a
`~/...` path in .env no longer fails a setup that starts, and keep the
quoted form literal so doctor does not pass one the Gateway rejects.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): close thread-scoped MCP sessions on thread deletion
Deleting a thread removed its filesystem data, checkpoints, runs, events,
feedback, metadata and browser session, but never closed its persistent MCP
sessions. MCPSessionPool.close_scope() had no production caller at all, so a
recreated thread id under the same user could reuse stale stateful MCP server
state, and the leaked owner task plus subprocess kept holding memory and file
descriptors.
Add MCPSessionPool.close_thread_scope(user_id, thread_id) and call it from the
reserved deletion path under the same best-effort policy as the neighbouring
cleanups. It matches every incarnation of the thread -- the legacy user:thread
scope and any v2:[user, thread, incarnation] scope -- because a deletion cannot
reliably read the current incarnation: the record may predate incarnation
tracking, and a new incarnation can be minted concurrently. close_scope() is
unchanged for callers that already hold one exact key.
Add mcp_scope_belongs_to_thread() to own the matching rule, and hoist the v2:
prefix into a shared constant so the encoder and matcher cannot drift.
Refs #5188
* test(mcp): bind owner identity in thread cleanup checks
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
