* fix(loop-detection): let a warned agent save collected results before its final answer
The warn-level loop messages told the agent to "Stop calling tools and
produce your final answer now." Subagents read this literally: one whose
task names an output file stops all tool calls, including the single
write that delivers its result, and finishes as "completed" with nothing
on disk.
Only the two warn-level texts change. They now ask the agent to stop
repeating the looping call and wrap up, first saving what it has
collected through a file-writing tool if its task asks for a file. The
per-tool warning names the looping tool. Thresholds and the hard-stop
path (which strips tool calls) are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(loop-detection): bound final-save recovery after warnings
* docs(middleware): keep inherited guidance within CI budget
---------
Co-authored-by: huangzz <110116@nd.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(artifacts): preserve account-scoped drafts across navigation
* docs(artifacts): keep draft lifecycle guidance within its budget
* fix(artifacts): retain drafts through unknown auth and isolate typing updates
* fix(artifacts): preserve drafts when visiting public showcases
* fix(artifacts): preserve edits after pending save
* fix(browser): publish captured screenshots and fetches atomically
_write_screenshot and _write_capture_output wrote the final path with a truncating
write and then returned its name, so a failure part-way through left a partial
capture that the caller reported as saved - and the leftovers then made the dedupe
helper pick a new name for the next capture. Publish through a temporary file plus
os.replace.
* fix(browser): preserve permissions on atomic capture publication
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): decode cmd.exe output with the ANSI code page under UTF-8 mode
The cmd.exe branch left its pipe decoder to the `encoding=None` default of
`_run_windows_command`, which follows Python text-mode semantics via
`locale.getpreferredencoding(False)`. Under Python UTF-8 mode
(`PYTHONUTF8=1`, `-X utf8`, the Windows installer's UTF-8 opt-in, and the
default once PEP 686 lands) that call reports `"utf-8"`, while cmd.exe and
the children it spawns keep writing the host ANSI code page - so every
non-ASCII byte in cmd output collapses to U+FFFD, with no fallback path.
Measured on a real zh-CN host under `PYTHONUTF8=1` (`getpreferredencoding`
-> `utf-8`, `getencoding` -> `cp936`):
- through the `encoding=None` default: `'��� �����Z
'`
- pinned to the ANSI code page: `'你好 日本語
'`
This branch pins the cmd.exe pipe decoder to `locale.getencoding()`
(3.11+, repo floor is 3.12), the same selection the MSYS and PowerShell
branches already use for their fallback (#6727, #6548).
## Scope note
The `encoding=None` default itself is deliberately unchanged: it mirrors
Python's own `text=True` semantics and is pinned by
`test_windows_capture_matches_subprocess_text_mode_encoding`. Only the
cmd.exe branch - whose output is known to be the ANSI code page - overrides
it, the same way the MSYS and PowerShell branches already pin their own
decoders.
Follow-up to willem-bd's review note on #6727 (inline comment 4247281255)
flagging this path.
## Testing
- new (POSIX-runnable, mocked): the cmd branch passes
`encoding=locale.getencoding()` to `_run_windows_command`, verified under
a simulated UTF-8-mode host (`getpreferredencoding` -> `utf-8`,
`getencoding` -> `cp936`).
- updated: `test_execute_command_uses_cmd_command_mode_on_windows` now also
pins the new `encoding` kwarg.
- new (Windows; skipped when the console output CP does not match the ANSI
page - sibling tests can change it process-wide): real `cmd.exe /c echo`
roundtrip through `execute_command` returns the text intact (passes on a
zh-CN host when run standalone).
- full `tests/test_local_sandbox_encoding.py`: 40 passed, 2 skipped under
both `PYTHONUTF8=0` and `PYTHONUTF8=1` (Windows 11, Python 3.13.12, repo
venv); the text-mode contract test stays green in all three parametrized
modes (`locale-code-page`, `PYTHONUTF8`, `-X-utf8`). Red run against
unmodified main: both new/updated cmd-branch tests fail (`KeyError:
'encoding'`).
- `ruff check` + `ruff format --check` clean on both files.
* fix(sandbox): use the cmd console output code page
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): add channels.leader_lease and gate it under multi-process signals
IM channels other than GitHub connect outbound to their platform. With more
than one Gateway instance each one opens that connection with the same bot
credentials: Telegram rejects the second poller with 409, Discord answers every
message twice and per-process ChannelManager state diverges. The multi-process
startup gate said nothing about channels; the only guard was a comment in the
Helm values.
Add the `channels.leader_lease` section (`enabled: auto|true|false`,
`lease_seconds`, `grace_seconds`; startup-only, registered in the reload
boundary) that the per-provider leader lease will read, validated by a Pydantic
model from the non-schema `channels` mapping. The gate now refuses
`enabled: false` under a multi-process signal while a non-GitHub provider is
enabled under `channels.*` or connectable through `channel_connections`, naming
the providers and the rollback; a malformed block is refused on every topology.
`auto` follows the multi-process signal, so a single instance is unchanged.
ChannelService strips the block before its per-provider loops and exposes it
as `leader_lease`; `ensure_ready_channels` only visits registered providers.
The two-Gateway harness overlay leaves the block alone. config_version 60 in
the example, the Helm chart and its README; docs replace the comment-only
"keep 1 replica" guidance with the gate.
* docs(changelog): link #6746
* fix(channels): stop recommending the unimplemented leader lease as a remedy
Review on #6746: the refusal told operators to switch channels.leader_lease
from false to auto/true "so one instance owns each provider's connection", but
the transport arbitration ships separately, so that change only bypassed the
clause and every worker still started the same bot.
The refusal now offers only the remedies that work today (a single instance, or
disabling the channels) and states that auto/true are reserved for the lease
that ships separately. A regression helper in the gate tests asserts the
remedies start with the rollback and that auto/true are never offered. The
config example, Helm values/README, CONFIGURATION.md, README, the deployment
guide (en/zh), the model docstrings and both changelogs describe auto/true as
reserved settings rather than an ownership guarantee.
* fix(channels): warn on an enabled unknown channel key; reword the changelog entry
Review on #6746: the registry guard in ensure_ready_channels skipped a
misspelled provider key (channels.telegam) silently, where it previously
reached _start_channel and logged 'Unknown channel type'. An enabled dict
block that is neither a registered provider nor leader_lease now logs a
warning that lists the registry.
The changelog entry claimed the chart and docs drop the "keep 1 replica"
guidance; they keep it (auto/true do not make several replicas safe yet) and
only stop relying on the comment alone.
* fix(dev): disable UI channel providers in multi-instance overlay
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(qq): parse allowed_users with the shared fail-closed allowlist
QQChannel built its allowlist with set(config.get("allowed_users") or []).
A single OpenID, which is what an `allowed_users: $QQ_ALLOWED_USERS`
reference resolves to, became a set of its characters, so the configured
user was silently dropped. A mapping authorized every key, including one
mapped to false, and a numeric scalar raised TypeError while the channel
was being constructed.
Parse it with the shared parse_allowed_users helper that Telegram and
WeChat already use: unset, null, empty collections and a blank string
allow everyone; a scalar OpenID is one entry; entries that are not
non-blank strings (such as numeric QQ account numbers) are dropped with a
warning, and a mapping is rejected as a whole; a configured value with no
valid OpenID denies every sender and logs an error. A scalar containing commas or whitespace warns
that it stays one literal ID. Binding codes still precede the allowlist.
* test(qq): gate group messages by member OpenID in allowlist tests
Every allowlist test sent a C2C frame, and the shared event() helper puts
the sender in both user_openid and member_openid, so a group-path
regression in sender resolution would pass unnoticed. admits() now takes
a kind; group frames carry only member_openid and a group_openid that
differs from the sender. The listed-OpenID test runs for both kinds, and a
new test shows that listing the group's OpenID does not admit its members.
Document that a blank or whitespace-only allowed_users string allows every
sender, like an unset value or an empty list.
* refactor(channels): share scalar allowlist separator warnings
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(client): convert the captured upload source
* test(uploads): cover ACL sources and scope capture failures
* fix(uploads): preserve concurrent uploads from one captured source
Combine the embedded client's source capture with no-replace publication for
first-time filenames and Markdown companions. Hold one private capture across
collision retries, retain original descriptor metadata, and keep replacement
behavior for names present when the call starts.
Port concurrent-upload and staging-publication regressions from PR #6687.
Extend source-edit/removal and real XLSX cases to both publication paths, and
cover edits or removal during two filename collisions in ordinary and event-loop
calls. Document capture, retry, publication, and cleanup contracts.
Validation: related suites 469 passed, 24 skipped; full offline suite 28,788
passed, 488 skipped; blocking-I/O suite 245 passed, 1 skipped; Ruff, guidance,
and diff checks passed. Nine race/collision cases failed on the #6632 baseline.
Co-authored-by: haotianqi666 <250384991+haotianqi666@users.noreply.github.com>
* docs: keep agent guidance within CI budgets
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: haotianqi666 <250384991+haotianqi666@users.noreply.github.com>
* fix(sofya): report a format error for results with no usable objects
`_response_results` skipped non-object entries but returned the filtered
list even when it was empty, so a non-empty `results` array holding no
objects fell through to "No results found" -- telling the agent the query
legitimately matched nothing instead of surfacing the malformed payload.
Treat a non-empty list with no object entries as malformed, matching the
containment rule #6603 documents for this provider family (brave #6550,
groundroute #6603, tavily #6645, webz). Both `web_search` and
`web_fetch` share this helper, so both now report the provider's format
error. Missing, null, and empty `results` keep returning "No results
found", and mixed lists still keep their valid entries in order.
* test(sofya): cover mixed fetch result entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(dingtalk): preserve Markdown table cell boundaries
DingTalk's sampleMarkdown fallback re-splits table rows on every pipe, so an
escaped pipe (\|) inside a cell is read as a column separator and every value
after it shifts one column to the left. The strip("|") wrapper also removes an
empty leading or trailing cell, dropping that value's column label.
Split rows on unescaped pipes only, and drop at most one outer border cell per
side, so escaped pipes and empty edge cells stay under the right label.
Adds six parameterized regressions plus one full fallback-adapter regression;
all seven fail on the base commit and pass after the fix.
* docs(dingtalk): document escaped-backslash boundary in table row splitter
Document the residual boundary case flagged in review: the
single-character lookbehind in _split_markdown_table_row cannot
distinguish an escaped pipe (\|) from an escaped backslash followed
by a real separator (\\|), so a cell ending in a literal backslash
absorbs the next cell. Full GFM backslash counting is out of scope
for this helper.
* feat(channels): render Feishu option clarifications as button cards
When the agent's ask_clarification carries options, the Feishu channel
now renders the question as an interactive card with one button per
option instead of plain text. This is the outbound half: card
construction plus a process-local pending registry that a later
card-action callback handler will validate clicks against.
- New feishu_card_interaction module owns request extraction, button
card rendering, and the pending registry (30-minute TTL, matching the
typed-reply clarification window)
- Manager attaches the clarification's structured human_input payload to
the final outbound's metadata so channels can render richer
affordances without touching the Gateway protocol
- Feishu channel renders the button card only when
card_interaction_enabled is set (default off); free-text, form, and
over-four-option clarifications keep the plain-text path
- Button cards bypass the markdown wrapper via a raw_card_content flag
on the card send/patch helpers; plain-text call signatures unchanged
* feat(channels): close the Feishu button-card click loop
The inbound half of interactive clarification cards: a click on an
option button validates, resolves the clarification, and marks the
card in place.
- Register the card-action callback only when card_interaction_enabled
is set; clicks arrive as WS EVENT frames through the SDK's callback
processor map, and the synchronous return value becomes the toast ack
- consume_if_originator atomically validates operator identity against
the pending entry: non-originators get a refusal toast and the entry
survives; unknown or expired requests are toast-only and dropped
- A validated click consumes once, updates the card to a terminal
resolved state, and injects a hidden inbound carrying the
human_input_response so the run resumes on the originating thread
with Web-card parity
- Manager translates the inbound's human_input_response metadata into
the human message's additional_kwargs with hide_from_ui set
* docs(channels): document Feishu interactive clarification cards
- README / README_zh: optional card_interaction_enabled setup, calling
out that card.action.trigger lives under the console's callback
configuration tab (separate from event subscriptions) and requires
publishing a new app version
- config.example.yaml: commented flag with the same console caveat
- backend/AGENTS.md: module ownership note for feishu_card_interaction
* fix(channels): close Feishu card-click routing and ack gaps from review
Address review on #6670:
- Carry the routing identity (topic_id, connection_id, owner_user_id)
on the pending entry and the synthetic inbound, so clicks resolve to
the clarified thread in topic-scoped chats and pass bound-identity
admission instead of opening a new thread or being rejected
- Validate option_value against the registered options at consume time:
an unknown value is treated like a stale card and the entry is kept
- Return a warning toast when a click cannot be dispatched, and flip
the card to its resolved state only after the inbound is durably in
the bus — never display a choice that was not recorded
- Drop the dead mark_sent/consume registry APIs and pass the payload's
own option ids through to the buttons
- Add an end-to-end test through the manager's thread resolution
pinning that a card answer reuses the clarified thread
* fix(channels): carry workspace_id on the Feishu card-click inbound
The connections-enabled admission path re-reads the binding with the
message's workspace_id, and Feishu connections are bound with
workspace_id = chat_id. The synthetic inbound set connection_id and
owner_user_id but never workspace_id, so the re-read missed and every
click was rejected as unbound after the entry was consumed.
- Set inbound.workspace_id from the entry's chat_id (Feishu's
chat-as-workspace binding convention)
- Extend the card-click e2e to require_bound_identity=True, driven
through the channel's real _resolve_card_click production
* fix(channels): stabilize Feishu clarification delivery and routing
Option cards could fail through the real reply SDK, resume another conversation,
or leave a stale text hint after the answer was accepted.
Unify raw-card delivery and persist trusted thread aliases before showing buttons.
Keep protocol request IDs separate from bounded, one-time callback tokens; validate
originators and option pairs, and retire only the accepted card's text/button state.
Treat valid hidden human answers as turn boundaries so old replies and artifacts
do not leak into the next answer. Cover consecutive clarifications and /new routing
through real SDK request models, stores, and the run journal.
Validation: 147 focused tests and 29,421 offline tests passed (405 skipped);
strict blocking-I/O passed 245 tests (1 skipped). Ruff and guidance checks passed.
* fix(channels): preserve clarification context in Feishu cards
Button cards replaced the full clarification text but omitted its background,
leaving users without the context needed to choose an option.
Carry optional context from request metadata and render its original Markdown
before the question. Keep question-only cards for absent, blank, or invalid context.
Cover rendering and all real SDK delivery paths, and update both user guides.
Validation: 156 focused tests, 245 strict blocking-I/O tests, Ruff, and guidance
checks passed. The offline run recorded 29,381 passed, 49 failed, and 405 skipped;
48 import failures reproduced before the fix. All 49 cases passed on both versions
in isolated environments, and the Feishu-to-export order check passed 13 tests.
* fix(scheduler): preserve unchanged once schedules near execution
* docs: keep schedule guidance within the size limit
* style(scheduler): use English for the once-edit comment
* docs(scheduler): keep guidance within directory budgets
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(acp): resolve an absolute Windows agent command before spawning
A command configured as an absolute path without an extension -- npm's global
shim layout, e.g. C:\npm\mcode beside mcode.cmd -- was handed to the ACP SDK
unchanged and always failed: create_subprocess_exec applies no PATHEXT, so it
ran npm's extensionless shell script and died with WinError 193 (or raised
FileNotFoundError when only the .cmd exists). The separator guard from #6598
was too broad: only relative paths need to stay with the spawn, because their
meaning is owned by the ACP-workspace cwd, while an absolute path does not
depend on the cwd.
Absolute paths now go through the same shutil.which lookup as bare names: the
lookup searches the command's own directory and probes PATHEXT, and the
configured value is kept when nothing matches so the not-found remediation
still fires. Relative paths (bin/agent, ./bin/agent) still go to the spawn
unchanged.
Docs: README.md and backend/packages/harness/deerflow/tools/AGENTS.md now
describe the bare/absolute vs relative split.
AI disclosure: this change was drafted with AI assistance at my request, under
my direction and review.
* fix(acp): keep already-absolute lookup results unchanged
Review feedback on #6656 (willem-bd): os.path.abspath collapsed a `..` that
follows a symlink, so an absolute POSIX command like
`/srv/current/../shared/agent` (with `current -> releases/v1`) was rewritten to
`/srv/shared/agent` -- a path that either does not exist or names a different
launcher. The anchoring to the Gateway cwd exists only for relative PATH
lookup results, where shutil.which cannot know the cwd the spawn will use; an
already-absolute result needs no rewrite.
Regressions: a platform-independent stub-lookup test pinning the literal
parent-segment path, and a POSIX-only test that builds the symlink layout and
asserts the configured command is returned unchanged.
* fix(runtime): mark cancel takeovers with the orphan-recovery stop reason
A cross-worker cancel that takes over an expired-lease run writes error but
left stop_reason unset. `orphan_recovered` is the only signal that lets a
store-only SSE/`wait` consumer synthesize END after a heartbeat, because the
producer is known to be gone. Without it, a subscriber that outlives the
crashed owner waits forever for an END no surviving worker publishes; periodic
reconciliation ignores the row because it is already terminal.
Mark the takeover like periodic orphan recovery: claim_for_takeover already
accepts the stop reason and the reconcile path already passes it. The new
tests pin the takeover row's marker and prove a real takeover wakes a waiting
consumer; three existing takeover test doubles now forward the store argument.
* docs(gateway): keep guidance within the 48KB budget
The takeover stop-reason sentence pushed backend/app/gateway/AGENTS.md over its 48KB hard limit, failing agent-guidance and the backend guidance-budget test. Trim the sentence; the rationale stays in the manager.py comment.
* fix(gateway): complete streams after scheduler orphan recovery
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(agents): skip content blocks with unhashable type values
has_visible_content and ParentContextSnapshot.from_state classified a
content block's type with set membership (`in {"text", "output_text"}`
plus the _MEDIA_BLOCK_TYPES frozenset). `type: []` / `type: {}` are
legal JSON from a provider, and hashing an unhashable value raises
TypeError instead of skipping the block:
- has_visible_content runs on every model response through the
llm-error-handling (_raise_for_empty_response), terminal-response and
model-length middlewares, so one malformed block turned a completed
response into a failed run instead of a skipped block;
- from_state builds the history every background subagent dispatch is
seeded from, so the same block failed the dispatch.
Probe the type by equality with tuples so a non-string type simply fails
the probe, mirroring #6614's fix for extract_response_text and the
isinstance guard clarification already uses for form field types. The
media constant becomes a tuple too, so the second probe in the same loop
cannot move the crash one line down.
Tests pin the RED-to-GREEN cycle at both call sites: unit-level
has_visible_content, middleware-level empty-response classification,
and the dispatch snapshot across human/ai/tool messages.
Fixes#6663
Signed-off-by: yetuge <2219677952@qq.com>
* fix(agents): warn when a snapshot skips a non-string-typed block
Review follow-up on #6664: with the malformed block now skipped
instead of raising, a provider that systematically emits non-string
types would degrade the child's history with nothing in the logs to
explain it. The elif chain cannot log on fall-through —
reasoning/signature and tool-use exclusions are deliberate and must
stay quiet — so the signal keys on the type not being a str, matching
the guard the review suggested.
has_visible_content keeps its silence on purpose: the empty-response
retry already surfaces the symptom there.
Fixes#6663
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* fix(tavily): contain malformed web search and fetch payloads
web_search and web_fetch normalized the SDK's decoded 200 body without
validating its shape: a `results`/`failed_results` container of the wrong
type, a non-object entry, or a missing leaf field escaped as a raw
TypeError/KeyError/AttributeError instead of the structured format error
brave, serper, serply, sofya, tencent_wsa, unbrowse and groundroute return.
Validate the container with one shared helper, skip non-object entries while
preserving valid objects in order, normalize missing leaf fields to empty
text, and coerce a non-string raw_content in web_fetch before truncating.
Diagnostics carry only the malformed container's name and type, never the
query, URL, key or payload values.
backend/tests/test_tavily_response_shapes.py drives the real tools and SDK
over an offline HTTPX MockTransport; it fails 32 of 36 cases on main and
passes with this change, while test_tavily_tools.py stays green.
* docs(readme): note Tavily payload containment
Follow the Brave and GroundRoute README entries in the setup wizard step; the containment contract itself lives in community/tavily/AGENTS.md.
* fix(tavily): normalize null search fields and test mixed fetch lists
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(tui): keep a refused message in the composer
on_composer_input_submitted cleared the composer before _send_to_agent
decided whether it could start a run. A prompt typed during an active run,
while an interrupted run on the same thread was still stopping, or when the
worker failed to start was replaced by a notice and survived only in input
history.
_send_to_agent now reports whether it started a run, and the submit handler
puts the exact submitted text back with the cursor at its end, so Enter
resends it. Accepted messages and local slash commands clear the composer as
before.
* fix(tui): keep a launch message that fails to start in the composer
on_mount sent the launch message through _send_to_agent and ignored the
result, so a worker start failure lost the message: it never reached the
composer or input history. Route both the launch message and composer
submissions through one _restore_draft helper.
* docs(changelog): reference #6666 for the TUI refused-message fix
* fix(tui): keep the palette closed over a restored slash draft
Restoring a refused bare skill activation such as /tdd posted
TextArea.Changed, which reopened the command palette, so Enter filled the
skill instead of resending it. Remember the restored draft and keep the
palette closed until it is edited. Restore the history copy of the text too,
so a trimmed slash command matches what Up recalls.
* feat(authz): admin user-management UI and disabled-account copy (#4063 gap 3)
Frontend slice of the account lifecycle backend (#6566):
- Settings gains an admin-only Users page (core/auth/admin-users.ts ->
GET/PATCH /api/v1/admin/users): email, read-only role, disabled state,
and a suspend/restore toggle per row; the Gateway's rejection detail
(last-active-admin 409) renders verbatim. The nav entry is hidden for
non-admins and the page guards independently; ?settings=users joins the
deep-link whitelist (now an exhaustive Record over SettingsSection).
- A session that turns out disabled maps account_disabled to a login
redirect that states the reason on all three surfaces: the SSR
AuthResult tag (consumed exhaustively by the workspace/artifacts/auth
layouts), the shared fetcher's 401 interception, and
AuthProvider.refreshUser. No next target is preserved - a disabled
account cannot sign back in. Login failures surface the localized
account_disabled copy when the code is present.
- userSchema stays disabled-optional (absent = not disabled); zh/en
locales and docs updated in the same change.
* fix(authz): address review on admin user UI and disabled copy (#4063 gap 3)
Review follow-ups on PR #6641:
- Direct /login visits with a suspended session now surface the SSR
account_disabled verdict through AuthDisabledContext instead of
discarding it (willem-bd): the auth layout wraps pages with the
provider and the login form seeds its reason from it — no redirect,
so no loop. Regression tests cover both verdict states.
- admin-users.ts parses both endpoints against userSchema /
z.array(userSchema) (zhfeng): malformed payloads fail the load
instead of rendering phantom rows; toggle error copy is scoped to
GatewayApiError with a localized fallback for anything else.
- The localized account_disabled copy is login-only (zhfeng): sign-up
keeps the backend message verbatim so a suspended account's email
cannot be probed through the register surface.
- Settings page folds the load banner and toggle error into a single
alert region (zhfeng); adds coded-envelope 409, malformed-row, and
load-failure tests; fetcher.ts documents the 401 body-consumption
contract.
- prettier across touched files (CI format gate).
The admin-users query key keeps user?.id, matching the
model-settings-page precedent (converge later per review).
* feat(dev): add a two-Gateway multi-instance harness script
There was no way to run the deployment.multi_instance topology locally:
scripts/serve.sh pins one Gateway to port 8001, the local nginx config has a
single upstream, and getting two processes past the multi-instance startup gate
needs Postgres, Redis, a shared DEER_FLOW_HOME, shared auth secrets and a
per-process DeerMem retrieval index by hand.
scripts/dev_multi_instance.sh (make dev-multi / dev-multi-check /
dev-multi-down) starts throwaway Postgres and Redis containers on loopback,
renders a config from the developer's config.yaml (or config.example.yaml),
and runs Gateway A on 8001 and Gateway B on 8011 with one shared home, one
copy of extensions_config.json and generated AUTH_JWT_SECRET,
DEER_FLOW_INTERNAL_AUTH_TOKEN and DEER_FLOW_CREDENTIALS_KEY values, plus an
optional round-robin nginx on 2027. B starts after A answers /health: Alembic
migrations are advisory-locked, but the LangGraph checkpointer/store setup()
that follows is not.
scripts/dev_multi_instance.py holds the testable parts: the config overlay
(neutralizing and reporting settings the gate refuses), the secrets file, the
nginx config, and `check`, which automates the scriptable cross-instance
acceptance checks: readiness with stream_bridge ok, A-issued sessions and the
internal token accepted by B, thread/upload/artifact visibility, a skill
toggle seen by B, nginx balancing, and SSE join on B with Last-Event-ID resume
on A. Tests pin that the rendered config passes the real
_enforce_postgres_for_multi_worker gate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link the two-Gateway harness entries to #6613
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(dev): keep the multi-instance harness inside its own state
Address review findings on the two-Gateway harness (#6613):
- Containers carry a deerflow.harness.state-dir label; up/down remove only
containers of their own state dir and refuse those of another checkout or
DEERFLOW_MI_STATE_DIR instead of force-removing them.
- Containers are removed with `docker rm -f -v`, so the anonymous volumes
the postgres and redis images declare no longer leak on every up/down.
- `up` installs an EXIT trap once it starts containers, so any later failure
(including B's port being taken after preflight) tears down A and both
containers; start_gateway returns failures instead of exiting the script,
and the Gateway is disowned so stopping it prints no job notice.
- The overlay removes any checkpointer section (a postgres one would write
checkpoints and the Store into the developer's database), points an
explicit sandbox.ownership and a Redis checkpoint_cache at the harness
Redis (keeping ownership timing), drops DeerMem storage_path (both
spellings) and the blob_storage root so data lands in the harness home,
gives a local AIO sandbox a state-dir-specific container_prefix, and
reports a non-DeerMem memory backend it keeps sharing.
- nginx paths are rendered as quoted string literals ('$' and control
characters are refused), so a state dir with spaces works.
- `check` skips the run-based SSE check when model discovery did not
succeed, unless --with-llm, and enforces the SSE time budget per raw line
so heartbeat-only traffic cannot hold it open.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(scripts): keep the harness guidance under the AGENTS.md soft limit
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat: return history search excerpts near keyword matches
* docs: use English for history search guidance and comments
* docs: sync Chinese history search excerpt contract
* docs: clarify history excerpt fallback for oversized matches
* fix(history): skip oversized occurrences when selecting excerpts
Continue scanning casefolded substring matches until an occurrence fits the 600-character excerpt, including overlaps. Cover later fitting hits through synchronous and asynchronous active/archive search and read tools, and update the documented contract.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(security): add DEER_FLOW_CREDENTIALS_KEY for at-rest credential encryption
There was no at-rest encryption key. ChannelCredentialCipher had no
production caller: the channel service, the scheduled-task notification
outbox and the channel-connections router all built
ChannelConnectionRepository without a cipher, so store_credentials refused
(RuntimeError) and get_credentials always returned None.
- New harness module deerflow.config.credentials_key: env-only
DEER_FLOW_CREDENTIALS_KEY (comma-separated Fernet keys; first encrypts,
all decrypt via MultiFernet), CredentialsCipher with encrypt_text /
decrypt_text / rotate_text, a fernet:v2: prefix with legacy fernet:v1:
values still readable, and get_credentials_cipher() as the process-wide
entry point. Malformed keys are refused without echoing key material.
Without the variable a single instance generates
{base_dir}/.credentials_key. Never derived from AUTH_JWT_SECRET.
- New deerflow.config.secret_file.read_or_create_secret_file: O_EXCL
create (0600) plus read-back with a bounded settle window, shared by
.credentials_key, .jwt_secret (fixes two replicas cold-starting on a
shared volume keeping different JWT secrets) and the managed-model key.
- ChannelCredentialCipher now builds on CredentialsCipher (same
constructors); the Gateway loads the cipher off the event loop at
startup when channel_connections is enabled and passes it to every
ChannelConnectionRepository. Undecryptable values stay "missing + warn".
- Startup gate: a malformed key stops startup in any topology; a declared
multi-instance deployment with channel_connections enabled and no key
is refused. Plain multi-worker shares the generated file and is allowed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(deploy): generate and inject DEER_FLOW_CREDENTIALS_KEY in Helm, compose and deploy.sh
- Helm: the app Secret generates a Fernet key (randBytes 32 translated to
the urlsafe alphabet) and preserves it across upgrades via lookup like
AUTH_JWT_SECRET; the gateway reads it through an optional secretKeyRef
so a user-managed existingAppSecret without it still boots (the Gateway
gate refuses only multi-instance deployments that store credentials).
README, NOTES and values document backup, rotation and existingAppSecret.
- scripts/deploy.sh resolves the key shell -> .env -> persisted file ->
generated, persisting it as $DEER_FLOW_HOME/.credentials_key (0600,
noclobber/O_EXCL with read-back): the same file the Gateway generates in
its runtime home, so either side's key is the one both keep.
- Both compose files pass DEER_FLOW_CREDENTIALS_KEY=${...:-} to the gateway.
- .env.example entry with a generation hint; README and CHANGELOG notes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): make abandoned secret-file replacement single-winner
A rename is last-writer-wins, so replicas that restart together and all
find the same abandoned empty secret file could each replace it and keep
a different value. The process that exclusively creates
<name>.replacing now owns the replacement, re-checks the file is still
empty, and publishes by renaming the claim over it; peers wait for that
value, bounded so a crashed replacer cannot block startup forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link DEER_FLOW_CREDENTIALS_KEY entries to #6611
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): name the setting that turns off a credentials-key consumer
The multi-instance refusal ended with "or disable
channel_connections.enabled=true"; it now says "or set
channel_connections.enabled=false".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): never expose a secret file before its value is written
O_EXCL published the name before its content, so a creator that stalled
mid-write left an empty file a peer replaced after the settle window; the
resumed creator then returned a value nobody persisted, or deleted the
peer's valid file when its own write failed. Creators now write and sync
a private temporary file and hard-link it to the name, which fails if
the name exists; a failed creator only removes its own temporary file.
Filesystems without hard links publish through the same single-winner
<name>.replacing claim that replaces abandoned empty files, so creation
and replacement are serialized there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(models): recover an empty managed-model key when no catalog exists
An interrupted first save could leave an empty key file; every later save
then passed empty bytes to Fernet and the endpoint returned 503 until the
file was removed by hand. An empty key with no catalog encrypted nothing,
so it now goes through the secret-file recovery path; next to an existing
catalog it is still refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(docker): pass the dotenv DEER_FLOW_CREDENTIALS_KEY through make docker-start
scripts/docker.sh runs Compose from docker/ without --env-file, so the dev
gateway's DEER_FLOW_CREDENTIALS_KEY=${DEER_FLOW_CREDENTIALS_KEY:-} entry
interpolated to empty and overrode the key the checkout .env supplied via
env_file. Export it from the dotenv like AUTH_TRUSTED_PROXIES.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(acp): resolve a bare agent command through PATH before spawning
`invoke_acp_agent` handed `acp_agents.<name>.command` straight to the ACP
SDK, which spawns it with `asyncio.create_subprocess_exec`. That API applies
no `PATHEXT` lookup on Windows, so a bare npm launcher (`npx`, `mcode`)
raised `FileNotFoundError` although its `.cmd` shim was on `PATH`. The
documented `command: npx` and `command: mcode` shapes could not start, and
the tool reported "was not found on PATH" for an installed launcher.
Resolve the configured command with `shutil.which` before spawning — the way
the MCP Python SDK normalizes stdio server commands — and keep the configured
value when nothing matches, so the existing install guidance still fires.
Fixes#6597
* fix(acp): resolve the ACP agent command off the event loop
`_resolve_agent_command` calls `shutil.which`, which stats the filesystem.
Calling it inline in the async `_invoke` tripped the strict BlockBuster gate
(`BlockingError: Blocking call to os.access`) for every command that resolves,
and would block the Gateway event loop in production.
Offload the lookup with `asyncio.to_thread`, matching the neighbouring
`_get_work_dir`, `_build_acp_mcp_servers`, and `_format_invocation_error`
calls.
The blocking-io anchor configured `command="test-agent"`, a name that resolves
nowhere, so `shutil.which` short-circuited before `os.access` and the gate
never covered the new call. Point it at `sys.executable` so the lookup is
really exercised, and cover the unresolvable-guidance branch end-to-end with a
new test.
Refs #6597
* fix(acp): keep the configured command in ACP remediation text
Review follow-ups on the PATH resolution:
- Only bare names are resolved. A configured path is interpreted by the spawn
against the per-thread ACP workspace, so resolving it against the Gateway's
working directory could pick a different file or hand the spawn a path that
then fails.
- `_format_invocation_error` receives `agent_config.command` again. A command
can resolve on PATH and still fail to launch (a broken shebang, for example),
and reporting the resolved path made "was not found on PATH" false and
dropped the `codex-acp` hint.
- The blocking-io anchor uses a bare name backed by a shim on a controlled PATH
instead of `sys.executable`, which the bare-name scoping no longer resolves.
Tests: the pass-through assertion in
`test_invoke_acp_agent_uses_fixed_acp_workspace` pins PATH, and new coverage
covers the bare-name scoping and the spawn-failure message.
Refs #6597
* docs(acp): correct the path-resolution rationale
A configured path is resolved by the spawn in its own process, so the
pass-through reason must not claim it is resolved against the ACP workspace:
that only holds on POSIX, while Windows resolves a relative executable against
the calling process's directory. State the platform-neutral reason instead.
Refs #6597
* fix(acp): resolve the launcher against the agent's own PATH
The ACP SDK merges acp_agents.<name>.env over its inherited environment at spawn time, so a configured PATH, not the Gateway's, decides which launcher the agent would find. Resolve a bare command against that effective PATH, and use the Gateway's only when the agent does not override it.
Return a successful lookup as an absolute path: a relative PATH entry makes shutil.which yield a result relative to the Gateway's cwd, while the spawn runs with cwd set to the ACP workspace, where that path resolves elsewhere or not at all.
Regressions: a real-subprocess test with same-named launchers in the Gateway and agent PATHs (asserting which one ran), and a real-subprocess test whose PATH holds a relative directory while the ACP workspace is elsewhere. Both are red on the pre-fix code.
* Fail closed on delegated fetch backends to close SSRF gap
* fix(community): resolve delegated-fetch credentials from one config snapshot
Address review feedback on the fail-closed SSRF guard:
- firecrawl/fastcrw client construction no longer re-reads get_app_config(),
so the API key and endpoint come from the same revision the endpoint screen
validated (prevents cross-revision credential misrouting).
- fix the browserless test double to accept the new two-argument
_get_browserless_client signature.
- document the check-only (DNS rebinding) caveat on
validate_delegated_backend_url.
- add a CHANGELOG upgrade note for the default self-hosted backend break.
* fix(doctor): consolidate delegated fetch backend checks and deployment guidance
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
