* fix(gateway): read subagent batch exports in small pages
The results.jsonl export loaded 500 full rows per page before streaming
them. A row can hold max_result_chars of report plus as much evidence
JSON (result_artifact), so with the default limit one request peaked
around 280 MB and decoding a page's JSON columns stalled the event loop
for about half a second; at the 1,000,000 ceiling a page was roughly
1.7 GB.
Read 10 rows per page instead. Output and ordering are unchanged; a
full export of 1,000 default-sized rows now peaks at 6 MB with a 46 ms
worst stall and takes the same time.
* docs(changelog): reference #6706
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(frontend): keep the thread list offset past hidden side chats
Primary thread lists hide sidecar threads on the client, so a page of 50
visible chats can consume more server rows. fetchInfiniteThreadsPage kept
that server offset in a Symbol property on the page array, but TanStack
Query's structural sharing (replaceEqualDeep) rebuilds a changed array
without symbol keys, and the list cache writers (rename, unread, pin,
delete, upsert) rebuild pages with map/filter. After the first refetch or
cache write, getInfiniteThreadsNextPageParam fell back to the sum of the
visible page lengths, so the next page started at 50 instead of 53 and
re-fetched rows already loaded. The same fallback treated a full page
that lost a row to a local delete as the end of the list.
Each cached page is now { threads, hiddenCount, hasMore }. The next offset
is the sum of visible and hidden rows over the loaded pages, so local
deletes and inserts still move it with the server list, and the end of the
list comes from the server response rather than the page length.
useInfiniteThreads projects the pages back to thread arrays with select,
so list components are unchanged.
* test(frontend): pin the sidebar's next-page offset past side chats in E2E
Three side chats in the first 50 server rows push the sidebar's second
page to offset 53. After a rename through the sidebar menu, scrolling to
the sentinel must request offset 53, not 50. On main this test fails with
the request at offset 50.
* docs(changelog): reference #6702
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(frontend): static-website gate and deep-link fallback for user management (#6641 follow-ups)
Close the two leftovers from the #6641 review:
- user-settings-page now gates on isStaticWebsiteOnly() like model management
(the static demo user is nominally an admin, but there is no Gateway to
query), and the users adminOnly copy gains the same 'unavailable in demos'
sentence the models copy already carries.
- the settings dialog derives the active section from the nav actually offered
to the current user, so a ?settings=users deep link (or a stale dialog
store) for a non-admin falls back to the first section instead of rendering
an orphan panel with no highlighted entry.
Tests: static demo loads nothing (mutation-verified red), non-admin deep link
falls back to models, admin deep link lands on user management; full unit
suite green.
* test(frontend): pin the static-on x admin matrix cell at the dialog level
Dialog-level coverage for the real demo path (review follow-up on #6691):
admin + isStaticWebsiteOnly() + ?settings=users keeps the Users nav entry
(the splice keys on the role only) while the page-level gate shows the
admin-only note and issues no query. Mutation-verified: adding the mode
flag to the nav splice turns this test red and only this test.
* feat(channels): render Feishu option clarifications as button cards
When the agent's ask_clarification carries options, the Feishu channel
now renders the question as an interactive card with one button per
option instead of plain text. This is the outbound half: card
construction plus a process-local pending registry that a later
card-action callback handler will validate clicks against.
- New feishu_card_interaction module owns request extraction, button
card rendering, and the pending registry (30-minute TTL, matching the
typed-reply clarification window)
- Manager attaches the clarification's structured human_input payload to
the final outbound's metadata so channels can render richer
affordances without touching the Gateway protocol
- Feishu channel renders the button card only when
card_interaction_enabled is set (default off); free-text, form, and
over-four-option clarifications keep the plain-text path
- Button cards bypass the markdown wrapper via a raw_card_content flag
on the card send/patch helpers; plain-text call signatures unchanged
* feat(channels): close the Feishu button-card click loop
The inbound half of interactive clarification cards: a click on an
option button validates, resolves the clarification, and marks the
card in place.
- Register the card-action callback only when card_interaction_enabled
is set; clicks arrive as WS EVENT frames through the SDK's callback
processor map, and the synchronous return value becomes the toast ack
- consume_if_originator atomically validates operator identity against
the pending entry: non-originators get a refusal toast and the entry
survives; unknown or expired requests are toast-only and dropped
- A validated click consumes once, updates the card to a terminal
resolved state, and injects a hidden inbound carrying the
human_input_response so the run resumes on the originating thread
with Web-card parity
- Manager translates the inbound's human_input_response metadata into
the human message's additional_kwargs with hide_from_ui set
* docs(channels): document Feishu interactive clarification cards
- README / README_zh: optional card_interaction_enabled setup, calling
out that card.action.trigger lives under the console's callback
configuration tab (separate from event subscriptions) and requires
publishing a new app version
- config.example.yaml: commented flag with the same console caveat
- backend/AGENTS.md: module ownership note for feishu_card_interaction
* fix(channels): close Feishu card-click routing and ack gaps from review
Address review on #6670:
- Carry the routing identity (topic_id, connection_id, owner_user_id)
on the pending entry and the synthetic inbound, so clicks resolve to
the clarified thread in topic-scoped chats and pass bound-identity
admission instead of opening a new thread or being rejected
- Validate option_value against the registered options at consume time:
an unknown value is treated like a stale card and the entry is kept
- Return a warning toast when a click cannot be dispatched, and flip
the card to its resolved state only after the inbound is durably in
the bus — never display a choice that was not recorded
- Drop the dead mark_sent/consume registry APIs and pass the payload's
own option ids through to the buttons
- Add an end-to-end test through the manager's thread resolution
pinning that a card answer reuses the clarified thread
* fix(channels): carry workspace_id on the Feishu card-click inbound
The connections-enabled admission path re-reads the binding with the
message's workspace_id, and Feishu connections are bound with
workspace_id = chat_id. The synthetic inbound set connection_id and
owner_user_id but never workspace_id, so the re-read missed and every
click was rejected as unbound after the entry was consumed.
- Set inbound.workspace_id from the entry's chat_id (Feishu's
chat-as-workspace binding convention)
- Extend the card-click e2e to require_bound_identity=True, driven
through the channel's real _resolve_card_click production
* fix(channels): stabilize Feishu clarification delivery and routing
Option cards could fail through the real reply SDK, resume another conversation,
or leave a stale text hint after the answer was accepted.
Unify raw-card delivery and persist trusted thread aliases before showing buttons.
Keep protocol request IDs separate from bounded, one-time callback tokens; validate
originators and option pairs, and retire only the accepted card's text/button state.
Treat valid hidden human answers as turn boundaries so old replies and artifacts
do not leak into the next answer. Cover consecutive clarifications and /new routing
through real SDK request models, stores, and the run journal.
Validation: 147 focused tests and 29,421 offline tests passed (405 skipped);
strict blocking-I/O passed 245 tests (1 skipped). Ruff and guidance checks passed.
* fix(channels): preserve clarification context in Feishu cards
Button cards replaced the full clarification text but omitted its background,
leaving users without the context needed to choose an option.
Carry optional context from request metadata and render its original Markdown
before the question. Keep question-only cards for absent, blank, or invalid context.
Cover rendering and all real SDK delivery paths, and update both user guides.
Validation: 156 focused tests, 245 strict blocking-I/O tests, Ruff, and guidance
checks passed. The offline run recorded 29,381 passed, 49 failed, and 405 skipped;
48 import failures reproduced before the fix. All 49 cases passed on both versions
in isolated environments, and the Feishu-to-export order check passed 13 tests.
* fix(skillscan): match fork bombs by shape, not spelling
`resource-fork-bomb` and the fork-bomb branch of `shell-destructive-command`
only matched the canonical `:(){ :|:& };:` rendering, so renamed or respaced
payloads produced no findings and passed the blocking gate.
Match the general shape instead - a shell function whose body recurses through
a pipe into a background job - keeping the pre-LLM static gate at least as
wide as the runtime sandbox audit middleware it backstops.
Closes#6651.
* fix(skillscan): bound the fork-bomb pattern and require self-recursion
Addresses the three review findings on #6689:
- Bound every quantifier in the shared fork-bomb pattern so a whitespace-free
blob costs O(n) instead of O(n^2) (measured ~0.15s at 200KB; the previous
spelling took ~5.5s at just 50KB).
- Require the function name to recur after the pipe (backreference), so a
benign pipeline such as `backup() { rsync -a src/ | logger & }` no longer
blocks as a CRITICAL finding.
- Keep both gates byte-identical again: skillscan's _FORK_BOMB_PATTERN and the
sandbox_audit_middleware fork-bomb entry, pinned by a parity test.
- The code comment now describes exactly what the pattern checks; pipe-less
self-backgrounding recursion and the spaced `|&` spelling are documented as
out of scope.
- New regression tests: near-miss pipelines in both suites, the parity
invariant, and a 200KB single-token scan-time ceiling.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(frontend): show a date-only run_at at midnight in the task's zone
A chat-created one-time task can store run_at as a bare date
("2026-12-07"). The server reads it with datetime.fromisoformat as
midnight in the task's timezone. onceRunAtInstant instead returned it
unchanged: its offset check matched the trailing "-07" as a UTC offset,
and its local-time pattern needed a time anyway. new Date() then read it
as UTC midnight, so the schedule line, chat card and edit form showed
08:00 in Asia/Shanghai and the previous evening west of UTC.
A bare date is now matched first and converted as midnight in the task's
zone, the same way a time-bearing local run_at is.
* docs(changelog): reference #6692
* test(frontend): pin the clock in the date-only run_at E2E
The detail page formats the schedule line against the real clock: it
drops the year inside the current year and switches to relative labels
in the final week. The exact-text assertion therefore failed throughout
2027 ("Once, Dec 7 00:00 ...") and again as "Once, Tue 00:00 ..." in the
week before the run. Fix "now" at 2026-10-01, the same instant the unit
test uses, so the assertion stays deterministic.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scheduler): preserve unchanged once schedules near execution
* docs: keep schedule guidance within the size limit
* style(scheduler): use English for the once-edit comment
* docs(scheduler): keep guidance within directory budgets
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(frontend): return to the opened workspace page after signing in
Opening a workspace page without a valid session sent the user to a bare
/login: the workspace layout called redirect("/login"), and a layout
cannot read the request URL, so the path and query were both lost. A
scheduled-task notification link (/workspace/scheduled-tasks?task_id=X)
opened after the session expired therefore landed on the default
workspace after sign-in.
For an unauthenticated verdict the layout now renders
WorkspaceLoginRedirect, which replaces the page with
/login?next=<path+query> from the browser. It renders nothing else, so
the workspace pages under it never mount.
AuthProvider.refreshUser and useRequireAuth kept the path but dropped
the query. They now share currentReturnPath() with the fetcher, which
already kept both.
* docs(changelog): reference #6681
* fix(frontend): keep the fragment in the sign-in return path
currentReturnPath() kept the path and query but not location.hash, so
an anchored deep link (/workspace/chats/x#section) lost its anchor across
the login round trip. It is now the single source for all four bounce
sites, so include the fragment. The server never receives the fragment,
so only the browser-side bounce can keep it. validateAuthNextPath, the
backend's validate_next_param and its quote() into /auth/callback all
carry a '#' through unchanged.
The models API test stubbed window.location without hash; a real
Location always has one, so the stub now carries it.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(acp): resolve an absolute Windows agent command before spawning
A command configured as an absolute path without an extension -- npm's global
shim layout, e.g. C:\npm\mcode beside mcode.cmd -- was handed to the ACP SDK
unchanged and always failed: create_subprocess_exec applies no PATHEXT, so it
ran npm's extensionless shell script and died with WinError 193 (or raised
FileNotFoundError when only the .cmd exists). The separator guard from #6598
was too broad: only relative paths need to stay with the spawn, because their
meaning is owned by the ACP-workspace cwd, while an absolute path does not
depend on the cwd.
Absolute paths now go through the same shutil.which lookup as bare names: the
lookup searches the command's own directory and probes PATHEXT, and the
configured value is kept when nothing matches so the not-found remediation
still fires. Relative paths (bin/agent, ./bin/agent) still go to the spawn
unchanged.
Docs: README.md and backend/packages/harness/deerflow/tools/AGENTS.md now
describe the bare/absolute vs relative split.
AI disclosure: this change was drafted with AI assistance at my request, under
my direction and review.
* fix(acp): keep already-absolute lookup results unchanged
Review feedback on #6656 (willem-bd): os.path.abspath collapsed a `..` that
follows a symlink, so an absolute POSIX command like
`/srv/current/../shared/agent` (with `current -> releases/v1`) was rewritten to
`/srv/shared/agent` -- a path that either does not exist or names a different
launcher. The anchoring to the Gateway cwd exists only for relative PATH
lookup results, where shutil.which cannot know the cwd the spawn will use; an
already-absolute result needs no rewrite.
Regressions: a platform-independent stub-lookup test pinning the literal
parent-segment path, and a POSIX-only test that builds the symlink layout and
asserts the configured command is returned unchanged.
* fix(runtime): mark cancel takeovers with the orphan-recovery stop reason
A cross-worker cancel that takes over an expired-lease run writes error but
left stop_reason unset. `orphan_recovered` is the only signal that lets a
store-only SSE/`wait` consumer synthesize END after a heartbeat, because the
producer is known to be gone. Without it, a subscriber that outlives the
crashed owner waits forever for an END no surviving worker publishes; periodic
reconciliation ignores the row because it is already terminal.
Mark the takeover like periodic orphan recovery: claim_for_takeover already
accepts the stop reason and the reconcile path already passes it. The new
tests pin the takeover row's marker and prove a real takeover wakes a waiting
consumer; three existing takeover test doubles now forward the store argument.
* docs(gateway): keep guidance within the 48KB budget
The takeover stop-reason sentence pushed backend/app/gateway/AGENTS.md over its 48KB hard limit, failing agent-guidance and the backend guidance-budget test. Trim the sentence; the rationale stays in the manager.py comment.
* fix(gateway): complete streams after scheduler orphan recovery
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): stop resolving sandbox and ACP env values a second time
AppConfig.resolve_env_variables already replaces every $VAR when
config.yaml loads. The AIO, E2B and OpenSandbox providers and the ACP
tool then resolved the result again, so a secret that itself starts with
$ (a bcrypt hash, a generated password) was looked up as a variable name
and reached the sandbox or agent as "" -- or as an unrelated host
variable's value when that name existed.
Pass the loaded values through unchanged, as BoxLite and the local
provider already do. The removed helpers' str() coercion was dead: both
fields are typed dict[str, str] and reject non-strings at load.
* docs(changelog): reference #6673
* docs(config): pin when sandbox and ACP env values are resolved
ACPAgentConfig.env and the SandboxConfig docstring said $ values are
resolved from the host environment without saying when, which reads as
"at spawn time" and invites a consumer-side second pass again. Both now
say it happens when config is loaded, matching SandboxConfig.environment.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
test_skillscan_env_launcher.py and test_channels.py put payloads of up to
52 KB directly in parametrize values, so pytest's generated item ids exceed
the 32767-character Windows environment-variable limit: PYTEST_CURRENT_TEST
raises ValueError at setup and teardown even though the products under test
behave correctly, and the giant ids flood -q reports.
The skillscan linearity test additionally spawned one child process per
chain. A fresh child spends 9-18 s importing the skills stack before the
4-168 ms scan, so its fixed 10 s timeout measured the import instead of the
scan and flipped to subprocess.TimeoutExpired on a loaded host.
Give the oversized cases short explicit ids, scan every chain in one child
process fed through stdin, and enforce the linearity bound on each scan
inside that child (5 s) while a 60 s outer timeout covers the import.
* fix(lark): publish the skill file and manifest atomically
_write_skill_file and _write_manifest wrote their final paths with a truncating
write, while the runtime files in the same module already go through
_publish_bytes_atomically. A partial skill file or manifest is unparseable, and
the manifest carries the content hash the install is verified against, so a
half-written one cannot be repaired. Route both through the existing helper.
* fix(lark): address skill publication review feedback
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(agents): skip content blocks with unhashable type values
has_visible_content and ParentContextSnapshot.from_state classified a
content block's type with set membership (`in {"text", "output_text"}`
plus the _MEDIA_BLOCK_TYPES frozenset). `type: []` / `type: {}` are
legal JSON from a provider, and hashing an unhashable value raises
TypeError instead of skipping the block:
- has_visible_content runs on every model response through the
llm-error-handling (_raise_for_empty_response), terminal-response and
model-length middlewares, so one malformed block turned a completed
response into a failed run instead of a skipped block;
- from_state builds the history every background subagent dispatch is
seeded from, so the same block failed the dispatch.
Probe the type by equality with tuples so a non-string type simply fails
the probe, mirroring #6614's fix for extract_response_text and the
isinstance guard clarification already uses for form field types. The
media constant becomes a tuple too, so the second probe in the same loop
cannot move the crash one line down.
Tests pin the RED-to-GREEN cycle at both call sites: unit-level
has_visible_content, middleware-level empty-response classification,
and the dispatch snapshot across human/ai/tool messages.
Fixes#6663
Signed-off-by: yetuge <2219677952@qq.com>
* fix(agents): warn when a snapshot skips a non-string-typed block
Review follow-up on #6664: with the malformed block now skipped
instead of raising, a provider that systematically emits non-string
types would degrade the child's history with nothing in the logs to
explain it. The elif chain cannot log on fall-through —
reasoning/signature and tool-use exclusions are deliberate and must
stay quiet — so the signal keys on the type not being a str, matching
the guard the review suggested.
has_visible_content keeps its silence on purpose: the empty-response
retry already surfaces the symptom there.
Fixes#6663
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* fix(tavily): contain malformed web search and fetch payloads
web_search and web_fetch normalized the SDK's decoded 200 body without
validating its shape: a `results`/`failed_results` container of the wrong
type, a non-object entry, or a missing leaf field escaped as a raw
TypeError/KeyError/AttributeError instead of the structured format error
brave, serper, serply, sofya, tencent_wsa, unbrowse and groundroute return.
Validate the container with one shared helper, skip non-object entries while
preserving valid objects in order, normalize missing leaf fields to empty
text, and coerce a non-string raw_content in web_fetch before truncating.
Diagnostics carry only the malformed container's name and type, never the
query, URL, key or payload values.
backend/tests/test_tavily_response_shapes.py drives the real tools and SDK
over an offline HTTPX MockTransport; it fails 32 of 36 cases on main and
passes with this change, while test_tavily_tools.py stays green.
* docs(readme): note Tavily payload containment
Follow the Brave and GroundRoute README entries in the setup wizard step; the containment contract itself lives in community/tavily/AGENTS.md.
* fix(tavily): normalize null search fields and test mixed fetch lists
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
_write_lark_cli_runtime_from_archive and _write_lark_cli_sandbox_launcher wrote
their file with a truncating open and chmod 0o755 afterwards, so a crash mid-write
left an executable fragment behind - for the extracted binary that file can be
large. Publish through a temporary file plus os.replace, as the flow-state writer
in the same module already does.
* fix(tools): publish the default agent's SOUL.md atomically
setup_agent persisted the default agent's global SOUL.md with a
truncating write_text. The open truncates the previous file before the
new content lands, so an interrupted setup — process kill, disk full,
an exception while flushing — leaves a truncated persona file, and the
prompt builder reads SOUL.md on every run, so the agent then silently
runs with part of its instructions missing.
The same file is already published atomically everywhere else: the
custom-agent store stages it to a temporary and commits with
os.replace, and #6618 replaced the identical truncating pattern for
the Lark CLI runtime files the same day. The default agent's copy was
the one remaining write of this file that could tear.
Publish through a staged temporary plus os.replace, mirroring the
store's _stage_temp commit, and clean the temporary up when the
publish fails so no partial artifact survives.
Tests: a failed publish (os.replace raising) keeps the previous
SOUL.md byte-for-byte and leaves no temporary behind, a control pins
that the old truncating write loses the content in the same failure,
and the happy path writes the full file.
Fixes#6667
Signed-off-by: yetuge <2219677952@qq.com>
* fix(tools): keep SOUL.md's mode and clean the temporary on staging failure
Review follow-up on #6668, both non-blocking suggestions that guard the
publish itself:
- NamedTemporaryFile creates the temporary with mode 0600, so the first
publish made the global SOUL.md 0600 (the old write_text produced 0644
& ~umask) and every re-publish clobbered a mode an operator had set.
Stat the target and chmod the temporary to that mode (0644 for a first
publish) before the replace — the order #6618 codified for the Lark
runtime files and atomic_write_extensions_config already follows.
- A failure inside the staging block (handle.write or the implicit
flush/close, e.g. ENOSPC) raised before the try started, orphaning the
delete=False temporary in base_dir. One try/finally now spans staging
and publishing, so neither phase can leave a temporary behind.
The exact POSIX mode semantics were verified outside the Windows host
(WSL Ubuntu: mkstemp is 0600, first publish is 0644, a re-publish keeps
an operator-set 0640, and both failure phases leave no temporary); the
exact-mode tests are therefore POSIX-gated, with a cross-platform test
pinning that the chmod lands on the temporary before the replace and a
staging-failure test that is red on the previous head.
The shared-helper suggestion is left as a follow-up: folding six
independent publishers into one helper is a cross-module refactor, not
part of this bug fix.
Fixes#6667
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* fix(tui): keep a refused message in the composer
on_composer_input_submitted cleared the composer before _send_to_agent
decided whether it could start a run. A prompt typed during an active run,
while an interrupted run on the same thread was still stopping, or when the
worker failed to start was replaced by a notice and survived only in input
history.
_send_to_agent now reports whether it started a run, and the submit handler
puts the exact submitted text back with the cursor at its end, so Enter
resends it. Accepted messages and local slash commands clear the composer as
before.
* fix(tui): keep a launch message that fails to start in the composer
on_mount sent the launch message through _send_to_agent and ignored the
result, so a worker start failure lost the message: it never reached the
composer or input history. Route both the launch message and composer
submissions through one _restore_draft helper.
* docs(changelog): reference #6666 for the TUI refused-message fix
* fix(tui): keep the palette closed over a restored slash draft
Restoring a refused bare skill activation such as /tdd posted
TextArea.Changed, which reopened the command palette, so Enter filled the
skill instead of resending it. Remember the restored draft and keep the
palette closed until it is edited. Restore the history copy of the text too,
so a trimmed slash command matches what Up recalls.
Refresh the locked lark-oapi from 1.5.5 to 1.7.3 without touching the
declared >=1.4.0 constraint, which already admits the new release.
Deployments syncing from the lockfile move off the 1.6.x line, where a
server-side protocol drift reportedly left long-connection clients
connected but receiving no event pushes. Keeping the constraint as-is
avoids shutting out downstream consumers pinned to older but working
releases (1.5.5's WS event routing, including the callback processor map
used by register_p2_card_action_trigger, is verified intact in source).
- Refresh uv.lock only: lark-oapi 1.5.5 -> 1.7.3, specifier unchanged
- Feishu channel, channel-connections, and thread-binding suites pass
on 1.7.3 (111 passed, 1 skipped); ruff clean
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(authz): admin user-management UI and disabled-account copy (#4063 gap 3)
Frontend slice of the account lifecycle backend (#6566):
- Settings gains an admin-only Users page (core/auth/admin-users.ts ->
GET/PATCH /api/v1/admin/users): email, read-only role, disabled state,
and a suspend/restore toggle per row; the Gateway's rejection detail
(last-active-admin 409) renders verbatim. The nav entry is hidden for
non-admins and the page guards independently; ?settings=users joins the
deep-link whitelist (now an exhaustive Record over SettingsSection).
- A session that turns out disabled maps account_disabled to a login
redirect that states the reason on all three surfaces: the SSR
AuthResult tag (consumed exhaustively by the workspace/artifacts/auth
layouts), the shared fetcher's 401 interception, and
AuthProvider.refreshUser. No next target is preserved - a disabled
account cannot sign back in. Login failures surface the localized
account_disabled copy when the code is present.
- userSchema stays disabled-optional (absent = not disabled); zh/en
locales and docs updated in the same change.
* fix(authz): address review on admin user UI and disabled copy (#4063 gap 3)
Review follow-ups on PR #6641:
- Direct /login visits with a suspended session now surface the SSR
account_disabled verdict through AuthDisabledContext instead of
discarding it (willem-bd): the auth layout wraps pages with the
provider and the login form seeds its reason from it — no redirect,
so no loop. Regression tests cover both verdict states.
- admin-users.ts parses both endpoints against userSchema /
z.array(userSchema) (zhfeng): malformed payloads fail the load
instead of rendering phantom rows; toggle error copy is scoped to
GatewayApiError with a localized fallback for anything else.
- The localized account_disabled copy is login-only (zhfeng): sign-up
keeps the backend message verbatim so a suspended account's email
cannot be probed through the register surface.
- Settings page folds the load banner and toggle error into a single
alert region (zhfeng); adds coded-envelope 409, malformed-row, and
load-failure tests; fetcher.ts documents the 401 body-consumption
contract.
- prettier across touched files (CI format gate).
The admin-users query key keeps user?.id, matching the
model-settings-page precedent (converge later per review).
* test(local-dev): normalize the upgrade script's MSYS path rendering on Windows
nginx-local-conf.sh computes its repo root with pwd -P, which inside
Git Bash yields the MSYS mount form (/tmp/... for the Windows temp dir)
— correct for its bash-native consumers (serve.sh, nginx.sh) but never
equal to the test's Windows-form Path.resolve(). Map the rendering back
through cygpath -w before comparing, so the tests read and compare the
rendered config on Git Bash hosts too.
Fixes#6657
* test(local-dev): harden Git Bash path normalization
---------
Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(tui): keep every word of an unquoted --print/--json message
--print and --json took a single optional value and never read the
positional words, so `deerflow --print summarize this repo` sent only
"summarize", `deerflow summarize this repo --print` exited 2 asking for a
message on a terminal, and with piped stdin the typed words were ignored.
Both flags now take every following word and fall back to words given
before them, as --cli does; a typed message wins over piped stdin. A
message split around the flag is refused with exit 2, because argparse
loses the word order between the two parts.
* docs(changelog): reference #6636
* fix(tui): refuse --print and --json together
Only one headless mode runs, so the other flag's words were dropped:
'--print hello --json world' ran with 'hello', and '--print --json hello'
asked for a message. Put both flags in a mutually exclusive group so argparse
exits 2 for every combination.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): add You.com web search provider
Adds You.com as an opt-in community `web_search` engine, mirroring the
existing community engines (tavily, brave, exa, firecrawl, groundroute).
You.com runs its own web index and returns extracted page snippets per
hit rather than a single meta description, so a research loop gets usable
context without a follow-up fetch.
The provider picks its endpoint by whether a key is configured, so it
works out of the box and scales up when the operator adds one:
- no key -> /v1/agents/search, keyless free tier
- YDC_API_KEY -> /v1/search with X-API-Key, full Search API
Endpoint and headers are always chosen together; the keyless endpoint
rejects an auth header, so a stale key can never be sent there. Both
return the same body shape, so one normalizer covers both.
httpx only, no new dependency and no pyproject extra, so uv.lock is
untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(community/youcom): review follow-ups — drop stray .pr-body.md, guard OverflowError, add 429 hint
* fix(doctor): make the You.com free-provider detail true with or without YDC_API_KEY
The doctor detail returned before any key check, so operators who had
set YDC_API_KEY still saw 'set YDC_API_KEY for the full Search API'.
Reworded to match the sibling free-provider entries and stay accurate in
both states.
* fix(search): sweep youcom max_results onto the shared owner
The single-owner gate that landed with #6563 flags this branch's private
_coerce_max_results as a newly hand-rolled copy, so backend-unit-tests
now fails after merging main. Follow the coordination note from review:
delegate to deerflow.community.search_max_results.coerce_max_results,
which owns the rejection profile (booleans, non-integral floats,
OverflowError), and keep only the 1-100 mirror of You.com's server-side
count clamp on top — clamping policy stays per-provider.
The .inf regression test now goes through the public tool path and also
pins the bool and non-integral-float shapes of the shared bar.
* docs(tools): use the bare you.com platform link in config and docs
Match every other provider entry: no other provider in config.example.yaml
or the en/zh tools.mdx uses campaign-tagged URLs, so drop the utm params
from the three you.com/platform links. Integration attribution stays in
the identifying User-Agent sent to You.com hosts.
* test(youcom): cover shared max_results ownership and bounds
Register You.com among the shared-owner providers and assert default handling and the 100-result cap through outbound request parameters.
---------
Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: valueadd-lgtm <339269770+valueadd-lgtm@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): preserve server binding history across gateway workers
A peer Gateway worker could keep a stale pooled session after another worker deleted and identically re-added a stdio server, because the final effective MCP slice matched the revision it had already applied.
Record a per-server lifecycle generation in the shared extensions config and consume it in the reader, discovery claim, and committed handoff so an observed generation change force-rebinds only the affected server. Legacy configs derive a stable in-memory token; deletions leave permanent tombstones.
* fix(mcp): fail closed on an invalid mcpLifecycle instead of retrying
An unverifiable mcpLifecycle made initialize_mcp_tools() unpublishable, so get_cached_mcp_tools() retried discovery forever and could keep respawning stdio servers. Abort discovery on an invalid ledger after conservatively retiring local MCP state; a fixed config recovers on the next call.
Also documents the rolling-upgrade prerequisite: cross-worker ABA detection requires configuration writes to go through a lifecycle-aware writer.
* refactor(mcp): share stdio fingerprinting across lifecycle paths
Reuse lifecycle._stdio_fingerprints from the applied-revision builder so the connection-identity rule has a single implementation, and move the unusable-server diagnostic into the shared helper.
Add a regression proving the Gateway Skills writer preserves mcpLifecycle byte-for-value, and document that DeerFlowClient.update_mcp_config can raise RuntimeError after the config was committed.
* docs(mcp): condense lifecycle guidance under the AGENTS chain limit
The effective mcp/AGENTS.md chain was 15 bytes under the 98,304-byte CI hard limit, so the standalone cross-worker lifecycle bullet pushed it over. Fold the mcpLifecycle and rolling-upgrade facts into the existing selective-reconciliation bullet and condense it, keeping the chain under the limit while preserving the contract.
* docs: trim inherited agent guidance below CI size limit
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scripts): reject local BIND_HOST values nginx would misread
nginx-local-conf.sh checked only for address and hostname characters.
BIND_HOST=0 rendered `listen 0:2026;`, which nginx resolves to 0.0.0.0 and
binds on every IPv4 interface, and 1.2.3 bound 1.2.0.3. Values nginx rejects
outright ([[fd00::1]], -foo) passed the serve.sh preflight and failed only
after stop_all had torn down the running stack.
Accept exactly a dotted-quad IPv4 address, an IPv6 address (brackets
allowed around IPv6 only), or an RFC 1123 hostname. Returning to the
loopback default removes a temp/nginx.local.conf rendered for an earlier
value, and stop_all now says why it can quit a rendered-config nginx
through the tracked config; a test pins the shared pid directive.
* docs(changelog): reference #6635
* docs(changelog): match the zh reference spacing
* fix(scripts): check every BIND_HOST hostname character before splitting labels
read -r -a stops at the first newline, so is_hostname validated only the
first line of a value like $'localhost\nbad' and the rest reached the
rendered listen directive. Check the allowed characters across the whole
value first, and cover multi-line, CR and tab values for each validator.
* fix(lark): publish the CLI runtime files atomically
install_shim wrote the shim, the launcher and the runtime marker with a
truncating open. The sandbox chmods the launcher 0o755 and execs it, so a crash
mid-write left an executable fragment, and the marker is parsed as JSON. Publish
through a temporary file plus os.replace, as the storage backends already do.
* style: drop the unused import ruff flags
* fix(lark): publish runtime permissions atomically
Set file permissions on temporary files before os.replace() to ensure
the launcher, shim, and runtime marker are atomically published with
correct permissions (0o755 for executables, 0o644 for the marker).
This addresses reviewer feedback from PR #6591:
- Permissions are now set before the atomic replace operation
- Prevents brief window where files have incorrect permissions
- The runtime marker now has mode 0o644 instead of 0600 (mkstemp default)
Also updates test_lark_shim_atomic.py:
- Fixes test_atomic_helper to match new function signature
- Adds test_install_shim_publishes_correct_permissions to verify
launcher/shim (0o755) and marker (0o644) permissions
Co-authored-by: IronMurphy <hepushuo@gmail.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
`workspace_changes/api.py` imports `deerflow.runtime.user_context` for the
AUTO sentinel (#6282). Importing that submodule runs the runtime package init,
which imports `.runs` -> `worker` -> `deerflow.workspace_changes` back for
`capture_workspace_snapshot` and friends, so importing `.api` as the package's
first statement failed with ImportError while those names were still unbound.
Resolve the re-export lazily via a module-level `__getattr__`, the pattern
`deerflow/subagents/__init__.py` already uses.
* feat(dev): add a two-Gateway multi-instance harness script
There was no way to run the deployment.multi_instance topology locally:
scripts/serve.sh pins one Gateway to port 8001, the local nginx config has a
single upstream, and getting two processes past the multi-instance startup gate
needs Postgres, Redis, a shared DEER_FLOW_HOME, shared auth secrets and a
per-process DeerMem retrieval index by hand.
scripts/dev_multi_instance.sh (make dev-multi / dev-multi-check /
dev-multi-down) starts throwaway Postgres and Redis containers on loopback,
renders a config from the developer's config.yaml (or config.example.yaml),
and runs Gateway A on 8001 and Gateway B on 8011 with one shared home, one
copy of extensions_config.json and generated AUTH_JWT_SECRET,
DEER_FLOW_INTERNAL_AUTH_TOKEN and DEER_FLOW_CREDENTIALS_KEY values, plus an
optional round-robin nginx on 2027. B starts after A answers /health: Alembic
migrations are advisory-locked, but the LangGraph checkpointer/store setup()
that follows is not.
scripts/dev_multi_instance.py holds the testable parts: the config overlay
(neutralizing and reporting settings the gate refuses), the secrets file, the
nginx config, and `check`, which automates the scriptable cross-instance
acceptance checks: readiness with stream_bridge ok, A-issued sessions and the
internal token accepted by B, thread/upload/artifact visibility, a skill
toggle seen by B, nginx balancing, and SSE join on B with Last-Event-ID resume
on A. Tests pin that the rendered config passes the real
_enforce_postgres_for_multi_worker gate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link the two-Gateway harness entries to #6613
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(dev): keep the multi-instance harness inside its own state
Address review findings on the two-Gateway harness (#6613):
- Containers carry a deerflow.harness.state-dir label; up/down remove only
containers of their own state dir and refuse those of another checkout or
DEERFLOW_MI_STATE_DIR instead of force-removing them.
- Containers are removed with `docker rm -f -v`, so the anonymous volumes
the postgres and redis images declare no longer leak on every up/down.
- `up` installs an EXIT trap once it starts containers, so any later failure
(including B's port being taken after preflight) tears down A and both
containers; start_gateway returns failures instead of exiting the script,
and the Gateway is disowned so stopping it prints no job notice.
- The overlay removes any checkpointer section (a postgres one would write
checkpoints and the Store into the developer's database), points an
explicit sandbox.ownership and a Redis checkpoint_cache at the harness
Redis (keeping ownership timing), drops DeerMem storage_path (both
spellings) and the blob_storage root so data lands in the harness home,
gives a local AIO sandbox a state-dir-specific container_prefix, and
reports a non-DeerMem memory backend it keeps sharing.
- nginx paths are rendered as quoted string literals ('$' and control
characters are refused), so a state dir with spaces works.
- `check` skips the run-based SSE check when model discovery did not
succeed, unless --with-llm, and enforces the SSE time budget per raw line
so heartbeat-only traffic cannot hold it open.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(scripts): keep the harness guidance under the AGENTS.md soft limit
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat: return history search excerpts near keyword matches
* docs: use English for history search guidance and comments
* docs: sync Chinese history search excerpt contract
* docs: clarify history excerpt fallback for oversized matches
* fix(history): skip oversized occurrences when selecting excerpts
Continue scanning casefolded substring matches until an occurrence fits the 600-character excerpt, including overlaps. Cover later fitting hits through synchronous and asynchronous active/archive search and read tools, and update the documented contract.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(agents): sanitize persisted URL content blocks before model calls
Threads whose checkpoints carry URL-sourced file blocks (written before
the MCP conversion-layer fix) fail every model call at serialization: the
Chat Completions translator rejects any file block carrying a url, and the
error handler answers each later turn with a non-retriable fallback. Add a
read-time sanitizer that rewrites those blocks in the model-bound request
view only, so poisoned threads heal on their next turn with no migration
and no user action.
- ModelContentCompatibilityMiddleware wraps the model call (sync and
async) and rewrites ToolMessage list content: URL-sourced file blocks
become "[Resource ({mime or unknown type}) available at {url}]" text
blocks, and image blocks whose scheme is not http/https/data get the
same placeholder. Fetchable image URLs, base64 payloads, text, and
non-ToolMessage messages pass through. The rewrite is order-preserving,
idempotent, and never writes state or mutates checkpoints, so artifact
capture and other state readers keep seeing the original blocks.
- Register on all three chains that replay checkpointed ToolMessages:
build_middlewares (lead; outer of the MODEL_PHYSICAL extension anchor so
its final-request guarantee holds), the SDK factory
_assemble_from_features (new step 13a; docstring updated and its stale
omission of TokenBudgetMiddleware fixed), and
build_subagent_runtime_middlewares (after DurableContext/summarization,
before the date/coalescer tail).
- Subagent coverage verified by tracing every subagent agent-creation
site: executor.py _create_agent is the only caller of
build_subagent_runtime_middlewares and serves both the task tool and the
durable batch worker (SubagentRuntime itself only carries dependencies),
and a wiring test now asserts the middleware is present in that
builder's output; the lead and factory chains gained the same wiring
assertions.
- Update the chain-order contract tests for the new slot: full-chain
order, loop-detection adjacency, custom loop-detection tail positions,
and the subagent chain count (22 -> 23).
* fix(mcp): stop emitting URL-sourced file blocks from ResourceLink results
MCP ResourceLink conversion produced a URL-sourced file content block for
every non-image link, which LangChain's Chat Completions translator rejects
at serialization time. Once such a block was checkpointed in the
ToolMessage, every subsequent turn of the thread failed at model-call time,
permanently bricking the conversation.
- Downgrade every ResourceLink that is not an http(s) image to a short text
placeholder carrying the resource name, MIME type, and the
already-virtualized URL, so host paths never leak and the model still
learns a resource was produced and where it lives.
- Preserve each downgraded link as structured data in the tool-message
artifact under resource_links (merged with structured_content when
present) for artifact capture and future MCP Apps rendering.
- Keep http(s) image links as image blocks; embedded base64 payloads, text
path rewriting, and the error-to-ToolException path are unchanged.
- Re-contract the file-migration tests to the new behavior and document
the conversion contract in the MCP AGENTS.md.
* docs(middlewares): add ModelContentCompatibilityMiddleware to the chain inventory
Insert the read-time sanitizer at its actual lead-chain position (after the
token-budget guard, before the custom-middleware injection point) and renumber
the tail entries 34-39 to 35-40.
* fix(agents): rewrite file blocks on url key presence, not truthiness
langchain-core's OpenAI translator raises for ANY file block carrying a
url key — an empty string or None included — so the read-time sanitizer
must not let such blocks through just because the value is falsy. A
persisted block with url="" or url=None would still brick every later
model call in the thread.
- Gate the file-block rewrite on "url" in block instead of a truthiness
check; a non-empty string URL keeps the
"[Resource (mime) available at url]" placeholder, while an empty or
None URL gets a location-less "[Resource (mime)]" placeholder, so no
unusable location is advertised to the model.
- Pin both shapes with new middleware tests (url="" and url=None file
blocks are both rewritten to text). The image branch is unchanged: an
empty-image case fails provider-side with a 400, not a bricked thread.
* fix(mcp): keep data URIs out of model text and resource_links
A data: scheme ResourceLink fell into the generic placeholder branch,
inlining megabytes of base64 into the model-visible text and into
resource_links, which is checkpointed state — against the repo policy
that data:/blob: URIs never enter state.
- Special-case data: links in _convert_call_tool_result: the
model-visible placeholder becomes "[Resource: name (mime) embedded
inline]" and the raw URI is not added to resource_links at all, so the
payload can neither bloat the context window nor persist in the
checkpoint.
- Pin the behavior with a conversion test asserting both the placeholder
wording and the absence of the payload/URI from every emitted block and
the artifact.
- Add the missing passthrough pin for an HTTPS:// uppercase-scheme image
link: scheme matching is case-insensitive, so the link stays an image
block (the block's URL validation normalizes the scheme to lowercase).
* fix(tools): consume resource_links in extract_artifacts_from_result
The MCP conversion layer preserves downgraded ResourceLinks as
{"name", "uri", "mime_type"} entries under the tool message
artifact's resource_links key, but the artifact extractor only read
structured_content — a result whose only artifact signal was
resource_links produced no entries and the file silently vanished from
the artifacts panel.
- Add a fourth extraction path: each resource_links entry becomes a file
artifact entry, reusing _is_referenceable_url like the
structured-content and content-block paths, so data:/blob: and other
unresolvable URIs stay rejected per existing policy. The entry is named
after the link when a name is present, falling back to the URI's final
segment; a string mime_type is carried through.
- Integration-test a result whose only signal is resource_links, plus the
referenceability gate (data:, ui://, and absolute-path URIs in one
artifact).
* docs(mcp): update the ResourceLink conversion contract bullet
Mention the new artifact-registry consumer and the data: special case so
the bullet stays accurate: uppercase http(s) schemes match
case-insensitively, data: URIs get the embedded-inline placeholder and
are dropped from resource_links, and extract_artifacts_from_result turns
resource_links into registry entries through the shared
referenceability gate.
* refactor(tools): add the canonical resource placeholder formatter
Both the MCP conversion layer and the read-time
ModelContentCompatibilityMiddleware downgrade model-incompatible resource
references to a text placeholder, but each built its own literal. Extract
the single formatter they will share so a thread healed at read time shows
the model the same placeholder shape a fresh conversion produces; the name
segment is optional because persisted content blocks carry no resource name.
* fix(mcp): keep host paths out of ResourceLink placeholder text
Review follow-up on the ResourceLink conversion contract. The placeholder
embedded the resolved URL unconditionally, so a link outside the user-data
tree (or converted without thread context) leaked a raw host path — a
file:// URI or bare filesystem path — into model-visible text, violating
US-16. Compute the resolved URL's scheme and withhold the location segment
when it spells out a host filesystem location (empty scheme, file, or a
single-letter Windows drive scheme); virtual paths and remote schemes keep
their location. The unresolved URI still lands in the artifact channel,
which the model never sees.
- add _model_visible_link_location gate and route the generic placeholder
through it, re-contracting the outside-user-data and no-context tests to
the location-less shape and pinning the Windows-drive case
- treat blob: like data: as non-referenceable inline input: drop its raw
URI from checkpointed resource_links and emit the location-less
placeholder instead of the mismatched 'embedded inline' wording, and fix
the comment that claimed blob: was already excluded
- replace every inline placeholder f-string in the ResourceLink branch
with the shared resource_placeholder_text formatter so the conversion
layer and the read-time middleware emit byte-identical shapes; data:
links now render in the canonical location-less form
- sync the ResourceLink conversion contract bullet in mcp/AGENTS.md and
the _convert_call_tool_result docstring with the host-path suppression
rule and blob: handling
* fix(agents): sanitize URL content blocks on any message role
Address the second-pass review of the request sanitizer with two
changes that share the same code path.
- Adopt the canonical resource_placeholder_text formatter from
deerflow.tools.resource_placeholder instead of the module-local
placeholder helpers, so the read-time rewrite and the MCP conversion
layer emit byte-identical placeholder text from one definition.
- Widen the rewrite from ToolMessage-only to every request message with
list-form content. User uploads are verified to produce base64 blocks,
but imported or cross-client history can carry a URL-sourced file
block on a HumanMessage or AIMessage, and the Chat Completions
serializer rejects such a block regardless of the role carrying it.
The rewrite stays request-view-only: state and checkpoints are never
mutated, and base64/file-id blocks and string content pass through
untouched on any role.
* docs(agents): state the ModelContentCompatibility placement contract
The comment above the registration claimed the sanitizer sits just
before Clarification. In the lead chain it is actually followed by the
custom/extension injection point and the whole terminal tail
(terminal-response, finish-reason, and the always-last
ClarificationMiddleware). Describe the real contract: after every
built-in content transform, before extension injection and the
terminal tail.
* docs(middlewares): clarify provenance stamps for request-view-only rewrites
The message-provenance rule read as if every rewrite needs a
provenance_kwargs stamp. Add one line noting the exception: rewrites
that only modify the model-bound request view and never enter state
need no stamps, citing InputSanitizationMiddleware,
DanglingToolCallMiddleware, and ModelContentCompatibilityMiddleware as
precedents.
* fix(agents): keep host paths and data URIs out of healed placeholders
The read-time ModelContentCompatibilityMiddleware rewrote URL-sourced
file blocks and non-fetchable image blocks into placeholders that
inlined the block's URL verbatim. Pre-fix checkpoints — exactly the
threads this middleware exists to heal — can carry a file:// URI, a bare
or Windows host path (links that never resolved to virtual paths), or a
data: URI embedding megabytes of base64, so healing leaked deployment
filesystem layout into prompts (US-16) and could inject the whole inline
payload into model-visible text.
The conversion layer already suppresses those forms via
_model_visible_link_location. Move that gate into the shared
deerflow.tools.resource_placeholder module as model_visible_location
(now also collapsing data:/blob: URIs) and route both middleware
branches through it, so both layers apply identical suppression rules
with one definition. Conversion behavior is unchanged: its data:/blob:
branch short-circuits before the gate, and the gate covers those schemes
for defense in depth.
- resource_placeholder.py gains model_visible_location alongside the
shared formatter; mcp/tools.py drops its private copy and imports it.
- The middleware's file and image branches pass the URL through the gate
before building the placeholder; module docstring documents the rules.
- Tests: gate unit tests (virtual/remote visible; file://, bare/Windows
host paths, data:/blob: withheld) and middleware regression tests
pinning that a pre-fix data: payload, file:// URI, and Windows drive
path heal to location-less placeholders.
- AGENTS.md bullets in mcp/ and middlewares/ state the shared gate.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(agents): downgrade malformed image URLs instead of raising in the sanitizer
The read-time ModelContentCompatibilityMiddleware gated persisted image
blocks on urlparse(url).scheme, but urlparse raises ValueError on
structurally invalid URLs — a bad IPv6 literal, invalid netloc
characters — instead of returning an unusable scheme. Pre-fix
checkpoints and imported history can carry exactly those URLs, and the
block stays in the checkpoint, so every later model call in the thread
raised again: the sanitizer re-bricked the very threads it exists to
heal, only with a framework-side exception instead of the langchain
serialization error.
Catch ValueError and treat the URL as non-fetchable, so the block
degrades into the same text placeholder as any other unresolvable
scheme (the shared model_visible_location gate already swallows
ValueError, so the location segment is withheld as well).
- wrap the image branch's urlparse call in try/except ValueError and
fall back to an empty scheme, keeping http/https/data pass-through
behavior unchanged for well-formed URLs
- add a regression test pinning that an image block with a bad IPv6
literal heals to the location-less placeholder without raising
Co-Authored-By: Codex <noreply@openai.com>
* fix(tools): dedupe artifact refs across extraction sources
One downgraded ResourceLink produced two registry entries: the typed
resource_links channel added an entry for the URI, and the free-text
scan over the conversion layer's placeholder text — which embeds the
same virtual path or remote URL — matched it again. The two handles
pointed at one resource, spent double of the render budget, and showed
the model two handles for a single artifact.
extract_artifacts_from_result now tracks seen real_refs across all
sources and keeps only the first entry per ref. Extraction order makes
the typed sources (structured content, resource_links, content blocks)
win over the free-text scan, so the surviving entry keeps the link's
display name and mime type.
- route all six entry-producing sites through a local _add_unique
helper keyed on real_ref; per-source behavior is unchanged
- document the cross-source dedup in the function docstring
- add a regression test covering a virtual-path and a remote-URL
placeholder together with their resource_links entries
Co-Authored-By: Codex <noreply@openai.com>
* docs(mcp): state the intentional data: image downgrade asymmetry
The conversion layer downgrades image/* data: ResourceLinks to
location-less text placeholders, while the read-time middleware lets
persisted data: image blocks pass through to the model. The two
contracts looked contradictory but are deliberately layered: the
conversion layer keeps new inline payloads out of checkpointed state,
and the middleware only preserves blocks already occupying one.
- mcp/AGENTS.md's ResourceLink conversion contract and the conversion
comment in mcp/tools.py now state the image/* data: downgrade and
why it differs from the read-time pass-through
- middlewares/AGENTS.md's ModelContentCompatibilityMiddleware entry
states the symmetric read-time rule
- pin the behavior with an image-mime data: ResourceLink conversion
test (location-less placeholder, nothing in the artifact channel)
Co-Authored-By: Codex <noreply@openai.com>
* docs(backend): compact AGENTS.md guidance to fit the chain byte budget
The branch's new contract bullets pushed the middlewares and mcp
effective AGENTS.md chains past the 98304-byte AG002 hard limit (the
sandbox/subagents chains on main were already over it). Compact the
guidance without dropping contracts:
- middlewares item 34 points at the MCP ResourceLink conversion
contract and the module docstring instead of restating both
- the mcp ResourceLink conversion contract bullet is tightened
- backend/AGENTS.md merges the File Upload and Context Summarization
bullet sprawl, folds the stray ATX test note into the upload outline
rule, drops the Nginx routing block the root guide already covers,
merges the duplicate MCP-task pointer bullet, compacts Vision
Support, and scopes the Project Structure tree to the backend
subtree (the root guide maps the full repo)
Co-Authored-By: Codex <noreply@openai.com>
* fix(agents): gate the file-URL downgrade on provider support and neutralize placeholder metadata
Two review findings on the read-time sanitizer:
- The file-block downgrade ran for every provider, but only Chat
Completions rejects URL-sourced file blocks: the OpenAI Responses API
(input_file file_url) and Anthropic (document url source) accept
http(s) document URLs, so a persisted block created for those
providers lost its document from the model request. Keep the block
when the selected model serializes http(s) file URLs natively; every
other URL shape (non-http schemes, empty/None, malformed) still
degrades to the placeholder.
- The placeholder interpolated persisted MIME/location metadata
verbatim, but the middleware runs after the input/tool-result
sanitization passes and historical URL blocks never went through them
(non-text content is left untouched), so values carrying
<system-reminder>-style tags or user-input boundary markers reached
the model raw. Route the interpolated metadata through the shared
neutralize_untrusted_tags guard in both the file and image branches.
Tests: provider gating (Responses/Anthropic preserve, Chat Completions
still downgrades, non-http and malformed URLs downgrade even for
capable providers) and metadata neutralization (injection tags
HTML-escaped, boundary tokens neutralized, file and image branches).
Co-Authored-By: Codex <noreply@openai.com>
* fix(tools): require a directory boundary before exposing virtual locations
model_visible_location admitted any URL starting with the virtual
prefix, so a sibling like /mnt/user-data-backups/ops/private/report.pdf
was treated as a virtual path and its host location reached
model-visible placeholder text — defeating the US-16 host-path
suppression the gate exists for. Require the exact virtual root or a
slash-delimited descendant; siblings now collapse to the location-less
placeholder like any other host path. Both the file and image
downgrade branches share the gate, and the conversion layer applies
the same rules to fresh results.
Regressions: gate unit tests (exact root and descendants stay visible;
/mnt/user-data-backups and /mnt/user-database siblings withheld) and
middleware tests pinning the location-less placeholder for both
branches.
Co-Authored-By: Codex <noreply@openai.com>
* docs(agents): scope the placeholder artifact-visibility claim to the conversion layer
The module docstring claimed the healed placeholder keeps the location
visible to the artifact free-text scan and that the link survives in
the structured artifact channel — true for the conversion layer, whose
placeholders are checkpointed, but not on the healing path: the request
view never reaches artifact capture, and no resource_links entry is
created for a downgraded block. Spell out what actually happens here:
a referenceable original block (http(s) URL or absolute host path) is
still captured, from the capture-time scan of the state block rather
than the placeholder, while a non-referenceable one (file://, data:,
blob:) yields no entry and survives only in the untouched checkpoint.
Co-Authored-By: Codex <noreply@openai.com>
* docs(mcp): clarify healing and artifact preservation contracts
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* test: route four unguarded symlink fixtures through symlink_or_skip
`tests/support/symlinks.py` was added in #5414 so suites that need a real
symlink skip cleanly when the host cannot create one. On a stock Windows host
(no Developer Mode, not elevated) `Path.symlink_to` / `os.symlink` raise
`OSError: [WinError 1314]`, and four fixtures still called them directly, so
they reported hard errors instead of skipping:
- tests/test_shared_reset_marker.py::test_marker_path_follows_a_symlinked_config
- tests/test_view_image_middleware.py::test_host_image_copy_is_scoped_to_current_user_and_thread
- tests/test_review_skill_package_tool.py::test_review_skill_package_rejects_symlink_planted_in_callers_outputs
- tests/test_channel_file_attachments.py::TestInboundFileSandboxPerms::test_make_inbound_file_sandbox_readable_swap_after_lstat_does_not_follow_symlink
These were the only symlink-related failures in a Windows run of the full
backend suite; the other bare symlink call sites there are already guarded by a
`skipif` or a `try/except OSError`. The view-image test plants its symlink
after its scoping assertions, so those still run before the skip. The
inbound-attachment test stages a symlink from inside a monkeypatched `os.lstat`
to exercise a TOCTOU race; a host that cannot create a symlink cannot stage
that race at all, so skipping is the only honest outcome.
* test: make channel symlink race fixture portable
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(security): add DEER_FLOW_CREDENTIALS_KEY for at-rest credential encryption
There was no at-rest encryption key. ChannelCredentialCipher had no
production caller: the channel service, the scheduled-task notification
outbox and the channel-connections router all built
ChannelConnectionRepository without a cipher, so store_credentials refused
(RuntimeError) and get_credentials always returned None.
- New harness module deerflow.config.credentials_key: env-only
DEER_FLOW_CREDENTIALS_KEY (comma-separated Fernet keys; first encrypts,
all decrypt via MultiFernet), CredentialsCipher with encrypt_text /
decrypt_text / rotate_text, a fernet:v2: prefix with legacy fernet:v1:
values still readable, and get_credentials_cipher() as the process-wide
entry point. Malformed keys are refused without echoing key material.
Without the variable a single instance generates
{base_dir}/.credentials_key. Never derived from AUTH_JWT_SECRET.
- New deerflow.config.secret_file.read_or_create_secret_file: O_EXCL
create (0600) plus read-back with a bounded settle window, shared by
.credentials_key, .jwt_secret (fixes two replicas cold-starting on a
shared volume keeping different JWT secrets) and the managed-model key.
- ChannelCredentialCipher now builds on CredentialsCipher (same
constructors); the Gateway loads the cipher off the event loop at
startup when channel_connections is enabled and passes it to every
ChannelConnectionRepository. Undecryptable values stay "missing + warn".
- Startup gate: a malformed key stops startup in any topology; a declared
multi-instance deployment with channel_connections enabled and no key
is refused. Plain multi-worker shares the generated file and is allowed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(deploy): generate and inject DEER_FLOW_CREDENTIALS_KEY in Helm, compose and deploy.sh
- Helm: the app Secret generates a Fernet key (randBytes 32 translated to
the urlsafe alphabet) and preserves it across upgrades via lookup like
AUTH_JWT_SECRET; the gateway reads it through an optional secretKeyRef
so a user-managed existingAppSecret without it still boots (the Gateway
gate refuses only multi-instance deployments that store credentials).
README, NOTES and values document backup, rotation and existingAppSecret.
- scripts/deploy.sh resolves the key shell -> .env -> persisted file ->
generated, persisting it as $DEER_FLOW_HOME/.credentials_key (0600,
noclobber/O_EXCL with read-back): the same file the Gateway generates in
its runtime home, so either side's key is the one both keep.
- Both compose files pass DEER_FLOW_CREDENTIALS_KEY=${...:-} to the gateway.
- .env.example entry with a generation hint; README and CHANGELOG notes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): make abandoned secret-file replacement single-winner
A rename is last-writer-wins, so replicas that restart together and all
find the same abandoned empty secret file could each replace it and keep
a different value. The process that exclusively creates
<name>.replacing now owns the replacement, re-checks the file is still
empty, and publishes by renaming the claim over it; peers wait for that
value, bounded so a crashed replacer cannot block startup forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link DEER_FLOW_CREDENTIALS_KEY entries to #6611
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): name the setting that turns off a credentials-key consumer
The multi-instance refusal ended with "or disable
channel_connections.enabled=true"; it now says "or set
channel_connections.enabled=false".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): never expose a secret file before its value is written
O_EXCL published the name before its content, so a creator that stalled
mid-write left an empty file a peer replaced after the settle window; the
resumed creator then returned a value nobody persisted, or deleted the
peer's valid file when its own write failed. Creators now write and sync
a private temporary file and hard-link it to the name, which fails if
the name exists; a failed creator only removes its own temporary file.
Filesystems without hard links publish through the same single-winner
<name>.replacing claim that replaces abandoned empty files, so creation
and replacement are serialized there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(models): recover an empty managed-model key when no catalog exists
An interrupted first save could leave an empty key file; every later save
then passed empty bytes to Fernet and the endpoint returned 503 until the
file was removed by hand. An empty key with no catalog encrypted nothing,
so it now goes through the secret-file recovery path; next to an existing
catalog it is still refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(docker): pass the dotenv DEER_FLOW_CREDENTIALS_KEY through make docker-start
scripts/docker.sh runs Compose from docker/ without --env-file, so the dev
gateway's DEER_FLOW_CREDENTIALS_KEY=${DEER_FLOW_CREDENTIALS_KEY:-} entry
interpolated to empty and overrode the key the checkout .env supplied via
env_file. Export it from the dotenv like AUTH_TRUSTED_PROXIES.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test: fix two Windows-host failures (subprocess encoding, budget-boundary epsilon)
- test_env_file: pin PYTHONIOENCODING=utf-8 in the stripped child env;
without it the child writes stderr in the host's ANSI codepage while
the UTF-8-mode parent decodes it, the reader thread dies on the first
non-ASCII byte, and stderr comes back None;
- test_jina_retries: give the budget-cap assertion a 1e-9 tolerance —
deadline = now + budget rounds, so the cap lands a few ulps above the
exact 0.05 (observed 0.05000000000291038).
Fixes#6626
* test: pin both sides of the capture and use an uptime-proof budget tolerance
- test_env_file: the parent's capture decodes with the host locale when
pytest runs without the Makefile's UTF-8 pins, so the child-side pin
alone mirrored the crash rather than eliminating it. Pin the capture
explicitly (encoding=utf-8, errors=replace, per the
test_local_sandbox_encoding.py precedent) and carry both suite pins in
the child env (PYTHONUTF8=1 + PYTHONIOENCODING=utf-8).
- test_jina_retries: the float error scales with loop-time magnitude
(host uptime), not with 0.05; a 1e-9 absolute slack leaves a ~389-day-
uptime flake window. Use a relative tolerance instead.
---------
Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com>
