* fix(scheduler): preserve unchanged once schedules near execution
* docs: keep schedule guidance within the size limit
* style(scheduler): use English for the once-edit comment
* docs(scheduler): keep guidance within directory budgets
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(frontend): return to the opened workspace page after signing in
Opening a workspace page without a valid session sent the user to a bare
/login: the workspace layout called redirect("/login"), and a layout
cannot read the request URL, so the path and query were both lost. A
scheduled-task notification link (/workspace/scheduled-tasks?task_id=X)
opened after the session expired therefore landed on the default
workspace after sign-in.
For an unauthenticated verdict the layout now renders
WorkspaceLoginRedirect, which replaces the page with
/login?next=<path+query> from the browser. It renders nothing else, so
the workspace pages under it never mount.
AuthProvider.refreshUser and useRequireAuth kept the path but dropped
the query. They now share currentReturnPath() with the fetcher, which
already kept both.
* docs(changelog): reference #6681
* fix(frontend): keep the fragment in the sign-in return path
currentReturnPath() kept the path and query but not location.hash, so
an anchored deep link (/workspace/chats/x#section) lost its anchor across
the login round trip. It is now the single source for all four bounce
sites, so include the fragment. The server never receives the fragment,
so only the browser-side bounce can keep it. validateAuthNextPath, the
backend's validate_next_param and its quote() into /auth/callback all
carry a '#' through unchanged.
The models API test stubbed window.location without hash; a real
Location always has one, so the stub now carries it.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(acp): resolve an absolute Windows agent command before spawning
A command configured as an absolute path without an extension -- npm's global
shim layout, e.g. C:\npm\mcode beside mcode.cmd -- was handed to the ACP SDK
unchanged and always failed: create_subprocess_exec applies no PATHEXT, so it
ran npm's extensionless shell script and died with WinError 193 (or raised
FileNotFoundError when only the .cmd exists). The separator guard from #6598
was too broad: only relative paths need to stay with the spawn, because their
meaning is owned by the ACP-workspace cwd, while an absolute path does not
depend on the cwd.
Absolute paths now go through the same shutil.which lookup as bare names: the
lookup searches the command's own directory and probes PATHEXT, and the
configured value is kept when nothing matches so the not-found remediation
still fires. Relative paths (bin/agent, ./bin/agent) still go to the spawn
unchanged.
Docs: README.md and backend/packages/harness/deerflow/tools/AGENTS.md now
describe the bare/absolute vs relative split.
AI disclosure: this change was drafted with AI assistance at my request, under
my direction and review.
* fix(acp): keep already-absolute lookup results unchanged
Review feedback on #6656 (willem-bd): os.path.abspath collapsed a `..` that
follows a symlink, so an absolute POSIX command like
`/srv/current/../shared/agent` (with `current -> releases/v1`) was rewritten to
`/srv/shared/agent` -- a path that either does not exist or names a different
launcher. The anchoring to the Gateway cwd exists only for relative PATH
lookup results, where shutil.which cannot know the cwd the spawn will use; an
already-absolute result needs no rewrite.
Regressions: a platform-independent stub-lookup test pinning the literal
parent-segment path, and a POSIX-only test that builds the symlink layout and
asserts the configured command is returned unchanged.
* fix(runtime): mark cancel takeovers with the orphan-recovery stop reason
A cross-worker cancel that takes over an expired-lease run writes error but
left stop_reason unset. `orphan_recovered` is the only signal that lets a
store-only SSE/`wait` consumer synthesize END after a heartbeat, because the
producer is known to be gone. Without it, a subscriber that outlives the
crashed owner waits forever for an END no surviving worker publishes; periodic
reconciliation ignores the row because it is already terminal.
Mark the takeover like periodic orphan recovery: claim_for_takeover already
accepts the stop reason and the reconcile path already passes it. The new
tests pin the takeover row's marker and prove a real takeover wakes a waiting
consumer; three existing takeover test doubles now forward the store argument.
* docs(gateway): keep guidance within the 48KB budget
The takeover stop-reason sentence pushed backend/app/gateway/AGENTS.md over its 48KB hard limit, failing agent-guidance and the backend guidance-budget test. Trim the sentence; the rationale stays in the manager.py comment.
* fix(gateway): complete streams after scheduler orphan recovery
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): stop resolving sandbox and ACP env values a second time
AppConfig.resolve_env_variables already replaces every $VAR when
config.yaml loads. The AIO, E2B and OpenSandbox providers and the ACP
tool then resolved the result again, so a secret that itself starts with
$ (a bcrypt hash, a generated password) was looked up as a variable name
and reached the sandbox or agent as "" -- or as an unrelated host
variable's value when that name existed.
Pass the loaded values through unchanged, as BoxLite and the local
provider already do. The removed helpers' str() coercion was dead: both
fields are typed dict[str, str] and reject non-strings at load.
* docs(changelog): reference #6673
* docs(config): pin when sandbox and ACP env values are resolved
ACPAgentConfig.env and the SandboxConfig docstring said $ values are
resolved from the host environment without saying when, which reads as
"at spawn time" and invites a consumer-side second pass again. Both now
say it happens when config is loaded, matching SandboxConfig.environment.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
test_skillscan_env_launcher.py and test_channels.py put payloads of up to
52 KB directly in parametrize values, so pytest's generated item ids exceed
the 32767-character Windows environment-variable limit: PYTEST_CURRENT_TEST
raises ValueError at setup and teardown even though the products under test
behave correctly, and the giant ids flood -q reports.
The skillscan linearity test additionally spawned one child process per
chain. A fresh child spends 9-18 s importing the skills stack before the
4-168 ms scan, so its fixed 10 s timeout measured the import instead of the
scan and flipped to subprocess.TimeoutExpired on a loaded host.
Give the oversized cases short explicit ids, scan every chain in one child
process fed through stdin, and enforce the linearity bound on each scan
inside that child (5 s) while a 60 s outer timeout covers the import.
* fix(lark): publish the skill file and manifest atomically
_write_skill_file and _write_manifest wrote their final paths with a truncating
write, while the runtime files in the same module already go through
_publish_bytes_atomically. A partial skill file or manifest is unparseable, and
the manifest carries the content hash the install is verified against, so a
half-written one cannot be repaired. Route both through the existing helper.
* fix(lark): address skill publication review feedback
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(agents): skip content blocks with unhashable type values
has_visible_content and ParentContextSnapshot.from_state classified a
content block's type with set membership (`in {"text", "output_text"}`
plus the _MEDIA_BLOCK_TYPES frozenset). `type: []` / `type: {}` are
legal JSON from a provider, and hashing an unhashable value raises
TypeError instead of skipping the block:
- has_visible_content runs on every model response through the
llm-error-handling (_raise_for_empty_response), terminal-response and
model-length middlewares, so one malformed block turned a completed
response into a failed run instead of a skipped block;
- from_state builds the history every background subagent dispatch is
seeded from, so the same block failed the dispatch.
Probe the type by equality with tuples so a non-string type simply fails
the probe, mirroring #6614's fix for extract_response_text and the
isinstance guard clarification already uses for form field types. The
media constant becomes a tuple too, so the second probe in the same loop
cannot move the crash one line down.
Tests pin the RED-to-GREEN cycle at both call sites: unit-level
has_visible_content, middleware-level empty-response classification,
and the dispatch snapshot across human/ai/tool messages.
Fixes#6663
Signed-off-by: yetuge <2219677952@qq.com>
* fix(agents): warn when a snapshot skips a non-string-typed block
Review follow-up on #6664: with the malformed block now skipped
instead of raising, a provider that systematically emits non-string
types would degrade the child's history with nothing in the logs to
explain it. The elif chain cannot log on fall-through —
reasoning/signature and tool-use exclusions are deliberate and must
stay quiet — so the signal keys on the type not being a str, matching
the guard the review suggested.
has_visible_content keeps its silence on purpose: the empty-response
retry already surfaces the symptom there.
Fixes#6663
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* fix(tavily): contain malformed web search and fetch payloads
web_search and web_fetch normalized the SDK's decoded 200 body without
validating its shape: a `results`/`failed_results` container of the wrong
type, a non-object entry, or a missing leaf field escaped as a raw
TypeError/KeyError/AttributeError instead of the structured format error
brave, serper, serply, sofya, tencent_wsa, unbrowse and groundroute return.
Validate the container with one shared helper, skip non-object entries while
preserving valid objects in order, normalize missing leaf fields to empty
text, and coerce a non-string raw_content in web_fetch before truncating.
Diagnostics carry only the malformed container's name and type, never the
query, URL, key or payload values.
backend/tests/test_tavily_response_shapes.py drives the real tools and SDK
over an offline HTTPX MockTransport; it fails 32 of 36 cases on main and
passes with this change, while test_tavily_tools.py stays green.
* docs(readme): note Tavily payload containment
Follow the Brave and GroundRoute README entries in the setup wizard step; the containment contract itself lives in community/tavily/AGENTS.md.
* fix(tavily): normalize null search fields and test mixed fetch lists
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
_write_lark_cli_runtime_from_archive and _write_lark_cli_sandbox_launcher wrote
their file with a truncating open and chmod 0o755 afterwards, so a crash mid-write
left an executable fragment behind - for the extracted binary that file can be
large. Publish through a temporary file plus os.replace, as the flow-state writer
in the same module already does.
* fix(tools): publish the default agent's SOUL.md atomically
setup_agent persisted the default agent's global SOUL.md with a
truncating write_text. The open truncates the previous file before the
new content lands, so an interrupted setup — process kill, disk full,
an exception while flushing — leaves a truncated persona file, and the
prompt builder reads SOUL.md on every run, so the agent then silently
runs with part of its instructions missing.
The same file is already published atomically everywhere else: the
custom-agent store stages it to a temporary and commits with
os.replace, and #6618 replaced the identical truncating pattern for
the Lark CLI runtime files the same day. The default agent's copy was
the one remaining write of this file that could tear.
Publish through a staged temporary plus os.replace, mirroring the
store's _stage_temp commit, and clean the temporary up when the
publish fails so no partial artifact survives.
Tests: a failed publish (os.replace raising) keeps the previous
SOUL.md byte-for-byte and leaves no temporary behind, a control pins
that the old truncating write loses the content in the same failure,
and the happy path writes the full file.
Fixes#6667
Signed-off-by: yetuge <2219677952@qq.com>
* fix(tools): keep SOUL.md's mode and clean the temporary on staging failure
Review follow-up on #6668, both non-blocking suggestions that guard the
publish itself:
- NamedTemporaryFile creates the temporary with mode 0600, so the first
publish made the global SOUL.md 0600 (the old write_text produced 0644
& ~umask) and every re-publish clobbered a mode an operator had set.
Stat the target and chmod the temporary to that mode (0644 for a first
publish) before the replace — the order #6618 codified for the Lark
runtime files and atomic_write_extensions_config already follows.
- A failure inside the staging block (handle.write or the implicit
flush/close, e.g. ENOSPC) raised before the try started, orphaning the
delete=False temporary in base_dir. One try/finally now spans staging
and publishing, so neither phase can leave a temporary behind.
The exact POSIX mode semantics were verified outside the Windows host
(WSL Ubuntu: mkstemp is 0600, first publish is 0644, a re-publish keeps
an operator-set 0640, and both failure phases leave no temporary); the
exact-mode tests are therefore POSIX-gated, with a cross-platform test
pinning that the chmod lands on the temporary before the replace and a
staging-failure test that is red on the previous head.
The shared-helper suggestion is left as a follow-up: folding six
independent publishers into one helper is a cross-module refactor, not
part of this bug fix.
Fixes#6667
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* fix(tui): keep a refused message in the composer
on_composer_input_submitted cleared the composer before _send_to_agent
decided whether it could start a run. A prompt typed during an active run,
while an interrupted run on the same thread was still stopping, or when the
worker failed to start was replaced by a notice and survived only in input
history.
_send_to_agent now reports whether it started a run, and the submit handler
puts the exact submitted text back with the cursor at its end, so Enter
resends it. Accepted messages and local slash commands clear the composer as
before.
* fix(tui): keep a launch message that fails to start in the composer
on_mount sent the launch message through _send_to_agent and ignored the
result, so a worker start failure lost the message: it never reached the
composer or input history. Route both the launch message and composer
submissions through one _restore_draft helper.
* docs(changelog): reference #6666 for the TUI refused-message fix
* fix(tui): keep the palette closed over a restored slash draft
Restoring a refused bare skill activation such as /tdd posted
TextArea.Changed, which reopened the command palette, so Enter filled the
skill instead of resending it. Remember the restored draft and keep the
palette closed until it is edited. Restore the history copy of the text too,
so a trimmed slash command matches what Up recalls.
Refresh the locked lark-oapi from 1.5.5 to 1.7.3 without touching the
declared >=1.4.0 constraint, which already admits the new release.
Deployments syncing from the lockfile move off the 1.6.x line, where a
server-side protocol drift reportedly left long-connection clients
connected but receiving no event pushes. Keeping the constraint as-is
avoids shutting out downstream consumers pinned to older but working
releases (1.5.5's WS event routing, including the callback processor map
used by register_p2_card_action_trigger, is verified intact in source).
- Refresh uv.lock only: lark-oapi 1.5.5 -> 1.7.3, specifier unchanged
- Feishu channel, channel-connections, and thread-binding suites pass
on 1.7.3 (111 passed, 1 skipped); ruff clean
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(authz): admin user-management UI and disabled-account copy (#4063 gap 3)
Frontend slice of the account lifecycle backend (#6566):
- Settings gains an admin-only Users page (core/auth/admin-users.ts ->
GET/PATCH /api/v1/admin/users): email, read-only role, disabled state,
and a suspend/restore toggle per row; the Gateway's rejection detail
(last-active-admin 409) renders verbatim. The nav entry is hidden for
non-admins and the page guards independently; ?settings=users joins the
deep-link whitelist (now an exhaustive Record over SettingsSection).
- A session that turns out disabled maps account_disabled to a login
redirect that states the reason on all three surfaces: the SSR
AuthResult tag (consumed exhaustively by the workspace/artifacts/auth
layouts), the shared fetcher's 401 interception, and
AuthProvider.refreshUser. No next target is preserved - a disabled
account cannot sign back in. Login failures surface the localized
account_disabled copy when the code is present.
- userSchema stays disabled-optional (absent = not disabled); zh/en
locales and docs updated in the same change.
* fix(authz): address review on admin user UI and disabled copy (#4063 gap 3)
Review follow-ups on PR #6641:
- Direct /login visits with a suspended session now surface the SSR
account_disabled verdict through AuthDisabledContext instead of
discarding it (willem-bd): the auth layout wraps pages with the
provider and the login form seeds its reason from it — no redirect,
so no loop. Regression tests cover both verdict states.
- admin-users.ts parses both endpoints against userSchema /
z.array(userSchema) (zhfeng): malformed payloads fail the load
instead of rendering phantom rows; toggle error copy is scoped to
GatewayApiError with a localized fallback for anything else.
- The localized account_disabled copy is login-only (zhfeng): sign-up
keeps the backend message verbatim so a suspended account's email
cannot be probed through the register surface.
- Settings page folds the load banner and toggle error into a single
alert region (zhfeng); adds coded-envelope 409, malformed-row, and
load-failure tests; fetcher.ts documents the 401 body-consumption
contract.
- prettier across touched files (CI format gate).
The admin-users query key keeps user?.id, matching the
model-settings-page precedent (converge later per review).
* test(local-dev): normalize the upgrade script's MSYS path rendering on Windows
nginx-local-conf.sh computes its repo root with pwd -P, which inside
Git Bash yields the MSYS mount form (/tmp/... for the Windows temp dir)
— correct for its bash-native consumers (serve.sh, nginx.sh) but never
equal to the test's Windows-form Path.resolve(). Map the rendering back
through cygpath -w before comparing, so the tests read and compare the
rendered config on Git Bash hosts too.
Fixes#6657
* test(local-dev): harden Git Bash path normalization
---------
Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(tui): keep every word of an unquoted --print/--json message
--print and --json took a single optional value and never read the
positional words, so `deerflow --print summarize this repo` sent only
"summarize", `deerflow summarize this repo --print` exited 2 asking for a
message on a terminal, and with piped stdin the typed words were ignored.
Both flags now take every following word and fall back to words given
before them, as --cli does; a typed message wins over piped stdin. A
message split around the flag is refused with exit 2, because argparse
loses the word order between the two parts.
* docs(changelog): reference #6636
* fix(tui): refuse --print and --json together
Only one headless mode runs, so the other flag's words were dropped:
'--print hello --json world' ran with 'hello', and '--print --json hello'
asked for a message. Put both flags in a mutually exclusive group so argparse
exits 2 for every combination.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): add You.com web search provider
Adds You.com as an opt-in community `web_search` engine, mirroring the
existing community engines (tavily, brave, exa, firecrawl, groundroute).
You.com runs its own web index and returns extracted page snippets per
hit rather than a single meta description, so a research loop gets usable
context without a follow-up fetch.
The provider picks its endpoint by whether a key is configured, so it
works out of the box and scales up when the operator adds one:
- no key -> /v1/agents/search, keyless free tier
- YDC_API_KEY -> /v1/search with X-API-Key, full Search API
Endpoint and headers are always chosen together; the keyless endpoint
rejects an auth header, so a stale key can never be sent there. Both
return the same body shape, so one normalizer covers both.
httpx only, no new dependency and no pyproject extra, so uv.lock is
untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(community/youcom): review follow-ups — drop stray .pr-body.md, guard OverflowError, add 429 hint
* fix(doctor): make the You.com free-provider detail true with or without YDC_API_KEY
The doctor detail returned before any key check, so operators who had
set YDC_API_KEY still saw 'set YDC_API_KEY for the full Search API'.
Reworded to match the sibling free-provider entries and stay accurate in
both states.
* fix(search): sweep youcom max_results onto the shared owner
The single-owner gate that landed with #6563 flags this branch's private
_coerce_max_results as a newly hand-rolled copy, so backend-unit-tests
now fails after merging main. Follow the coordination note from review:
delegate to deerflow.community.search_max_results.coerce_max_results,
which owns the rejection profile (booleans, non-integral floats,
OverflowError), and keep only the 1-100 mirror of You.com's server-side
count clamp on top — clamping policy stays per-provider.
The .inf regression test now goes through the public tool path and also
pins the bool and non-integral-float shapes of the shared bar.
* docs(tools): use the bare you.com platform link in config and docs
Match every other provider entry: no other provider in config.example.yaml
or the en/zh tools.mdx uses campaign-tagged URLs, so drop the utm params
from the three you.com/platform links. Integration attribution stays in
the identifying User-Agent sent to You.com hosts.
* test(youcom): cover shared max_results ownership and bounds
Register You.com among the shared-owner providers and assert default handling and the 100-result cap through outbound request parameters.
---------
Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: valueadd-lgtm <339269770+valueadd-lgtm@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): preserve server binding history across gateway workers
A peer Gateway worker could keep a stale pooled session after another worker deleted and identically re-added a stdio server, because the final effective MCP slice matched the revision it had already applied.
Record a per-server lifecycle generation in the shared extensions config and consume it in the reader, discovery claim, and committed handoff so an observed generation change force-rebinds only the affected server. Legacy configs derive a stable in-memory token; deletions leave permanent tombstones.
* fix(mcp): fail closed on an invalid mcpLifecycle instead of retrying
An unverifiable mcpLifecycle made initialize_mcp_tools() unpublishable, so get_cached_mcp_tools() retried discovery forever and could keep respawning stdio servers. Abort discovery on an invalid ledger after conservatively retiring local MCP state; a fixed config recovers on the next call.
Also documents the rolling-upgrade prerequisite: cross-worker ABA detection requires configuration writes to go through a lifecycle-aware writer.
* refactor(mcp): share stdio fingerprinting across lifecycle paths
Reuse lifecycle._stdio_fingerprints from the applied-revision builder so the connection-identity rule has a single implementation, and move the unusable-server diagnostic into the shared helper.
Add a regression proving the Gateway Skills writer preserves mcpLifecycle byte-for-value, and document that DeerFlowClient.update_mcp_config can raise RuntimeError after the config was committed.
* docs(mcp): condense lifecycle guidance under the AGENTS chain limit
The effective mcp/AGENTS.md chain was 15 bytes under the 98,304-byte CI hard limit, so the standalone cross-worker lifecycle bullet pushed it over. Fold the mcpLifecycle and rolling-upgrade facts into the existing selective-reconciliation bullet and condense it, keeping the chain under the limit while preserving the contract.
* docs: trim inherited agent guidance below CI size limit
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scripts): reject local BIND_HOST values nginx would misread
nginx-local-conf.sh checked only for address and hostname characters.
BIND_HOST=0 rendered `listen 0:2026;`, which nginx resolves to 0.0.0.0 and
binds on every IPv4 interface, and 1.2.3 bound 1.2.0.3. Values nginx rejects
outright ([[fd00::1]], -foo) passed the serve.sh preflight and failed only
after stop_all had torn down the running stack.
Accept exactly a dotted-quad IPv4 address, an IPv6 address (brackets
allowed around IPv6 only), or an RFC 1123 hostname. Returning to the
loopback default removes a temp/nginx.local.conf rendered for an earlier
value, and stop_all now says why it can quit a rendered-config nginx
through the tracked config; a test pins the shared pid directive.
* docs(changelog): reference #6635
* docs(changelog): match the zh reference spacing
* fix(scripts): check every BIND_HOST hostname character before splitting labels
read -r -a stops at the first newline, so is_hostname validated only the
first line of a value like $'localhost\nbad' and the rest reached the
rendered listen directive. Check the allowed characters across the whole
value first, and cover multi-line, CR and tab values for each validator.
* fix(lark): publish the CLI runtime files atomically
install_shim wrote the shim, the launcher and the runtime marker with a
truncating open. The sandbox chmods the launcher 0o755 and execs it, so a crash
mid-write left an executable fragment, and the marker is parsed as JSON. Publish
through a temporary file plus os.replace, as the storage backends already do.
* style: drop the unused import ruff flags
* fix(lark): publish runtime permissions atomically
Set file permissions on temporary files before os.replace() to ensure
the launcher, shim, and runtime marker are atomically published with
correct permissions (0o755 for executables, 0o644 for the marker).
This addresses reviewer feedback from PR #6591:
- Permissions are now set before the atomic replace operation
- Prevents brief window where files have incorrect permissions
- The runtime marker now has mode 0o644 instead of 0600 (mkstemp default)
Also updates test_lark_shim_atomic.py:
- Fixes test_atomic_helper to match new function signature
- Adds test_install_shim_publishes_correct_permissions to verify
launcher/shim (0o755) and marker (0o644) permissions
Co-authored-by: IronMurphy <hepushuo@gmail.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
`workspace_changes/api.py` imports `deerflow.runtime.user_context` for the
AUTO sentinel (#6282). Importing that submodule runs the runtime package init,
which imports `.runs` -> `worker` -> `deerflow.workspace_changes` back for
`capture_workspace_snapshot` and friends, so importing `.api` as the package's
first statement failed with ImportError while those names were still unbound.
Resolve the re-export lazily via a module-level `__getattr__`, the pattern
`deerflow/subagents/__init__.py` already uses.
* feat(dev): add a two-Gateway multi-instance harness script
There was no way to run the deployment.multi_instance topology locally:
scripts/serve.sh pins one Gateway to port 8001, the local nginx config has a
single upstream, and getting two processes past the multi-instance startup gate
needs Postgres, Redis, a shared DEER_FLOW_HOME, shared auth secrets and a
per-process DeerMem retrieval index by hand.
scripts/dev_multi_instance.sh (make dev-multi / dev-multi-check /
dev-multi-down) starts throwaway Postgres and Redis containers on loopback,
renders a config from the developer's config.yaml (or config.example.yaml),
and runs Gateway A on 8001 and Gateway B on 8011 with one shared home, one
copy of extensions_config.json and generated AUTH_JWT_SECRET,
DEER_FLOW_INTERNAL_AUTH_TOKEN and DEER_FLOW_CREDENTIALS_KEY values, plus an
optional round-robin nginx on 2027. B starts after A answers /health: Alembic
migrations are advisory-locked, but the LangGraph checkpointer/store setup()
that follows is not.
scripts/dev_multi_instance.py holds the testable parts: the config overlay
(neutralizing and reporting settings the gate refuses), the secrets file, the
nginx config, and `check`, which automates the scriptable cross-instance
acceptance checks: readiness with stream_bridge ok, A-issued sessions and the
internal token accepted by B, thread/upload/artifact visibility, a skill
toggle seen by B, nginx balancing, and SSE join on B with Last-Event-ID resume
on A. Tests pin that the rendered config passes the real
_enforce_postgres_for_multi_worker gate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link the two-Gateway harness entries to #6613
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(dev): keep the multi-instance harness inside its own state
Address review findings on the two-Gateway harness (#6613):
- Containers carry a deerflow.harness.state-dir label; up/down remove only
containers of their own state dir and refuse those of another checkout or
DEERFLOW_MI_STATE_DIR instead of force-removing them.
- Containers are removed with `docker rm -f -v`, so the anonymous volumes
the postgres and redis images declare no longer leak on every up/down.
- `up` installs an EXIT trap once it starts containers, so any later failure
(including B's port being taken after preflight) tears down A and both
containers; start_gateway returns failures instead of exiting the script,
and the Gateway is disowned so stopping it prints no job notice.
- The overlay removes any checkpointer section (a postgres one would write
checkpoints and the Store into the developer's database), points an
explicit sandbox.ownership and a Redis checkpoint_cache at the harness
Redis (keeping ownership timing), drops DeerMem storage_path (both
spellings) and the blob_storage root so data lands in the harness home,
gives a local AIO sandbox a state-dir-specific container_prefix, and
reports a non-DeerMem memory backend it keeps sharing.
- nginx paths are rendered as quoted string literals ('$' and control
characters are refused), so a state dir with spaces works.
- `check` skips the run-based SSE check when model discovery did not
succeed, unless --with-llm, and enforces the SSE time budget per raw line
so heartbeat-only traffic cannot hold it open.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(scripts): keep the harness guidance under the AGENTS.md soft limit
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat: return history search excerpts near keyword matches
* docs: use English for history search guidance and comments
* docs: sync Chinese history search excerpt contract
* docs: clarify history excerpt fallback for oversized matches
* fix(history): skip oversized occurrences when selecting excerpts
Continue scanning casefolded substring matches until an occurrence fits the 600-character excerpt, including overlaps. Cover later fitting hits through synchronous and asynchronous active/archive search and read tools, and update the documented contract.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(agents): sanitize persisted URL content blocks before model calls
Threads whose checkpoints carry URL-sourced file blocks (written before
the MCP conversion-layer fix) fail every model call at serialization: the
Chat Completions translator rejects any file block carrying a url, and the
error handler answers each later turn with a non-retriable fallback. Add a
read-time sanitizer that rewrites those blocks in the model-bound request
view only, so poisoned threads heal on their next turn with no migration
and no user action.
- ModelContentCompatibilityMiddleware wraps the model call (sync and
async) and rewrites ToolMessage list content: URL-sourced file blocks
become "[Resource ({mime or unknown type}) available at {url}]" text
blocks, and image blocks whose scheme is not http/https/data get the
same placeholder. Fetchable image URLs, base64 payloads, text, and
non-ToolMessage messages pass through. The rewrite is order-preserving,
idempotent, and never writes state or mutates checkpoints, so artifact
capture and other state readers keep seeing the original blocks.
- Register on all three chains that replay checkpointed ToolMessages:
build_middlewares (lead; outer of the MODEL_PHYSICAL extension anchor so
its final-request guarantee holds), the SDK factory
_assemble_from_features (new step 13a; docstring updated and its stale
omission of TokenBudgetMiddleware fixed), and
build_subagent_runtime_middlewares (after DurableContext/summarization,
before the date/coalescer tail).
- Subagent coverage verified by tracing every subagent agent-creation
site: executor.py _create_agent is the only caller of
build_subagent_runtime_middlewares and serves both the task tool and the
durable batch worker (SubagentRuntime itself only carries dependencies),
and a wiring test now asserts the middleware is present in that
builder's output; the lead and factory chains gained the same wiring
assertions.
- Update the chain-order contract tests for the new slot: full-chain
order, loop-detection adjacency, custom loop-detection tail positions,
and the subagent chain count (22 -> 23).
* fix(mcp): stop emitting URL-sourced file blocks from ResourceLink results
MCP ResourceLink conversion produced a URL-sourced file content block for
every non-image link, which LangChain's Chat Completions translator rejects
at serialization time. Once such a block was checkpointed in the
ToolMessage, every subsequent turn of the thread failed at model-call time,
permanently bricking the conversation.
- Downgrade every ResourceLink that is not an http(s) image to a short text
placeholder carrying the resource name, MIME type, and the
already-virtualized URL, so host paths never leak and the model still
learns a resource was produced and where it lives.
- Preserve each downgraded link as structured data in the tool-message
artifact under resource_links (merged with structured_content when
present) for artifact capture and future MCP Apps rendering.
- Keep http(s) image links as image blocks; embedded base64 payloads, text
path rewriting, and the error-to-ToolException path are unchanged.
- Re-contract the file-migration tests to the new behavior and document
the conversion contract in the MCP AGENTS.md.
* docs(middlewares): add ModelContentCompatibilityMiddleware to the chain inventory
Insert the read-time sanitizer at its actual lead-chain position (after the
token-budget guard, before the custom-middleware injection point) and renumber
the tail entries 34-39 to 35-40.
* fix(agents): rewrite file blocks on url key presence, not truthiness
langchain-core's OpenAI translator raises for ANY file block carrying a
url key — an empty string or None included — so the read-time sanitizer
must not let such blocks through just because the value is falsy. A
persisted block with url="" or url=None would still brick every later
model call in the thread.
- Gate the file-block rewrite on "url" in block instead of a truthiness
check; a non-empty string URL keeps the
"[Resource (mime) available at url]" placeholder, while an empty or
None URL gets a location-less "[Resource (mime)]" placeholder, so no
unusable location is advertised to the model.
- Pin both shapes with new middleware tests (url="" and url=None file
blocks are both rewritten to text). The image branch is unchanged: an
empty-image case fails provider-side with a 400, not a bricked thread.
* fix(mcp): keep data URIs out of model text and resource_links
A data: scheme ResourceLink fell into the generic placeholder branch,
inlining megabytes of base64 into the model-visible text and into
resource_links, which is checkpointed state — against the repo policy
that data:/blob: URIs never enter state.
- Special-case data: links in _convert_call_tool_result: the
model-visible placeholder becomes "[Resource: name (mime) embedded
inline]" and the raw URI is not added to resource_links at all, so the
payload can neither bloat the context window nor persist in the
checkpoint.
- Pin the behavior with a conversion test asserting both the placeholder
wording and the absence of the payload/URI from every emitted block and
the artifact.
- Add the missing passthrough pin for an HTTPS:// uppercase-scheme image
link: scheme matching is case-insensitive, so the link stays an image
block (the block's URL validation normalizes the scheme to lowercase).
* fix(tools): consume resource_links in extract_artifacts_from_result
The MCP conversion layer preserves downgraded ResourceLinks as
{"name", "uri", "mime_type"} entries under the tool message
artifact's resource_links key, but the artifact extractor only read
structured_content — a result whose only artifact signal was
resource_links produced no entries and the file silently vanished from
the artifacts panel.
- Add a fourth extraction path: each resource_links entry becomes a file
artifact entry, reusing _is_referenceable_url like the
structured-content and content-block paths, so data:/blob: and other
unresolvable URIs stay rejected per existing policy. The entry is named
after the link when a name is present, falling back to the URI's final
segment; a string mime_type is carried through.
- Integration-test a result whose only signal is resource_links, plus the
referenceability gate (data:, ui://, and absolute-path URIs in one
artifact).
* docs(mcp): update the ResourceLink conversion contract bullet
Mention the new artifact-registry consumer and the data: special case so
the bullet stays accurate: uppercase http(s) schemes match
case-insensitively, data: URIs get the embedded-inline placeholder and
are dropped from resource_links, and extract_artifacts_from_result turns
resource_links into registry entries through the shared
referenceability gate.
* refactor(tools): add the canonical resource placeholder formatter
Both the MCP conversion layer and the read-time
ModelContentCompatibilityMiddleware downgrade model-incompatible resource
references to a text placeholder, but each built its own literal. Extract
the single formatter they will share so a thread healed at read time shows
the model the same placeholder shape a fresh conversion produces; the name
segment is optional because persisted content blocks carry no resource name.
* fix(mcp): keep host paths out of ResourceLink placeholder text
Review follow-up on the ResourceLink conversion contract. The placeholder
embedded the resolved URL unconditionally, so a link outside the user-data
tree (or converted without thread context) leaked a raw host path — a
file:// URI or bare filesystem path — into model-visible text, violating
US-16. Compute the resolved URL's scheme and withhold the location segment
when it spells out a host filesystem location (empty scheme, file, or a
single-letter Windows drive scheme); virtual paths and remote schemes keep
their location. The unresolved URI still lands in the artifact channel,
which the model never sees.
- add _model_visible_link_location gate and route the generic placeholder
through it, re-contracting the outside-user-data and no-context tests to
the location-less shape and pinning the Windows-drive case
- treat blob: like data: as non-referenceable inline input: drop its raw
URI from checkpointed resource_links and emit the location-less
placeholder instead of the mismatched 'embedded inline' wording, and fix
the comment that claimed blob: was already excluded
- replace every inline placeholder f-string in the ResourceLink branch
with the shared resource_placeholder_text formatter so the conversion
layer and the read-time middleware emit byte-identical shapes; data:
links now render in the canonical location-less form
- sync the ResourceLink conversion contract bullet in mcp/AGENTS.md and
the _convert_call_tool_result docstring with the host-path suppression
rule and blob: handling
* fix(agents): sanitize URL content blocks on any message role
Address the second-pass review of the request sanitizer with two
changes that share the same code path.
- Adopt the canonical resource_placeholder_text formatter from
deerflow.tools.resource_placeholder instead of the module-local
placeholder helpers, so the read-time rewrite and the MCP conversion
layer emit byte-identical placeholder text from one definition.
- Widen the rewrite from ToolMessage-only to every request message with
list-form content. User uploads are verified to produce base64 blocks,
but imported or cross-client history can carry a URL-sourced file
block on a HumanMessage or AIMessage, and the Chat Completions
serializer rejects such a block regardless of the role carrying it.
The rewrite stays request-view-only: state and checkpoints are never
mutated, and base64/file-id blocks and string content pass through
untouched on any role.
* docs(agents): state the ModelContentCompatibility placement contract
The comment above the registration claimed the sanitizer sits just
before Clarification. In the lead chain it is actually followed by the
custom/extension injection point and the whole terminal tail
(terminal-response, finish-reason, and the always-last
ClarificationMiddleware). Describe the real contract: after every
built-in content transform, before extension injection and the
terminal tail.
* docs(middlewares): clarify provenance stamps for request-view-only rewrites
The message-provenance rule read as if every rewrite needs a
provenance_kwargs stamp. Add one line noting the exception: rewrites
that only modify the model-bound request view and never enter state
need no stamps, citing InputSanitizationMiddleware,
DanglingToolCallMiddleware, and ModelContentCompatibilityMiddleware as
precedents.
* fix(agents): keep host paths and data URIs out of healed placeholders
The read-time ModelContentCompatibilityMiddleware rewrote URL-sourced
file blocks and non-fetchable image blocks into placeholders that
inlined the block's URL verbatim. Pre-fix checkpoints — exactly the
threads this middleware exists to heal — can carry a file:// URI, a bare
or Windows host path (links that never resolved to virtual paths), or a
data: URI embedding megabytes of base64, so healing leaked deployment
filesystem layout into prompts (US-16) and could inject the whole inline
payload into model-visible text.
The conversion layer already suppresses those forms via
_model_visible_link_location. Move that gate into the shared
deerflow.tools.resource_placeholder module as model_visible_location
(now also collapsing data:/blob: URIs) and route both middleware
branches through it, so both layers apply identical suppression rules
with one definition. Conversion behavior is unchanged: its data:/blob:
branch short-circuits before the gate, and the gate covers those schemes
for defense in depth.
- resource_placeholder.py gains model_visible_location alongside the
shared formatter; mcp/tools.py drops its private copy and imports it.
- The middleware's file and image branches pass the URL through the gate
before building the placeholder; module docstring documents the rules.
- Tests: gate unit tests (virtual/remote visible; file://, bare/Windows
host paths, data:/blob: withheld) and middleware regression tests
pinning that a pre-fix data: payload, file:// URI, and Windows drive
path heal to location-less placeholders.
- AGENTS.md bullets in mcp/ and middlewares/ state the shared gate.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(agents): downgrade malformed image URLs instead of raising in the sanitizer
The read-time ModelContentCompatibilityMiddleware gated persisted image
blocks on urlparse(url).scheme, but urlparse raises ValueError on
structurally invalid URLs — a bad IPv6 literal, invalid netloc
characters — instead of returning an unusable scheme. Pre-fix
checkpoints and imported history can carry exactly those URLs, and the
block stays in the checkpoint, so every later model call in the thread
raised again: the sanitizer re-bricked the very threads it exists to
heal, only with a framework-side exception instead of the langchain
serialization error.
Catch ValueError and treat the URL as non-fetchable, so the block
degrades into the same text placeholder as any other unresolvable
scheme (the shared model_visible_location gate already swallows
ValueError, so the location segment is withheld as well).
- wrap the image branch's urlparse call in try/except ValueError and
fall back to an empty scheme, keeping http/https/data pass-through
behavior unchanged for well-formed URLs
- add a regression test pinning that an image block with a bad IPv6
literal heals to the location-less placeholder without raising
Co-Authored-By: Codex <noreply@openai.com>
* fix(tools): dedupe artifact refs across extraction sources
One downgraded ResourceLink produced two registry entries: the typed
resource_links channel added an entry for the URI, and the free-text
scan over the conversion layer's placeholder text — which embeds the
same virtual path or remote URL — matched it again. The two handles
pointed at one resource, spent double of the render budget, and showed
the model two handles for a single artifact.
extract_artifacts_from_result now tracks seen real_refs across all
sources and keeps only the first entry per ref. Extraction order makes
the typed sources (structured content, resource_links, content blocks)
win over the free-text scan, so the surviving entry keeps the link's
display name and mime type.
- route all six entry-producing sites through a local _add_unique
helper keyed on real_ref; per-source behavior is unchanged
- document the cross-source dedup in the function docstring
- add a regression test covering a virtual-path and a remote-URL
placeholder together with their resource_links entries
Co-Authored-By: Codex <noreply@openai.com>
* docs(mcp): state the intentional data: image downgrade asymmetry
The conversion layer downgrades image/* data: ResourceLinks to
location-less text placeholders, while the read-time middleware lets
persisted data: image blocks pass through to the model. The two
contracts looked contradictory but are deliberately layered: the
conversion layer keeps new inline payloads out of checkpointed state,
and the middleware only preserves blocks already occupying one.
- mcp/AGENTS.md's ResourceLink conversion contract and the conversion
comment in mcp/tools.py now state the image/* data: downgrade and
why it differs from the read-time pass-through
- middlewares/AGENTS.md's ModelContentCompatibilityMiddleware entry
states the symmetric read-time rule
- pin the behavior with an image-mime data: ResourceLink conversion
test (location-less placeholder, nothing in the artifact channel)
Co-Authored-By: Codex <noreply@openai.com>
* docs(backend): compact AGENTS.md guidance to fit the chain byte budget
The branch's new contract bullets pushed the middlewares and mcp
effective AGENTS.md chains past the 98304-byte AG002 hard limit (the
sandbox/subagents chains on main were already over it). Compact the
guidance without dropping contracts:
- middlewares item 34 points at the MCP ResourceLink conversion
contract and the module docstring instead of restating both
- the mcp ResourceLink conversion contract bullet is tightened
- backend/AGENTS.md merges the File Upload and Context Summarization
bullet sprawl, folds the stray ATX test note into the upload outline
rule, drops the Nginx routing block the root guide already covers,
merges the duplicate MCP-task pointer bullet, compacts Vision
Support, and scopes the Project Structure tree to the backend
subtree (the root guide maps the full repo)
Co-Authored-By: Codex <noreply@openai.com>
* fix(agents): gate the file-URL downgrade on provider support and neutralize placeholder metadata
Two review findings on the read-time sanitizer:
- The file-block downgrade ran for every provider, but only Chat
Completions rejects URL-sourced file blocks: the OpenAI Responses API
(input_file file_url) and Anthropic (document url source) accept
http(s) document URLs, so a persisted block created for those
providers lost its document from the model request. Keep the block
when the selected model serializes http(s) file URLs natively; every
other URL shape (non-http schemes, empty/None, malformed) still
degrades to the placeholder.
- The placeholder interpolated persisted MIME/location metadata
verbatim, but the middleware runs after the input/tool-result
sanitization passes and historical URL blocks never went through them
(non-text content is left untouched), so values carrying
<system-reminder>-style tags or user-input boundary markers reached
the model raw. Route the interpolated metadata through the shared
neutralize_untrusted_tags guard in both the file and image branches.
Tests: provider gating (Responses/Anthropic preserve, Chat Completions
still downgrades, non-http and malformed URLs downgrade even for
capable providers) and metadata neutralization (injection tags
HTML-escaped, boundary tokens neutralized, file and image branches).
Co-Authored-By: Codex <noreply@openai.com>
* fix(tools): require a directory boundary before exposing virtual locations
model_visible_location admitted any URL starting with the virtual
prefix, so a sibling like /mnt/user-data-backups/ops/private/report.pdf
was treated as a virtual path and its host location reached
model-visible placeholder text — defeating the US-16 host-path
suppression the gate exists for. Require the exact virtual root or a
slash-delimited descendant; siblings now collapse to the location-less
placeholder like any other host path. Both the file and image
downgrade branches share the gate, and the conversion layer applies
the same rules to fresh results.
Regressions: gate unit tests (exact root and descendants stay visible;
/mnt/user-data-backups and /mnt/user-database siblings withheld) and
middleware tests pinning the location-less placeholder for both
branches.
Co-Authored-By: Codex <noreply@openai.com>
* docs(agents): scope the placeholder artifact-visibility claim to the conversion layer
The module docstring claimed the healed placeholder keeps the location
visible to the artifact free-text scan and that the link survives in
the structured artifact channel — true for the conversion layer, whose
placeholders are checkpointed, but not on the healing path: the request
view never reaches artifact capture, and no resource_links entry is
created for a downgraded block. Spell out what actually happens here:
a referenceable original block (http(s) URL or absolute host path) is
still captured, from the capture-time scan of the state block rather
than the placeholder, while a non-referenceable one (file://, data:,
blob:) yields no entry and survives only in the untouched checkpoint.
Co-Authored-By: Codex <noreply@openai.com>
* docs(mcp): clarify healing and artifact preservation contracts
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* test: route four unguarded symlink fixtures through symlink_or_skip
`tests/support/symlinks.py` was added in #5414 so suites that need a real
symlink skip cleanly when the host cannot create one. On a stock Windows host
(no Developer Mode, not elevated) `Path.symlink_to` / `os.symlink` raise
`OSError: [WinError 1314]`, and four fixtures still called them directly, so
they reported hard errors instead of skipping:
- tests/test_shared_reset_marker.py::test_marker_path_follows_a_symlinked_config
- tests/test_view_image_middleware.py::test_host_image_copy_is_scoped_to_current_user_and_thread
- tests/test_review_skill_package_tool.py::test_review_skill_package_rejects_symlink_planted_in_callers_outputs
- tests/test_channel_file_attachments.py::TestInboundFileSandboxPerms::test_make_inbound_file_sandbox_readable_swap_after_lstat_does_not_follow_symlink
These were the only symlink-related failures in a Windows run of the full
backend suite; the other bare symlink call sites there are already guarded by a
`skipif` or a `try/except OSError`. The view-image test plants its symlink
after its scoping assertions, so those still run before the skip. The
inbound-attachment test stages a symlink from inside a monkeypatched `os.lstat`
to exercise a TOCTOU race; a host that cannot create a symlink cannot stage
that race at all, so skipping is the only honest outcome.
* test: make channel symlink race fixture portable
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(security): add DEER_FLOW_CREDENTIALS_KEY for at-rest credential encryption
There was no at-rest encryption key. ChannelCredentialCipher had no
production caller: the channel service, the scheduled-task notification
outbox and the channel-connections router all built
ChannelConnectionRepository without a cipher, so store_credentials refused
(RuntimeError) and get_credentials always returned None.
- New harness module deerflow.config.credentials_key: env-only
DEER_FLOW_CREDENTIALS_KEY (comma-separated Fernet keys; first encrypts,
all decrypt via MultiFernet), CredentialsCipher with encrypt_text /
decrypt_text / rotate_text, a fernet:v2: prefix with legacy fernet:v1:
values still readable, and get_credentials_cipher() as the process-wide
entry point. Malformed keys are refused without echoing key material.
Without the variable a single instance generates
{base_dir}/.credentials_key. Never derived from AUTH_JWT_SECRET.
- New deerflow.config.secret_file.read_or_create_secret_file: O_EXCL
create (0600) plus read-back with a bounded settle window, shared by
.credentials_key, .jwt_secret (fixes two replicas cold-starting on a
shared volume keeping different JWT secrets) and the managed-model key.
- ChannelCredentialCipher now builds on CredentialsCipher (same
constructors); the Gateway loads the cipher off the event loop at
startup when channel_connections is enabled and passes it to every
ChannelConnectionRepository. Undecryptable values stay "missing + warn".
- Startup gate: a malformed key stops startup in any topology; a declared
multi-instance deployment with channel_connections enabled and no key
is refused. Plain multi-worker shares the generated file and is allowed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(deploy): generate and inject DEER_FLOW_CREDENTIALS_KEY in Helm, compose and deploy.sh
- Helm: the app Secret generates a Fernet key (randBytes 32 translated to
the urlsafe alphabet) and preserves it across upgrades via lookup like
AUTH_JWT_SECRET; the gateway reads it through an optional secretKeyRef
so a user-managed existingAppSecret without it still boots (the Gateway
gate refuses only multi-instance deployments that store credentials).
README, NOTES and values document backup, rotation and existingAppSecret.
- scripts/deploy.sh resolves the key shell -> .env -> persisted file ->
generated, persisting it as $DEER_FLOW_HOME/.credentials_key (0600,
noclobber/O_EXCL with read-back): the same file the Gateway generates in
its runtime home, so either side's key is the one both keep.
- Both compose files pass DEER_FLOW_CREDENTIALS_KEY=${...:-} to the gateway.
- .env.example entry with a generation hint; README and CHANGELOG notes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): make abandoned secret-file replacement single-winner
A rename is last-writer-wins, so replicas that restart together and all
find the same abandoned empty secret file could each replace it and keep
a different value. The process that exclusively creates
<name>.replacing now owns the replacement, re-checks the file is still
empty, and publishes by renaming the claim over it; peers wait for that
value, bounded so a crashed replacer cannot block startup forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link DEER_FLOW_CREDENTIALS_KEY entries to #6611
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): name the setting that turns off a credentials-key consumer
The multi-instance refusal ended with "or disable
channel_connections.enabled=true"; it now says "or set
channel_connections.enabled=false".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): never expose a secret file before its value is written
O_EXCL published the name before its content, so a creator that stalled
mid-write left an empty file a peer replaced after the settle window; the
resumed creator then returned a value nobody persisted, or deleted the
peer's valid file when its own write failed. Creators now write and sync
a private temporary file and hard-link it to the name, which fails if
the name exists; a failed creator only removes its own temporary file.
Filesystems without hard links publish through the same single-winner
<name>.replacing claim that replaces abandoned empty files, so creation
and replacement are serialized there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(models): recover an empty managed-model key when no catalog exists
An interrupted first save could leave an empty key file; every later save
then passed empty bytes to Fernet and the endpoint returned 503 until the
file was removed by hand. An empty key with no catalog encrypted nothing,
so it now goes through the secret-file recovery path; next to an existing
catalog it is still refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(docker): pass the dotenv DEER_FLOW_CREDENTIALS_KEY through make docker-start
scripts/docker.sh runs Compose from docker/ without --env-file, so the dev
gateway's DEER_FLOW_CREDENTIALS_KEY=${DEER_FLOW_CREDENTIALS_KEY:-} entry
interpolated to empty and overrode the key the checkout .env supplied via
env_file. Export it from the dotenv like AUTH_TRUSTED_PROXIES.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test: fix two Windows-host failures (subprocess encoding, budget-boundary epsilon)
- test_env_file: pin PYTHONIOENCODING=utf-8 in the stripped child env;
without it the child writes stderr in the host's ANSI codepage while
the UTF-8-mode parent decodes it, the reader thread dies on the first
non-ASCII byte, and stderr comes back None;
- test_jina_retries: give the budget-cap assertion a 1e-9 tolerance —
deadline = now + budget rounds, so the cap lands a few ulps above the
exact 0.05 (observed 0.05000000000291038).
Fixes#6626
* test: pin both sides of the capture and use an uptime-proof budget tolerance
- test_env_file: the parent's capture decodes with the host locale when
pytest runs without the Makefile's UTF-8 pins, so the child-side pin
alone mirrored the crash rather than eliminating it. Pin the capture
explicitly (encoding=utf-8, errors=replace, per the
test_local_sandbox_encoding.py precedent) and carry both suite pins in
the child env (PYTHONUTF8=1 + PYTHONIOENCODING=utf-8).
- test_jina_retries: the float error scales with loop-time magnitude
(host uptime), not with 0.05; a 1e-9 absolute slack leaves a ~389-day-
uptime flake window. Use a relative tolerance instead.
---------
Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com>
* fix(runtime): drain staged-terminal runs on shutdown before checkpointer teardown
RunManager.shutdown() selected only pending/running records, but with an event
store a run stages its terminal status in memory and keeps finalizing: journal
flush, delivery receipt, workspace scan, the duration checkpoint write and the
deferred terminal commit, all while terminal_commit_pending is set. A Gateway
shutdown landing in that window did not await the run, so langgraph_runtime
could close the checkpointer before the duration write landed - the #3373
class the drain exists to prevent.
Drain those records too, using the existing _awaits_terminal_commit state that
_renew_leases already relies on; await them without cancelling (a cancel would
skip their terminal tail) and never overwrite their staged terminal status
with interrupted.
* fix(runtime): surface shutdown terminal finalizer failures
Cover timed-out staged terminal runs for both commit barriers and terminal outcomes. Log failed finalizers with their run ID, staged status, and traceback while preserving status and completing the drain.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(acp): resolve a bare agent command through PATH before spawning
`invoke_acp_agent` handed `acp_agents.<name>.command` straight to the ACP
SDK, which spawns it with `asyncio.create_subprocess_exec`. That API applies
no `PATHEXT` lookup on Windows, so a bare npm launcher (`npx`, `mcode`)
raised `FileNotFoundError` although its `.cmd` shim was on `PATH`. The
documented `command: npx` and `command: mcode` shapes could not start, and
the tool reported "was not found on PATH" for an installed launcher.
Resolve the configured command with `shutil.which` before spawning — the way
the MCP Python SDK normalizes stdio server commands — and keep the configured
value when nothing matches, so the existing install guidance still fires.
Fixes#6597
* fix(acp): resolve the ACP agent command off the event loop
`_resolve_agent_command` calls `shutil.which`, which stats the filesystem.
Calling it inline in the async `_invoke` tripped the strict BlockBuster gate
(`BlockingError: Blocking call to os.access`) for every command that resolves,
and would block the Gateway event loop in production.
Offload the lookup with `asyncio.to_thread`, matching the neighbouring
`_get_work_dir`, `_build_acp_mcp_servers`, and `_format_invocation_error`
calls.
The blocking-io anchor configured `command="test-agent"`, a name that resolves
nowhere, so `shutil.which` short-circuited before `os.access` and the gate
never covered the new call. Point it at `sys.executable` so the lookup is
really exercised, and cover the unresolvable-guidance branch end-to-end with a
new test.
Refs #6597
* fix(acp): keep the configured command in ACP remediation text
Review follow-ups on the PATH resolution:
- Only bare names are resolved. A configured path is interpreted by the spawn
against the per-thread ACP workspace, so resolving it against the Gateway's
working directory could pick a different file or hand the spawn a path that
then fails.
- `_format_invocation_error` receives `agent_config.command` again. A command
can resolve on PATH and still fail to launch (a broken shebang, for example),
and reporting the resolved path made "was not found on PATH" false and
dropped the `codex-acp` hint.
- The blocking-io anchor uses a bare name backed by a shim on a controlled PATH
instead of `sys.executable`, which the bare-name scoping no longer resolves.
Tests: the pass-through assertion in
`test_invoke_acp_agent_uses_fixed_acp_workspace` pins PATH, and new coverage
covers the bare-name scoping and the spawn-failure message.
Refs #6597
* docs(acp): correct the path-resolution rationale
A configured path is resolved by the spawn in its own process, so the
pass-through reason must not claim it is resolved against the ACP workspace:
that only holds on POSIX, while Windows resolves a relative executable against
the calling process's directory. State the platform-neutral reason instead.
Refs #6597
* fix(acp): resolve the launcher against the agent's own PATH
The ACP SDK merges acp_agents.<name>.env over its inherited environment at spawn time, so a configured PATH, not the Gateway's, decides which launcher the agent would find. Resolve a bare command against that effective PATH, and use the Gateway's only when the agent does not override it.
Return a successful lookup as an absolute path: a relative PATH entry makes shutil.which yield a result relative to the Gateway's cwd, while the spawn runs with cwd set to the ACP workspace, where that path resolves elsewhere or not at all.
Regressions: a real-subprocess test with same-named launchers in the Gateway and agent PATHs (asserting which one ran), and a real-subprocess test whose PATH holds a relative directory while the ACP workspace is elsewhere. Both are red on the pre-fix code.
* Fail closed on delegated fetch backends to close SSRF gap
* fix(community): resolve delegated-fetch credentials from one config snapshot
Address review feedback on the fail-closed SSRF guard:
- firecrawl/fastcrw client construction no longer re-reads get_app_config(),
so the API key and endpoint come from the same revision the endpoint screen
validated (prevents cross-revision credential misrouting).
- fix the browserless test double to accept the new two-argument
_get_browserless_client signature.
- document the check-only (DNS rebinding) caveat on
validate_delegated_backend_url.
- add a CHANGELOG upgrade note for the default self-hosted backend break.
* fix(doctor): consolidate delegated fetch backend checks and deployment guidance
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(groundroute): contain malformed web search and fetch payloads
web_search and web_fetch assumed the GroundRoute 200 body was an object with
a `results` list of objects, and read it outside the request try/except. A
body that arrived as a bare list, as an object with a non-list `results`, or
with non-object entries raised AttributeError / KeyError out of the tool
boundary, so the tool call failed with a stack trace instead of the provider's
format error.
Validate the payload before normalizing it, following the containment rule
already applied to brave (#6550), serper, serply, sofya, tencent_wsa and
unbrowse: a non-object payload, a non-list `results` container, or a non-empty
list holding no objects returns the structured format error, non-object entries
in an otherwise valid list are skipped in order, and missing, null or empty
results keep returning "No results found".
* fix(groundroute): coerce non-string fetch fields
web_fetch sliced the fetched page's `content` (falling back to `snippet`)
to the snippet limit without checking its type, so a 200 body whose result
object held a number, boolean, or object there raised TypeError / KeyError
out of the tool boundary -- the same escape the previous commit closed for
the container shape. A null title also rendered as "# None".
Coerce non-string content to text before truncating, and render a missing
or non-string title as empty, matching the coercion sofya and unbrowse
already apply to their fetch fields.
* docs(groundroute): scope the falsy-title note to actual behavior
The guidance said a non-string title renders as empty, but the code only
empties falsy titles; a truthy non-string title still renders as its text
form via the f-string. Say "falsy" instead.
* feat(authz): account disable/enable across every authentication surface (#4063 gap 3)
Vertical slice 1 of the user-lifecycle work: an operator can suspend an
account and every authentication surface rejects it, while the row —
credentials, role, OAuth linkage — is retained so re-enabling restores
exactly what was suspended.
- users.disabled (NOT NULL, server default 0) via migration
0036_user_disabled; User/UserResponse carry and expose it.
- Enforcement at all four surfaces: password login (no credential
comparison happens), JWT session resolution (401 ACCOUNT_DISABLED —
sessions die at the next request), PAT resolution (the generic 401, no
oracle), OAuth provisioning (403 with a contact-admin message; the
link is kept so re-enable restores access).
- SQLiteUserRepository.set_disabled: single-column, serialized write
(same idiom as update_system_role) with the last-ACTIVE-admin guard —
disabled admins cannot authenticate, so they cannot restore management
access; re-enabling is always allowed. Field-scoping completed both
ways: the credential writer mirrors and never writes disabled, so
a stale password-change snapshot cannot re-enable a suspended account.
- PATCH /api/v1/admin/users/{id} becomes a partial account update
(system_role and/or disabled), each field routed to its field-scoped
writer; empty bodies 422.
Tests 11->18: route (disable/re-enable, last-active-admin 409, empty
422, combined update), real-repo (field isolation both directions,
disabled-admin-not-active, guard inside the serialized write, re-enable
always), login rejection with the CORRECT password (only the disabled
gate can reject) plus resolver-visible flag. Mutations verified red:
guard removed (M19), login gate removed (M20 — after hardening the test
with a real hash so the gate is load-bearing).
* fix(authz): portable boolean default, alternate-authenticator suspension, active-admin role guard
Review round 1 on #6566 — three P1s:
- [PG default] sa.text("0") is a raw integer expression PostgreSQL
rejects as a BOOLEAN default, breaking startup migration and fresh
create_all bootstraps alike. Both the migration and the ORM model now
use sa.false()/false() — dialect-rendered, portable.
- [alternate authenticators] WebSockets bypass AuthMiddleware, so a
disabled user's still-valid cookie could open a NEW browser/stream
connection and drive the retained session; the standalone LangGraph
authenticate callback accepted the same suspended JWT. Both now route
through a shared post-lookup validator (validate_resolved_session_user
in deps.py — token_version + disabled in one place) that the HTTP
dependency also uses, so a lifecycle change lands everywhere at once.
- [active-admin role guard] update_system_role counted every admin row,
so 'disable B, then demote self' — or a combined {system_role, disabled}
self-update — could strand the deployment with zero administrators able
to authenticate or re-enable anyone. The demotion guard now counts
ACTIVE admins only (disabled admins cannot authenticate), under the same
serialization boundary.
Tests 18->22: shared-validator verdicts, WS admission after suspension
(real token, real repo), LangGraph-callback admission after suspension,
disable-then-demote-self + combined self-update both rejected (fake-repo
route level and real-repo update_system_role level). Mutations verified
red: helper's disabled check removed (M21 — three admission tests fail),
active-admin predicate reverted to all-admins (M22 — repo-level strand
scenario fails). 366 tests green on the affected surface.
* fix(migrations): chain 0036_user_disabled after upstream 0037
The maintainer main-merge brought 0037_project_document_summaries,
which branched from 0035 independently of this PR's 0036 — alembic now
sees two heads and every engine bootstrap fails. Chain 0036 after 0037
(single head; revision numbers are labels, the graph is what matters).
* fix(migrations): renumber user_disabled to 0038 and follow the chain-test convention
Upstream landed its own 0036 (run_idempotency_request) with 0037 chained
after it while this PR was in review — my 0036_user_disabled collided on
the number and my interim chain-fix (0036 revises 0037) left the graph
in a state upstream's chain tests reject (heads == [0037] hardcoded).
- 0036_user_disabled -> 0038_user_disabled, revises
0037_project_document_summaries: single head 0038.
- Chain tests follow the repo convention (each migration PR appends the
new head): 0032/0035 gain the DOC_SUMMARIES link and CURRENT_HEAD =
0038; 0037's single-head assertion moves to CURRENT_HEAD with its
own-revision link check unchanged.
* fix(migrations): pin the remaining chain tests to the 0038_user_disabled head
The 0027/0028, 0029 and 0034 chain tests still pinned the single head to
0037_project_document_summaries (and 0034 asserted the head's parent was
0036_run_idempotency_request), so the renumbered lifecycle migration broke
two unit-test shards. Follow the chain-test convention: every pinned head
moves to 0038_user_disabled and 0034 walks the chain through 0037.
* fix(authz): enforce account suspension on the internal-auth dispatch surface
Round-3 review finding: the internal-auth principal is synthesized from
trusted headers without a users-row lookup, so a suspended user with an
active channel binding kept driving the agent, and scheduler-owned runs
(their fabricated in-process requests skip AuthMiddleware entirely) kept
launching. Two gates now enforce the lifecycle there:
- AuthMiddleware rejects owner-bound internal calls whose users row is
disabled (401 account_disabled) at the single gate every internal HTTP
call passes through. Auth-disabled mode is exempt (no suspendable
accounts, and its owner header carries the synthetic identity), and
owner ids without a users row stay allowed — a nonexistent account
cannot be suspended.
- start_run re-asserts it at run admission after the owner is resolved
for attribution: the scheduler records the rejection as the
occurrence's launch error and no run record is created.
Also addresses the remaining round-3 nits: drop the inline token_version
check duplicated by validate_resolved_session_user, correct the
local_provider comment (login never surfaces ACCOUNT_DISABLED), refresh
the stale 0038 docstring header, and restore the deps module caches in
the two suspension tests instead of leaking a closed engine.
* fix(auth): carry the disabled field on the frontend synthetic users
UserResponse now always includes disabled, so the auth-disabled contract
e2e (gateway /auth/me deep-equals the frontend synthetic user) failed
without the field on AUTH_DISABLED_USER. Add disabled to both synthetic
users and declare it optional on the zod schema — absent still parses as
not disabled for pre-gap-3 backends.
* fix(authz): log when the internal-owner suspension check is skipped
The middleware gate's users-lookup failure path silently degraded to
allow; keep the fail-open behavior but leave a WARNING with the
exception, so a store that cannot be read is observable in production
logs instead of invisible.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
