* feat(agent): expose read-only run status tool
* refactor(tools): name the subagent dispatch tool set
Hoist the hardcoded subagent dispatch tool-name literal into _SUBAGENT_DISPATCH_TOOL_NAMES so a future dispatch-tool rename has one place to update. No behavior change: the folded-subagent context_fill guard compares the same two names.
* test(tools): account for run status in collision assembly
* docs: keep script guidance within size budget
* fix(startup): detect extras for browser-configured IM channels
* docs(startup): clarify channel extra provider source
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scripts): reject a directory passed to support_bundle --out
`--out` names the zip, but the path went straight to `zipfile.ZipFile`, so
pointing it at an existing directory ended in a raw `PermissionError` traceback
(Windows) instead of the CLI's usual `error: ...` + exit 2.
`create_support_bundle` now raises the `ValueError` the CLI already maps to a
clean message, and the regression test pins exit 2 with no traceback.
* docs(scripts): say "zip file path" for support_bundle --out
Review nit: the help text said only "Output zip path", so the
file-not-directory contract the new validation enforces was invisible until a
run failed.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scripts): find Git Bash under a PortableGit mingw64 layout
`scripts/run-with-git-bash.cmd` derives `bash.exe` from `git.exe`'s parent,
which only holds for the Git for Windows `<root>/cmd` and `<root>/bin`
layouts. PortableGit and GitHub Desktop ship `git.exe` in
`<root>/mingw64/bin`, so the wrapper looked for `<root>/mingw64/bin/bash.exe`
and reported "Could not locate Git for Windows Bash" even though
`<root>/bin/bash.exe` exists. Every Makefile recipe that runs a repository
shell script failed on those hosts.
Resolve the shell under the Git installation root instead, and mirror the rule
in `backend/tests/support/shell.py` (documented as mirroring the wrapper) so
the shell-script suites stop skipping on the same hosts.
* test(backend): pin both Git layouts in one shell-discovery suite
Review follow-up for the PortableGit probe:
- `_git_roots` now probes `git.exe`'s parent and grandparent unconditionally —
exactly the two directories the Makefile wrapper checks — so the batch probe
and the Python mirror cannot drift; the docstring records that the first
`mingw64` candidate normally misses and is kept only to mirror it.
- fold the candidate-layout cases and the Windows-only wrapper case into
`tests/test_support_shell.py`, which already owns this module's unit tests,
instead of a second file with a duplicate `_touch` helper.
- report the wrapper's stdout in the failure message: its diagnostic goes to
stdout, so `stderr` was empty on the discovery-failure path.
* feat(deploy): select production Compose env files
Use an operator-selected dotenv file consistently for interpolation, container environment, secret probes and build extras. Preserve default selection and shutdown after a profile is removed.
Co-Authored-By: Codex <noreply@openai.com>
* fix(deploy): address production env selection review
Clarify Docker launcher entry points and render real Compose through the launcher to verify both service environments.
Co-Authored-By: Codex <noreply@openai.com>
* test(deploy): cover unreadable env selectors
Reject unreadable explicit selectors before Docker is invoked, using POSIX permission checks and restoring the fixture mode in finally.
Co-Authored-By: Codex <noreply@openai.com>
* test(deploy): scope dotenv assertions to selected secrets
---------
Co-authored-by: Codex <noreply@openai.com>
* fix(models): fall through an unusable credential source instead of ending the lookup
load_claude_code_credential's env branch returned None even when
_credential_from_direct_token rejected the value, so a valid
~/.claude/.credentials.json was never consulted for a blank token.
load_codex_cli_credential's 'or' chain stops at the first truthy layout,
so a blank or non-string top-level access_token hid a usable 'token' or
'tokens.access_token' in the same file -- and made the loader disagree
with scripts/doctor.py, which validates all three layouts.
Fixes#7139
* fix(models): try both env token sources and log the rejected one
A whitespace-only CLAUDE_CODE_OAUTH_TOKEN is rejected by _credential_from_direct_token, but os.getenv(A) or os.getenv(B) never reads ANTHROPIC_AUTH_TOKEN while A is truthy, so the blank first variable hid a usable second one. Test each source with the same non-blank rule, and log the rejected variable at debug (no token material) so the resolution outcome stays observable.
Also refresh the doctor.py mirror comment: the loader rejects whitespace-only tokens in every layout now, so the documented strictness divergence is gone.
* style(tests): format credential fallback log assertion
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(docker): make dev provisioner opt-in via compose profile
Gate the Kubernetes provisioner behind a Compose profile and have the
wrapper enable it only when config selects provisioner sandbox mode.
This lets the default Docker Compose stack start without kubeconfig.
Fixes#3232
* test(docker): render every compose profile in the env-file regression test
main added backend/tests/test_compose_default_bind_host.py, whose _render helper asserted that the dev provisioner still loads ../.env. `docker compose config` without a profile omits profile-gated services, so that assertion failed once the provisioner became opt-in.
Pass `--profile *` from the helper: the test is about an existing .env reaching every service, so it has to render the services a profile gates rather than only the ones `up` starts by default.
* docs(docker): restore script mode and localize direct Compose guidance
The dev provisioner is now opt-in, but the earlier edit also flipped scripts/docker.sh from 100755 to 100644 even though README.md still documents direct `./scripts/docker.sh` invocation. Restore the executable bit.
Also add the direct-Compose guidance to the translated READMEs so readers of README_zh.md, README_ja.md, and README_fr.md learn that `--profile provisioner` selects the Kubernetes mode and the default stack does not need Kubernetes.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scripts): feed config-upgrade inline python on stdin, not argv
On Windows setups where python/uv resolve through shim layers that
reparse arguments with cmd.exe (e.g. pyenv-win), the multi-line
`python -c "..."` payloads in config-upgrade.sh get mangled mid-flight
(observed as `IndentationError: unexpected indent` on a stray
`|| goto :error` line) and `make dev` fails before the Gateway starts.
Feed both inline scripts on stdin instead, which is safe across every
spawn chain, and load the checkout .env explicitly inside the resolver
probe: the default load_dotenv() discovery walks up from cwd/__file__,
which a stdin-fed script (`python -`) cannot satisfy. Pin both with a
static regression test.
* test(config): isolate upgrade fixtures from inherited dotenv selectors
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): arbitrate IM transports with a per-provider leader lease
Every Gateway instance connected every enabled IM platform with the same
bot credentials, so channels could not run on more than one instance
(Telegram 409 storms, Discord answering twice, scattered per-process
state). channels.leader_lease was reserved by #6746; this implements it.
- channel_leases table (migration 0041): one row per provider. Acquire is
one dialect-native upsert decided against the database clock and bumps
a fencing epoch; renew extends only a live lease at the holder's epoch;
release sets the expiry to 0 so a peer takes over on its next tick.
- app/channels/leadership.py: the holder's local monotonic deadline
(send time of the last confirmed renewal + lease_seconds); holds() turns
false one renewal interval early, so the transport stops at least
grace_seconds before a peer may connect. DB calls are bounded.
- ChannelService: start fence in _start_channel (no DB call) behind every
start path; fixed-rate supervisor renews/acquires and runs transport
starts/stops as per-provider tasks; takeover reloads the provider's
saved config; failed starts back off; shutdown releases only leases
whose transport stopped. get_status reports leader/leader_here and the
holder's transport state on every instance.
- Notification worker leaves rows of providers another instance holds.
- Gate: the enabled=false refusal now recommends auto/true; an explicit
true with database.backend memory is refused when a channel is enabled.
- Docs, Helm, deployment guides, AGENTS.md and CHANGELOG updated.
* docs(changelog): link the channel leader lease entry to #6843
* fix(channels): see peer-held leases for providers absent from local config
Review on #6843: a provider connected from the web UI on a peer never
enters this instance's channel config, so the notification worker here
still claimed its rows (and parked them for 900s) and GET /api/channels/
reported no leader for it. Every supervision round, and startup, now also
reads the whole lease table once (bounded), and notification exclusions
and status use those observations: a provider whose live lease a peer
holds is left to it even without local config.
Running that read concurrently with renewals exposed that, through
SQLAlchemy's aiosqlite adapter, an UPDATE ... RETURNING racing a read on
another connection can hand back one of its own bind values instead of
the column (stored rows stay correct; plain sqlite3 and raw aiosqlite do
not show it). A lease must never take such a row for a confirmed
renewal, so acquire/renew/release now decide by affected-row count and
acquire reads its epoch back in the same transaction.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): add channels.leader_lease and gate it under multi-process signals
IM channels other than GitHub connect outbound to their platform. With more
than one Gateway instance each one opens that connection with the same bot
credentials: Telegram rejects the second poller with 409, Discord answers every
message twice and per-process ChannelManager state diverges. The multi-process
startup gate said nothing about channels; the only guard was a comment in the
Helm values.
Add the `channels.leader_lease` section (`enabled: auto|true|false`,
`lease_seconds`, `grace_seconds`; startup-only, registered in the reload
boundary) that the per-provider leader lease will read, validated by a Pydantic
model from the non-schema `channels` mapping. The gate now refuses
`enabled: false` under a multi-process signal while a non-GitHub provider is
enabled under `channels.*` or connectable through `channel_connections`, naming
the providers and the rollback; a malformed block is refused on every topology.
`auto` follows the multi-process signal, so a single instance is unchanged.
ChannelService strips the block before its per-provider loops and exposes it
as `leader_lease`; `ensure_ready_channels` only visits registered providers.
The two-Gateway harness overlay leaves the block alone. config_version 60 in
the example, the Helm chart and its README; docs replace the comment-only
"keep 1 replica" guidance with the gate.
* docs(changelog): link #6746
* fix(channels): stop recommending the unimplemented leader lease as a remedy
Review on #6746: the refusal told operators to switch channels.leader_lease
from false to auto/true "so one instance owns each provider's connection", but
the transport arbitration ships separately, so that change only bypassed the
clause and every worker still started the same bot.
The refusal now offers only the remedies that work today (a single instance, or
disabling the channels) and states that auto/true are reserved for the lease
that ships separately. A regression helper in the gate tests asserts the
remedies start with the rollback and that auto/true are never offered. The
config example, Helm values/README, CONFIGURATION.md, README, the deployment
guide (en/zh), the model docstrings and both changelogs describe auto/true as
reserved settings rather than an ownership guarantee.
* fix(channels): warn on an enabled unknown channel key; reword the changelog entry
Review on #6746: the registry guard in ensure_ready_channels skipped a
misspelled provider key (channels.telegam) silently, where it previously
reached _start_channel and logged 'Unknown channel type'. An enabled dict
block that is neither a registered provider nor leader_lease now logs a
warning that lists the registry.
The changelog entry claimed the chart and docs drop the "keep 1 replica"
guidance; they keep it (auto/true do not make several replicas safe yet) and
only stop relying on the comment alone.
* fix(dev): disable UI channel providers in multi-instance overlay
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(extensions): add example guide and packaging checks
* fix(extensions): align packaging checks with installer
Reject Windows drive-qualified wheel paths before extraction and reuse the installer entry-point validator. Cover unsafe paths, dotted attributes, and valid dotted modules in regression tests.
* fix(extensions): skip unused wheel extraction
Only extract wheels when browser manifests need validation, with a backend-only regression test. Link the three remaining example READMEs in the developer guide.
* docs(frontend): trim guidance intro below size budget
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): add You.com web search provider
Adds You.com as an opt-in community `web_search` engine, mirroring the
existing community engines (tavily, brave, exa, firecrawl, groundroute).
You.com runs its own web index and returns extracted page snippets per
hit rather than a single meta description, so a research loop gets usable
context without a follow-up fetch.
The provider picks its endpoint by whether a key is configured, so it
works out of the box and scales up when the operator adds one:
- no key -> /v1/agents/search, keyless free tier
- YDC_API_KEY -> /v1/search with X-API-Key, full Search API
Endpoint and headers are always chosen together; the keyless endpoint
rejects an auth header, so a stale key can never be sent there. Both
return the same body shape, so one normalizer covers both.
httpx only, no new dependency and no pyproject extra, so uv.lock is
untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(community/youcom): review follow-ups — drop stray .pr-body.md, guard OverflowError, add 429 hint
* fix(doctor): make the You.com free-provider detail true with or without YDC_API_KEY
The doctor detail returned before any key check, so operators who had
set YDC_API_KEY still saw 'set YDC_API_KEY for the full Search API'.
Reworded to match the sibling free-provider entries and stay accurate in
both states.
* fix(search): sweep youcom max_results onto the shared owner
The single-owner gate that landed with #6563 flags this branch's private
_coerce_max_results as a newly hand-rolled copy, so backend-unit-tests
now fails after merging main. Follow the coordination note from review:
delegate to deerflow.community.search_max_results.coerce_max_results,
which owns the rejection profile (booleans, non-integral floats,
OverflowError), and keep only the 1-100 mirror of You.com's server-side
count clamp on top — clamping policy stays per-provider.
The .inf regression test now goes through the public tool path and also
pins the bool and non-integral-float shapes of the shared bar.
* docs(tools): use the bare you.com platform link in config and docs
Match every other provider entry: no other provider in config.example.yaml
or the en/zh tools.mdx uses campaign-tagged URLs, so drop the utm params
from the three you.com/platform links. Integration attribution stays in
the identifying User-Agent sent to You.com hosts.
* test(youcom): cover shared max_results ownership and bounds
Register You.com among the shared-owner providers and assert default handling and the 100-result cap through outbound request parameters.
---------
Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: valueadd-lgtm <339269770+valueadd-lgtm@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scripts): reject local BIND_HOST values nginx would misread
nginx-local-conf.sh checked only for address and hostname characters.
BIND_HOST=0 rendered `listen 0:2026;`, which nginx resolves to 0.0.0.0 and
binds on every IPv4 interface, and 1.2.3 bound 1.2.0.3. Values nginx rejects
outright ([[fd00::1]], -foo) passed the serve.sh preflight and failed only
after stop_all had torn down the running stack.
Accept exactly a dotted-quad IPv4 address, an IPv6 address (brackets
allowed around IPv6 only), or an RFC 1123 hostname. Returning to the
loopback default removes a temp/nginx.local.conf rendered for an earlier
value, and stop_all now says why it can quit a rendered-config nginx
through the tracked config; a test pins the shared pid directive.
* docs(changelog): reference #6635
* docs(changelog): match the zh reference spacing
* fix(scripts): check every BIND_HOST hostname character before splitting labels
read -r -a stops at the first newline, so is_hostname validated only the
first line of a value like $'localhost\nbad' and the rest reached the
rendered listen directive. Check the allowed characters across the whole
value first, and cover multi-line, CR and tab values for each validator.
* feat(dev): add a two-Gateway multi-instance harness script
There was no way to run the deployment.multi_instance topology locally:
scripts/serve.sh pins one Gateway to port 8001, the local nginx config has a
single upstream, and getting two processes past the multi-instance startup gate
needs Postgres, Redis, a shared DEER_FLOW_HOME, shared auth secrets and a
per-process DeerMem retrieval index by hand.
scripts/dev_multi_instance.sh (make dev-multi / dev-multi-check /
dev-multi-down) starts throwaway Postgres and Redis containers on loopback,
renders a config from the developer's config.yaml (or config.example.yaml),
and runs Gateway A on 8001 and Gateway B on 8011 with one shared home, one
copy of extensions_config.json and generated AUTH_JWT_SECRET,
DEER_FLOW_INTERNAL_AUTH_TOKEN and DEER_FLOW_CREDENTIALS_KEY values, plus an
optional round-robin nginx on 2027. B starts after A answers /health: Alembic
migrations are advisory-locked, but the LangGraph checkpointer/store setup()
that follows is not.
scripts/dev_multi_instance.py holds the testable parts: the config overlay
(neutralizing and reporting settings the gate refuses), the secrets file, the
nginx config, and `check`, which automates the scriptable cross-instance
acceptance checks: readiness with stream_bridge ok, A-issued sessions and the
internal token accepted by B, thread/upload/artifact visibility, a skill
toggle seen by B, nginx balancing, and SSE join on B with Last-Event-ID resume
on A. Tests pin that the rendered config passes the real
_enforce_postgres_for_multi_worker gate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link the two-Gateway harness entries to #6613
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(dev): keep the multi-instance harness inside its own state
Address review findings on the two-Gateway harness (#6613):
- Containers carry a deerflow.harness.state-dir label; up/down remove only
containers of their own state dir and refuse those of another checkout or
DEERFLOW_MI_STATE_DIR instead of force-removing them.
- Containers are removed with `docker rm -f -v`, so the anonymous volumes
the postgres and redis images declare no longer leak on every up/down.
- `up` installs an EXIT trap once it starts containers, so any later failure
(including B's port being taken after preflight) tears down A and both
containers; start_gateway returns failures instead of exiting the script,
and the Gateway is disowned so stopping it prints no job notice.
- The overlay removes any checkpointer section (a postgres one would write
checkpoints and the Store into the developer's database), points an
explicit sandbox.ownership and a Redis checkpoint_cache at the harness
Redis (keeping ownership timing), drops DeerMem storage_path (both
spellings) and the blob_storage root so data lands in the harness home,
gives a local AIO sandbox a state-dir-specific container_prefix, and
reports a non-DeerMem memory backend it keeps sharing.
- nginx paths are rendered as quoted string literals ('$' and control
characters are refused), so a state dir with spaces works.
- `check` skips the run-based SSE check when model discovery did not
succeed, unless --with-llm, and enforces the SSE time budget per raw line
so heartbeat-only traffic cannot hold it open.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(scripts): keep the harness guidance under the AGENTS.md soft limit
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): add DEER_FLOW_CREDENTIALS_KEY for at-rest credential encryption
There was no at-rest encryption key. ChannelCredentialCipher had no
production caller: the channel service, the scheduled-task notification
outbox and the channel-connections router all built
ChannelConnectionRepository without a cipher, so store_credentials refused
(RuntimeError) and get_credentials always returned None.
- New harness module deerflow.config.credentials_key: env-only
DEER_FLOW_CREDENTIALS_KEY (comma-separated Fernet keys; first encrypts,
all decrypt via MultiFernet), CredentialsCipher with encrypt_text /
decrypt_text / rotate_text, a fernet:v2: prefix with legacy fernet:v1:
values still readable, and get_credentials_cipher() as the process-wide
entry point. Malformed keys are refused without echoing key material.
Without the variable a single instance generates
{base_dir}/.credentials_key. Never derived from AUTH_JWT_SECRET.
- New deerflow.config.secret_file.read_or_create_secret_file: O_EXCL
create (0600) plus read-back with a bounded settle window, shared by
.credentials_key, .jwt_secret (fixes two replicas cold-starting on a
shared volume keeping different JWT secrets) and the managed-model key.
- ChannelCredentialCipher now builds on CredentialsCipher (same
constructors); the Gateway loads the cipher off the event loop at
startup when channel_connections is enabled and passes it to every
ChannelConnectionRepository. Undecryptable values stay "missing + warn".
- Startup gate: a malformed key stops startup in any topology; a declared
multi-instance deployment with channel_connections enabled and no key
is refused. Plain multi-worker shares the generated file and is allowed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(deploy): generate and inject DEER_FLOW_CREDENTIALS_KEY in Helm, compose and deploy.sh
- Helm: the app Secret generates a Fernet key (randBytes 32 translated to
the urlsafe alphabet) and preserves it across upgrades via lookup like
AUTH_JWT_SECRET; the gateway reads it through an optional secretKeyRef
so a user-managed existingAppSecret without it still boots (the Gateway
gate refuses only multi-instance deployments that store credentials).
README, NOTES and values document backup, rotation and existingAppSecret.
- scripts/deploy.sh resolves the key shell -> .env -> persisted file ->
generated, persisting it as $DEER_FLOW_HOME/.credentials_key (0600,
noclobber/O_EXCL with read-back): the same file the Gateway generates in
its runtime home, so either side's key is the one both keep.
- Both compose files pass DEER_FLOW_CREDENTIALS_KEY=${...:-} to the gateway.
- .env.example entry with a generation hint; README and CHANGELOG notes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): make abandoned secret-file replacement single-winner
A rename is last-writer-wins, so replicas that restart together and all
find the same abandoned empty secret file could each replace it and keep
a different value. The process that exclusively creates
<name>.replacing now owns the replacement, re-checks the file is still
empty, and publishes by renaming the claim over it; peers wait for that
value, bounded so a crashed replacer cannot block startup forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link DEER_FLOW_CREDENTIALS_KEY entries to #6611
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): name the setting that turns off a credentials-key consumer
The multi-instance refusal ended with "or disable
channel_connections.enabled=true"; it now says "or set
channel_connections.enabled=false".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): never expose a secret file before its value is written
O_EXCL published the name before its content, so a creator that stalled
mid-write left an empty file a peer replaced after the settle window; the
resumed creator then returned a value nobody persisted, or deleted the
peer's valid file when its own write failed. Creators now write and sync
a private temporary file and hard-link it to the name, which fails if
the name exists; a failed creator only removes its own temporary file.
Filesystems without hard links publish through the same single-winner
<name>.replacing claim that replaces abandoned empty files, so creation
and replacement are serialized there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(models): recover an empty managed-model key when no catalog exists
An interrupted first save could leave an empty key file; every later save
then passed empty bytes to Fernet and the endpoint returned 503 until the
file was removed by hand. An empty key with no catalog encrypted nothing,
so it now goes through the secret-file recovery path; next to an existing
catalog it is still refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(docker): pass the dotenv DEER_FLOW_CREDENTIALS_KEY through make docker-start
scripts/docker.sh runs Compose from docker/ without --env-file, so the dev
gateway's DEER_FLOW_CREDENTIALS_KEY=${DEER_FLOW_CREDENTIALS_KEY:-} entry
interpolated to empty and overrode the key the checkout .env supplied via
env_file. Export it from the dotenv like AUTH_TRUSTED_PROXIES.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Fail closed on delegated fetch backends to close SSRF gap
* fix(community): resolve delegated-fetch credentials from one config snapshot
Address review feedback on the fail-closed SSRF guard:
- firecrawl/fastcrw client construction no longer re-reads get_app_config(),
so the API key and endpoint come from the same revision the endpoint screen
validated (prevents cross-revision credential misrouting).
- fix the browserless test double to accept the new two-argument
_get_browserless_client signature.
- document the check-only (DNS rebinding) caveat on
validate_delegated_backend_url.
- add a CHANGELOG upgrade note for the default self-hosted backend break.
* fix(doctor): consolidate delegated fetch backend checks and deployment guidance
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scripts): bind local make dev and make start to loopback
serve.sh and backend/Makefile started the Gateway with --host 0.0.0.0,
nginx.local.conf listened on every interface, and Next.js kept its
all-interfaces default outside Windows, so a local stack on a LAN or VPN
exposed ports 2026, 8001 and 3000, including /setup before the first
admin existed. The Docker stack and the README already document a
loopback-only default.
Bind the Gateway and frontend to 127.0.0.1 and keep nginx on loopback
unless BIND_HOST is set, the variable the Docker stack already honors.
nginx-local-conf.sh renders a copy listening on BIND_HOST, and serve.sh
resolves it before stopping anything so an invalid value cannot tear
down a running stack.
* docs(changelog): reference #6587
* docs(changelog): separate the loopback entry from the next bullet
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(release): reject a stale backend/uv.lock in the version gate
verify_versions.sh compared only Chart.yaml, pyproject.toml and
package.json, so bumping those three by hand passed the v* gate while
backend/uv.lock still recorded the previous root package version. The
backend image installs with uv sync --locked, so the chart and the other
images published while the backend image failed to build.
Run uv lock --check in backend/ as part of the gate, letting uv own the
PEP 440 normalization, and fail when uv is missing. The reusable
verify-versions workflow installs the uv version the backend image pins.
* docs(changelog): reference #6588
* fix(release): explain an unparsed lock version and stop blaming the lock for every uv failure
Print a fallback when the lock's root entry is not found, so the job log
does not show an empty field, and word the uv lock --check annotation so a
network error or uv crash is not reported as a stale lock.
* fix(auth): key the compose login lockout by client, not by nginx
Failed logins are counted per client IP, and the Gateway honors X-Real-IP
only from a peer listed in AUTH_TRUSTED_PROXIES. The compose files never
set it, and every browser request reaches the Gateway from the nginx
container, so all logins shared nginx's address: five wrong passwords
from anyone locked every user out for five minutes.
AUTH_TRUSTED_PROXIES now also accepts hostnames, resolved off the event
loop and cached for 10 seconds (failed lookups included, and only when
X-Real-IP is present), and both compose files default it to the bundled
nginx service, whose address Docker assigns. nginx overwrites X-Real-IP
with $remote_addr on every Gateway route; a test pins that for the three
maintained configs.
* docs(changelog): reference #6519
* fix(auth): share one trusted-proxy lookup across concurrent cache misses
A burst of setup-status / login requests on an expired hostname entry
started one getaddrinfo per request while DNS was slow, occupying the
default executor that password verification also uses. The cache check
and an in-flight task registration now happen without a yield between
them; later callers join that task through asyncio.shield, so a cancelled
request cannot cancel the lookup the others joined, and the task caches
its own result.
The compose default keeps `:-` on purpose: every browser request arrives
through nginx, so an empty AUTH_TRUSTED_PROXIES would only bring back the
shared lockout. The compose comment and .env.example say so, and the
rendered-compose test now covers the explicitly empty value.
* fix(docker): keep a .env AUTH_TRUSTED_PROXIES under make docker-start
docker.sh start runs Compose from docker/ without --env-file, so the dev
compose file's ${AUTH_TRUSTED_PROXIES:-nginx} interpolated from the shell
only, and the resolved environment entry outranked the operator's value
loaded through env_file. Export it from .env alongside the proxy variables
(a shell export still wins), and strip a CRLF before the quotes.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(deploy): fail fast when DEER_FLOW_HOME is not writable
`make docker-start` bind-mounts the host `backend/` directory into the gateway
container, whose process runs as root, so any state it creates under
`backend/.deer-flow` ends up owned by root on the host. A later `make up` runs
`scripts/deploy.sh` as the invoking user and only fails deep into the run with a
bare "Permission denied" while persisting generated secrets (deploy.sh:234).
Check writability right after `mkdir -p "$DEER_FLOW_HOME"` and, when it fails,
print the exact `chown` recovery command instead of letting the run get halfway
through. `make down` is excluded so stopping the stack is never blocked.
Fixes#6462
* test(deploy): cover runtime home writability preflight
* fix(deploy): diagnose persisted secret permissions and preserve teardown
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(wizard): add Opper as an LLM provider
Opper (https://opper.ai) is an EU-hosted AI gateway: 700+ models from
50+ providers behind one OpenAI-compatible API and one key. This adds it
to the setup wizard's LLM_PROVIDERS next to OpenRouter, with the same
shape: langchain_openai:ChatOpenAI pointed at
https://api.opper.ai/v3/compat, env var OPPER_API_KEY.
Models use Opper's bare model ids: claude-sonnet-4-6 (default), gpt-5.5,
gemini-3.8-flash and gpt-5.4-mini. Because the GPT ids are bare,
langchain-openai drops the 0.7 temperature for them the same way it does
for direct OpenAI gpt-5 models.
Checked: python -m py_compile scripts/wizard/providers.py, and each
model id against Opper's public model list
(GET https://api.opper.ai/v3/models). Not run locally: make lint and
make test.
Disclosure: Opper is my startup.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* docs: add Opper to the config example and READMEs
Add an Opper model example next to the other OpenAI-compatible
gateways, as requested in review:
- config.example.yaml: commented opper-claude-sonnet-4-6 block after
the other gateway examples (ChatOpenAI + base_url, OPPER_API_KEY)
- README.md and the translated READMEs: the same entry in the manual
model configuration examples, after OpenRouter
- .env.example and backend/docs/CONFIGURATION.md: OPPER_API_KEY, in the
shape of the Novita lines
claude-sonnet-4-6 is a pooled id in Opper's public catalogue
(GET https://api.opper.ai/v3/models). config.example.yaml still parses
with yaml.safe_load; config_version is unchanged because the schema is.
Disclosure: Opper is my startup.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(deploy): let make up start on a checkout without .env files
.env and frontend/.env are gitignored and make up does not create them,
but docker/docker-compose.yaml listed both as short-form env_file entries,
which Compose treats as required. On a fresh clone make up, make down and
make prod-logs all stopped with "env file .../.env not found" (exit 14)
before building or touching anything.
Mark the three entries optional with the long-form path/required syntax
the development compose file already uses. Compose loads them when they
exist and skips them otherwise; an unreadable or non-file .env still
fails. This needs Docker Compose 2.24+, the floor the README already
documents, and deploy.sh's startup-failure hint now names it for older
clients.
deploy.sh deliberately does not seed the files from the examples the way
docker.sh start does: .env.example carries placeholder API keys the
production Gateway would receive, and make config skips files that exist.
* docs(changelog): reference #6370
* test(deploy): gate Compose clients and cover make up
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(make): name what make clean deletes and refuse under a live gateway
`make help` described `make clean` as cleaning up "temporary files", but it
deletes backend/.deer-flow: the local database, users, threads, uploads,
memory, and secrets. Both Docker stacks mount that directory into the
deer-flow-gateway container, and `make stop` only stops local services, so
the data could be deleted under a running Gateway.
Say what is deleted in the help text and before deleting, and refuse while
the deer-flow-gateway container is running. The check runs before `make
stop`, which would otherwise stop the running stack's sandboxes first; a
missing or unreachable Docker passes.
* docs(changelog): reference #6351 for the make clean guard
* fix(make): share make clean's data list and pin stop before the delete
The help line listed "database, threads, uploads, secrets" while the
deletion notice also named users and memory. Define the list once in
RUNTIME_DATA_CONTENTS and use it in both.
Running sandbox containers left behind by `make down` are already stopped
by `make stop` (serve.sh --stop reaps deer-flow-sandbox*), which runs after
the guard and before the delete. Note that in the guard script, and pin
the ordering: the stub stop now records that the data still exists.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): read uvicorn's WEB_CONCURRENCY in the remaining worker-count readers
* fix(gateway): name the worker spelling an operator can act on
The browser refusal and the dedupe warning both took the count but printed
GATEWAY_WORKERS, so on the launches that pass uvicorn no --workers the advice
silences the gate while the extra processes keep starting. Resolve which
variable set the count and print that, and let an unparsable GATEWAY_WORKERS
fall through to WEB_CONCURRENCY instead of reporting a single worker.
Also stop the comments from claiming the readers are equivalent: they share
the two spellings and the blank-means-unset rule, but channel_connections.py
turns an unparsable value into 0 and refuses while these stay inert.
* docs: state the single-worker requirement under both spellings
The browser tool group and the inbound dedupe warning now refuse or warn on
WEB_CONCURRENCY as well, so guidance that names only GATEWAY_WORKERS=1 is
incomplete for the shipped Dockerfile and serve.sh launches, where uvicorn
takes the process count from WEB_CONCURRENCY.
* test(features): isolate the browser capability surface from the shell's worker count
browser_capability() asks for the Gateway worker count, which this change makes
readable from WEB_CONCURRENCY, so an exported count in the invoking shell turned
test_features_reports_browser_control_enabled_when_configured_and_runtime_available
into a failure that has nothing to do with the router.
* test(scripts): cover discord extra auto-detection
The discord mapping in scripts/detect_uv_extras.py worked but was neither
listed in the "currently maps" docstring nor covered by a test, unlike the
buzz mapping that grew up beside it.
* fix(channels): point the Discord missing-dependency hint at the discord extra
discord.py is the only IM transport that is not a core dependency — it ships in
the optional `discord` extra, which detect_uv_extras.py, backend/Dockerfile and
deploy.sh already expand. Its ImportError hint told users to run `uv add
discord.py` instead, which rewrites pyproject.toml/uv.lock in a tree every
restart syncs with `--locked`. Mirror the buzz hint wording and note the extra
in config.example.yaml next to the buzz block.
---------
Co-authored-by: huyengiang101086-spec <270684046+huyengiang101086-spec@users.noreply.github.com>
* fix(config): upgrade the config.yaml the Gateway loads
scripts/config-upgrade.sh resolved DEER_FLOW_CONFIG_PATH, then
backend/config.yaml, then <checkout>/config.yaml. make dev exports
DEER_FLOW_PROJECT_ROOT=<checkout>, so the Gateway reads the checkout copy
first: with both files present the script upgraded a file nothing loads
and left the live config outdated while reporting success. It also
ignored DEER_FLOW_PROJECT_ROOT and a DEER_FLOW_CONFIG_PATH set in .env,
and fell back to another file when DEER_FLOW_CONFIG_PATH named a
missing one.
Ask the harness resolver (AppConfig.resolve_config_path) instead of
copying its order, defaulting DEER_FLOW_PROJECT_ROOT to the checkout as
serve.sh does. A missing DEER_FLOW_CONFIG_PATH or invalid project root
now fails with the Gateway's error; no config anywhere still creates
<checkout>/config.yaml from the example.
* docs(changelog): reference #5991
* test(config): pin config-upgrade seeding a missing config from the example
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(doctor): check the config file the Gateway actually loads
make doctor always inspected <checkout>/config.yaml and ignored
DEER_FLOW_CONFIG_PATH and DEER_FLOW_PROJECT_ROOT. A missing override
that stops the Gateway from starting still reported "config.yaml found"
and "config.yaml loadable", and a valid override pointing elsewhere got
the wrong file checked.
Doctor now resolves the path through the harness's own resolver,
AppConfig.resolve_config_path, after defaulting DEER_FLOW_PROJECT_ROOT
to the checkout as scripts/serve.sh does for make dev. An override the
Gateway would reject fails "config.yaml found" with the Gateway's error
and the config checks skip; an unimportable harness is reported rather
than raised.
* docs(changelog): reference #5987
* fix(doctor): hand the config-location variables over like serve.sh
Review follow-up. serve.sh sources the checkout .env over the shell and
then replaces an unset or empty DEER_FLOW_PROJECT_ROOT with the
checkout. Doctor loaded .env shell-first and only defaulted an unset
root, so a shell DEER_FLOW_CONFIG_PATH / DEER_FLOW_PROJECT_ROOT that
differed from .env, or an empty root, made it check a different file
than the Gateway loads.
For those two variables the .env value now overrides the shell (other
keys keep doctor's shell-first loading), and an empty root becomes the
checkout, before the harness resolver runs.
* fix(doctor): report any harness import failure and expand ~ like source
Review follow-up.
- The harness-import guard caught only ImportError, so a harness that is
present but broken differently (TypeError/SyntaxError while the module
body runs) still crashed doctor. Catch Exception, as
check_config_loadable does, and name the exception type in the report.
- serve.sh's `source .env` expands an unquoted leading `~` in
DEER_FLOW_CONFIG_PATH / DEER_FLOW_PROJECT_ROOT and keeps a quoted one
literal; dotenv_values expands neither. Expand the unquoted form so a
`~/...` path in .env no longer fails a setup that starts, and keep the
quoted form literal so doctor does not pass one the Gateway rejects.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): add Unbrowse web fetch provider
Add a community web_fetch provider backed by Unbrowse
(https://unbrowse.ai), a hosted service that returns a page as markdown,
over plain HTTP when that is enough and through a cloud browser when the
page needs JavaScript. The tool sends one JSON-RPC tools/call POST
(unbrowse.scrape) to Unbrowse's MCP endpoint, so it needs no extra Python
package and no MCP client (uses httpx, already a dependency).
Unbrowse is not a general web search engine, so only web_fetch is added.
Changes:
- backend/packages/harness/deerflow/community/unbrowse/{__init__,tools}.py
web_fetch_tool with the same output shape as the Sofya, Exa and
Firecrawl fetch tools (# <title> plus markdown, 4KB cap). API key is read
from the config.yaml api_key field or the UNBROWSE_API_KEY env var.
Optional render: auto|never|always; unknown values log a warning and fall
back to auto. JSON-RPC errors, isError tool results, HTTP errors and
malformed payloads come back as "Error: ..." strings.
- backend/tests/test_unbrowse_tools.py: key resolution, render handling,
request shape, structuredContent vs text payloads, error paths.
- scripts/doctor.py, scripts/wizard/providers.py: register the key and the
web_fetch provider; backend/tests/test_doctor.py covers both doctor paths.
- config.example.yaml, .env.example, backend/docs/CONFIGURATION.md,
tools/AGENTS.md, EN/ZH harness/tools.mdx: document the provider.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* docs(changelog): add Unbrowse web_fetch provider entry
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(deploy): honor BETTER_AUTH_SECRET and DEER_FLOW_INTERNAL_AUTH_TOKEN from .env
deploy.sh only checked the shell before reloading a persisted secret or
generating a fresh one and exporting it. Compose interpolation lets shell
variables outrank --env-file, so a secret written to the repo-root .env --
the surface every deployment doc points at -- was silently replaced by the
generated one: sessions were signed with a secret the operator never chose
and Gateway workers outside the stack got 401 with the configured token.
Leave a .env-provided secret unexported so Compose parses it itself
(shell -> .env -> persisted file -> generated), and keep generating when
the shell exports an empty value, since Compose would otherwise pass the
empty string through.
* docs(changelog): reference #5928 in the deploy.sh dotenv secrets entries
* fix(deploy): decide whether .env provides a secret with Compose's own parser
read_dotenv_value only matches KEY=VALUE and returns the literal text, but
Compose also accepts `KEY: VALUE` lines and interpolates `${VAR}` inside
values. So `TOKEN: abc` looked absent (and got shadowed by a generated
token) while `TOKEN=${UNSET}` looked present (and suppressed generation,
leaving the stack with an empty token).
Ask Compose for its resolved interpolation environment instead
(`docker compose config --environment`, same --env-file, stub project on
stdin, project directory docker/) and treat a resolved-empty value as
missing. Clients older than Compose 2.28 lack the flag and fall back to
the plain reader. Tests cover the canned Compose answers, the fallback,
and every dotenv spelling against the real Compose client when installed.
* fix(deploy): resolve .env secrets through Compose on every supported client
`config --environment` needs Compose 2.28, so 2.24–2.27 clients took the
plain KEY=VALUE fallback and kept both defects: `TOKEN: abc` was missed
and shadowed, `TOKEN=${UNSET}` counted as present and left the stack with
an empty token.
Drop the fallback. Render a stub project whose only environment entry is
`${KEY}` through `docker compose config` — same --env-file and project
directory as the real command — and read the interpolated value back;
`""` means empty or unset. This is plain interpolation, available on every
Compose v2, and verified identical on 2.24.0, 2.27.0 and 2.32. A failing
probe stops the script instead of guessing. The real-Compose tests now run
against the docker CLI and any standalone binaries listed in
DEER_FLOW_TEST_COMPOSE_BINARIES.
* fix(dev): keep whole worktree paths when serve.sh builds DEERFLOW_ROOTS
serve.sh collects every deer-flow worktree root so `make stop` / `make dev`
can reclaim the shared dev ports (8001/3000/2026) from a sibling checkout.
It parsed `git worktree list --porcelain` with `awk '{print $2}'`, but that
output does not quote paths, so a worktree at `.../deer flow two` was
recorded as `.../deer`. `_is_deerflow_pid` matches open files against
"<root>/", which the truncated root never satisfies, so a Gateway or
frontend started from that worktree was treated as a foreign process and
the start aborted with "port already in use".
Take the remainder of the line after the `worktree ` key instead. The
dedupe and longest-first ordering that follow are unchanged.
The new test builds a scratch repo with a spaced linked worktree and runs
the DEERFLOW_ROOTS block from serve.sh against it, from both the main
checkout and the spaced one.
* docs(changelog): reference #5856 in the worktree-roots entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(release): refresh backend/uv.lock when bumping the version
`scripts/bump_version.sh` rewrote backend/pyproject.toml,
frontend/package.json and the Helm chart's `version`/`appVersion`, but
`backend/uv.lock` also records the root package version (uv keeps its PEP 440
form, so `2.1.0-rc0` is stored as `2.1.0rc0`). The documented release step left
the lock pinning the previous version, and that is not cosmetic drift:
`uv lock --check` (lint CI) fails on a stale lock and `uv sync --locked`
(e2e / plugin workflows) refuses the tree, so the release commit could not pass
CI. RELEASING.md only mentioned the manual `cd backend && uv lock` step for
release candidates.
The script now refreshes the lock via `uv lock` and refuses to start without
`uv` — it exits before editing anything rather than leaving a half-bumped tree
whose lock CI rejects. `uv lock` stays the authority for the normalized
prerelease form; the lock diff for a bump is a single line. RELEASING.md and
AGENTS.md now describe the lock as part of the bump.
Tests: backend/tests/test_bump_version_lockfile_sync.py (red on main, green
here) pins both halves — the lock moves with the sources, and a missing uv is an
error instead of a silent no-op.
* docs(changelog): record the release bump lockfile fix (#5859)
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(memory): redact queued memory payloads (pii_redaction follow-up)
Closes the remaining vector from #3190 (memory storage): request-scoped
redaction leaves originals in thread state, and MemoryMiddleware queued
that raw conversation into the durable extraction buffer — so the
extraction model's input and the persisted facts carried raw PII even
with pii_redaction.enabled.
- MemoryMiddleware gains pii_redaction_config (same pattern as
summarization / durable-context / title); the payload returned by
_resolve_add_args() is redacted at the enqueue boundary before
add()/aadd(), covering both after_agent and aafter_agent
- Reuses the shared redact_text() seam from #5527 via
active_pii_detectors/_Redactor/_redact_content; the buffered payload
itself becomes clean, not just the extraction model's view
- Wired at the lead-agent assembly site
- Tests: queue-payload redaction enabled/disabled, detector toggles,
message immutability, token stability across turns
* fix(memory): reserve existing placeholder indices across the queued batch
Review round 1 on #5577: thread state mixes raw user turns with
already-redacted tool results and summaries, so a token in a later
message must not collide with a new value in an earlier one. Pre-scan
the whole queued batch with redactor.reserve() before allocating.
Also extend the two MemoryMiddleware stubs in
test_lead_agent_model_resolution.py to accept the new
pii_redaction_config kwarg (CI backend-unit-tests red).
* fix(memory): redact the compaction-triggered memory_flush_hook queue too
Review round 2 on #5577: with memory + summarization enabled,
memory_flush_hook() queues list(event.messages_to_summarize) directly
into MemoryManager.add_nowait(), bypassing MemoryMiddleware entirely —
and compaction removes those messages from state right after, so the
after-agent redaction can never repair a raw batch queued there.
Extract the batch redaction (reserve pre-scan + rebuild) into a shared
redact_queued_messages() used by both MemoryMiddleware's enqueue
boundary and the hook; the hook gains an optional pii_redaction_config
wired via functools.partial at the summarization factory (subagents use
skip_memory_flush=True and are unaffected). Regression covers the hook
path enabled/disabled/absent.
* test(summarization): unwrap the partial in the factory hook identity check
edd916e wrapped memory_flush_hook in functools.partial to carry the
pii_redaction config, but missed committing the matching update to
test_factory_attaches_memory_flush_hook_by_default — a functools.partial
is not equal to its wrapped function, so the identity assertion failed
deterministically (caught by static review on #5577).
* fix(memory): cover tool-call arguments and lenient text blocks in queued payloads
Review round 4 on #5577 — two reproduced gaps still allowed raw PII into
memory payloads with pii_redaction.enabled=true:
- OpenViking retains the full message object, so parsed tool_calls args
and provider-format arguments in additional_kwargs bypassed the
content-only redaction; queued payloads now redact tool-call args
recursively (and reserve them in the pre-scan).
- DeerMem's format_conversation_for_update reads p.get("text") of any
dict block, not only type=="text" — _redact_content now redacts every
string-valued text field so a lenient consumer cannot bypass it.
Both repro shapes are pinned as regressions.
* fix(pii): value-derived placeholders for stable capture identity
Review round 5 on #5577: sequential allocation is order-dependent — the
same unchanged message re-redacted in a later batch can get a different
token, which flips OpenViking's content-signature and replays already
recorded turns into persistent memory sessions. Placeholders are now a
pure function of the matched value (sha256-6 per category): identities
stay stable across batches, enqueues, seams, and backends with no shared
state and no stored mapping.
- _Redactor becomes stateless; the reserve machinery (index pre-scans,
_PLACEHOLDER_PATTERN) is removed along with its call sites
- redact_texts keeps its signature, now a simple per-text redaction
- reserve-semantics tests are superseded by cross-enqueue identity tests
(two successive captures where the second batch carries higher existing
placeholders); the compiled-graph echo regex matches the new format
* fix(pii): 128-bit placeholder digests; redact tool-argument keys
Review round 6 on #5577:
- [P2] The 24-bit truncation collided distinct identities (reproduced:
contact3513@ / contact3727@example.com both rendered [EMAIL_575dbb]).
Retain 128 bits of the digest so separate people stay separate at any
realistic volume; the compiled-graph echo regex and the AGENTS.md entry
follow the new format.
- [P2] _redact_strings redacted values but preserved mapping keys, so
{"contacts": {"alice@example.com": "manager"}} kept the raw email
as a key. Redact string keys along with values; deterministic 128-bit
tokens make distinct-key collisions a hash-collision non-event, and a
redacted key equal to another key merges the entries. Regression pins
the email-keyed arguments shape.
* docs(agents): trim PiiRedactionMiddleware entry; drop stale reserve wording
The chain grew past the 98304-byte AG002 hard limit again (main growth
plus the round-6 format example). Trim the entry to fit, and replace the
stale reserve-indices wording with the value-derived semantics that
replaced it.
* docs(agents): shrink PiiRedactionMiddleware entry below the chain budget
main moved (#5609) and the base comparison flipped the nearly-over chain
back to an AG002 error (98441 > 98304). Trim the entry to the facts that
are not documented elsewhere: seams, allowlist, order, enqueue boundary.
* fix(pii): letters-only value-derived tokens; wire the factory SDK path
Review round 7 on #5577:
- [P2] Hex digests carry digit runs that later digit-anchored detectors
re-scan, corrupting minted tokens into nested placeholders (reproduced:
user280@example.com -> [EMAIL_…[PHONE_…]…], ~1/140 emails). The 128-bit
digest is now encoded over a-z, so a minted token survives the full
pinned detector order.
- [P2] create_deerflow_agent (the SDK path) constructed MemoryMiddleware
and DurableContextMiddleware without pii_redaction_config; the knob is
now an explicit optional parameter threaded through
_assemble_from_features (the factory reads no config files by design,
so SDK callers pass it explicitly).
- [Nit] hoist the Command rebuild comprehension out of the per-message
loop in _redact_result.
- [Nit] drop the dead _iter_tool_calls helper and Iterator import.
* test(factory): pin pii_redaction_config wiring; document the parameter
Review round 8 on #5577:
- [P3] test_create_deerflow_agent.py had zero coverage of the
pii_redaction_config parameter; two regressions now assert that
create_deerflow_agent(pii_redaction_config=cfg) builds a
MemoryMiddleware and a DurableContextMiddleware carrying cfg, and that
the default is None (redaction off).
- [Nit] document the parameter in the create_deerflow_agent docstring:
None leaves redaction off; a middleware= takeover bypasses the
assembly that would apply it.
* fix(pii): HMAC-scoped placeholder tokens; install model-bound redaction in the SDK chain
Review round 10 on #5577:
- [P2] sha256(category||value) is an unkeyed, globally deterministic
fingerprint: a provider or memory backend can confirm candidate values
offline and link identical values across users and threads, contradicting
the documented no-re-identification guarantee. Placeholders are now
HMAC-derived over a deployment-scoped token_secret (new config field);
the docstring documents the linkability scope for both modes.
- [P2] _assemble_from_features() never appended PiiRedactionMiddleware, so
the SDK knob enabled only the downstream seams while the primary model
still received raw content. The middleware is now appended in the same
relative position as the lead-agent base chain when the gate is on,
with a model-call regression through the assembled SDK chain.
* docs(pii): document token_secret; update the placeholder format examples
Follow-up to the round-10 review on #5577:
- config.example.yaml documents the new token_secret knob: a
deployment-scoped secret makes placeholders HMAC-derived and only
linkable within one deployment; leaving it unset keeps publicly
computable fingerprints (documented trade).
- the PiiRedactionConfig class docstring's placeholder example now
matches the shipped format (128-bit HMAC digests over base-26
letters).
* docs(pii): update the module docstring's placeholder format example
The module docstring still showed the sequential [EMAIL_1] example;
align it with the shipped value-derived format (128-bit HMAC digests
over base-26 letters).
* fix(pii): require a non-empty token_secret when redaction is enabled
Review round 11 on #5577: with the secret optional, the default enabled
configuration minted empty-key HMAC digests — publicly computable, globally
linkable fingerprints of the raw values, confirmable by offline guessing for
low-entropy identifiers such as phone numbers.
- PiiRedactionConfig now rejects enabled=true without a non-empty
token_secret (minimum 16 characters); the unkeyed mode is gone entirely.
- The request-scoped redactor in _process_request now carries the configured
key: it built its redactor without it, so user-message placeholders
silently diverged from every other seam for the same value. The empty-key
guard in _placeholder_token is the backstop that keeps every path keyed.
- Module/config docstrings and config.example.yaml no longer describe the
placeholders as irreversible or the secret as optional; the stale
reserve-semantics wording and the "memory path not covered" note are
corrected.
- Regressions: validation errors for missing/blank/short secrets, and the
user-message boundary minting the same keyed token as redact_text.
* fix(memory): redact original_user_content on queued memory payloads
Review round 12 on #5577: UploadsMiddleware preserves the raw user turn in
additional_kwargs.original_user_content, but redact_queued_messages rewrote
only content, tool_calls, and provider-format tool_calls — the queued copy
still carried the raw value, so the memory queue and backend received raw PII
with redaction enabled.
The shared helper now redacts the provenance string field on the queued copy
(same batch redactor, merged with the existing additional_kwargs rewrite,
rebuilt via model_copy — thread-state messages stay untouched), covering both
the enqueue boundary and the compaction flush hook. Regressions pin the
redacted provenance field through both paths and the untouched original.
* fix(memory): redact human_input_response on queued memory payloads
Review round 13 on #5577: clarification replies keep the raw user answer in
additional_kwargs.human_input_response.value (DeerMem reads that mapping via
its read_human_input_response mirror), but redact_queued_messages rewrote
only content, tool_calls, and original_user_content — the queued copy still
carried the raw value into the memory queue and backend.
The shared helper now rebuilds that mapping with value redacted (same batch
redactor) while preserving the protocol fields (version/kind/source/
request_id/option_id/response_kind), merged with the existing
additional_kwargs rewrite and rebuilt via model_copy so thread-state
messages stay untouched. Regressions pin the redacted value, preserved
protocol fields, the untouched original, and the flush-hook path.
* chore(config): bump config_version to 47; generate token_secret on upgrade
Review round 13 on #5577: existing deployments can legitimately sit at
config_version 46 with pii_redaction.enabled: true and no token_secret; the
mandatory-secret validator would fail their startup while config.example.yaml
still said 46, so the upgrade checker considered them current and emitted no
migration warning.
- Bump config_version to 47 so the outdated-config warning fires for v46.
- Add a v47 migration to scripts/config-upgrade.sh: when pii_redaction is
enabled and token_secret is missing/blank, generate a random deployment-
scoped secret (secrets.token_urlsafe) and persist it, so the upgraded
deployment starts under the mandatory validation.
- Regressions: a v46 enabled config is flagged outdated against the real
example; the upgrade run generates a >= 16-char secret and the result
passes AppConfig validation; a disabled config gets no secret.
- Also switch the two v41 script tests from a hardcoded "bash" to
SCRIPT_BASH/skipif, matching the documented Windows constraint and the
neighboring tests (hardcoded bash resolves to the WSL stub there).
* chore(helm): sync chart config_version to 47
Review round 14 on #5577: check_config_version.sh fails whenever the chart's
embedded config_version lags config.example.yaml, and the chart workflow is
triggered by config.example.yaml — so the v47 bump left this head failing
that check. Update the values.yaml config block and the README example to
47; scripts/check_config_version.sh now passes.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
