* feat(channels): add channels.leader_lease and gate it under multi-process signals
IM channels other than GitHub connect outbound to their platform. With more
than one Gateway instance each one opens that connection with the same bot
credentials: Telegram rejects the second poller with 409, Discord answers every
message twice and per-process ChannelManager state diverges. The multi-process
startup gate said nothing about channels; the only guard was a comment in the
Helm values.
Add the `channels.leader_lease` section (`enabled: auto|true|false`,
`lease_seconds`, `grace_seconds`; startup-only, registered in the reload
boundary) that the per-provider leader lease will read, validated by a Pydantic
model from the non-schema `channels` mapping. The gate now refuses
`enabled: false` under a multi-process signal while a non-GitHub provider is
enabled under `channels.*` or connectable through `channel_connections`, naming
the providers and the rollback; a malformed block is refused on every topology.
`auto` follows the multi-process signal, so a single instance is unchanged.
ChannelService strips the block before its per-provider loops and exposes it
as `leader_lease`; `ensure_ready_channels` only visits registered providers.
The two-Gateway harness overlay leaves the block alone. config_version 60 in
the example, the Helm chart and its README; docs replace the comment-only
"keep 1 replica" guidance with the gate.
* docs(changelog): link #6746
* fix(channels): stop recommending the unimplemented leader lease as a remedy
Review on #6746: the refusal told operators to switch channels.leader_lease
from false to auto/true "so one instance owns each provider's connection", but
the transport arbitration ships separately, so that change only bypassed the
clause and every worker still started the same bot.
The refusal now offers only the remedies that work today (a single instance, or
disabling the channels) and states that auto/true are reserved for the lease
that ships separately. A regression helper in the gate tests asserts the
remedies start with the rollback and that auto/true are never offered. The
config example, Helm values/README, CONFIGURATION.md, README, the deployment
guide (en/zh), the model docstrings and both changelogs describe auto/true as
reserved settings rather than an ownership guarantee.
* fix(channels): warn on an enabled unknown channel key; reword the changelog entry
Review on #6746: the registry guard in ensure_ready_channels skipped a
misspelled provider key (channels.telegam) silently, where it previously
reached _start_channel and logged 'Unknown channel type'. An enabled dict
block that is neither a registered provider nor leader_lease now logs a
warning that lists the registry.
The changelog entry claimed the chart and docs drop the "keep 1 replica"
guidance; they keep it (auto/true do not make several replicas safe yet) and
only stop relying on the comment alone.
* fix(dev): disable UI channel providers in multi-instance overlay
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(extensions): add example guide and packaging checks
* fix(extensions): align packaging checks with installer
Reject Windows drive-qualified wheel paths before extraction and reuse the installer entry-point validator. Cover unsafe paths, dotted attributes, and valid dotted modules in regression tests.
* fix(extensions): skip unused wheel extraction
Only extract wheels when browser manifests need validation, with a backend-only regression test. Link the three remaining example READMEs in the developer guide.
* docs(frontend): trim guidance intro below size budget
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): add You.com web search provider
Adds You.com as an opt-in community `web_search` engine, mirroring the
existing community engines (tavily, brave, exa, firecrawl, groundroute).
You.com runs its own web index and returns extracted page snippets per
hit rather than a single meta description, so a research loop gets usable
context without a follow-up fetch.
The provider picks its endpoint by whether a key is configured, so it
works out of the box and scales up when the operator adds one:
- no key -> /v1/agents/search, keyless free tier
- YDC_API_KEY -> /v1/search with X-API-Key, full Search API
Endpoint and headers are always chosen together; the keyless endpoint
rejects an auth header, so a stale key can never be sent there. Both
return the same body shape, so one normalizer covers both.
httpx only, no new dependency and no pyproject extra, so uv.lock is
untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(community/youcom): review follow-ups — drop stray .pr-body.md, guard OverflowError, add 429 hint
* fix(doctor): make the You.com free-provider detail true with or without YDC_API_KEY
The doctor detail returned before any key check, so operators who had
set YDC_API_KEY still saw 'set YDC_API_KEY for the full Search API'.
Reworded to match the sibling free-provider entries and stay accurate in
both states.
* fix(search): sweep youcom max_results onto the shared owner
The single-owner gate that landed with #6563 flags this branch's private
_coerce_max_results as a newly hand-rolled copy, so backend-unit-tests
now fails after merging main. Follow the coordination note from review:
delegate to deerflow.community.search_max_results.coerce_max_results,
which owns the rejection profile (booleans, non-integral floats,
OverflowError), and keep only the 1-100 mirror of You.com's server-side
count clamp on top — clamping policy stays per-provider.
The .inf regression test now goes through the public tool path and also
pins the bool and non-integral-float shapes of the shared bar.
* docs(tools): use the bare you.com platform link in config and docs
Match every other provider entry: no other provider in config.example.yaml
or the en/zh tools.mdx uses campaign-tagged URLs, so drop the utm params
from the three you.com/platform links. Integration attribution stays in
the identifying User-Agent sent to You.com hosts.
* test(youcom): cover shared max_results ownership and bounds
Register You.com among the shared-owner providers and assert default handling and the 100-result cap through outbound request parameters.
---------
Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: valueadd-lgtm <339269770+valueadd-lgtm@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scripts): reject local BIND_HOST values nginx would misread
nginx-local-conf.sh checked only for address and hostname characters.
BIND_HOST=0 rendered `listen 0:2026;`, which nginx resolves to 0.0.0.0 and
binds on every IPv4 interface, and 1.2.3 bound 1.2.0.3. Values nginx rejects
outright ([[fd00::1]], -foo) passed the serve.sh preflight and failed only
after stop_all had torn down the running stack.
Accept exactly a dotted-quad IPv4 address, an IPv6 address (brackets
allowed around IPv6 only), or an RFC 1123 hostname. Returning to the
loopback default removes a temp/nginx.local.conf rendered for an earlier
value, and stop_all now says why it can quit a rendered-config nginx
through the tracked config; a test pins the shared pid directive.
* docs(changelog): reference #6635
* docs(changelog): match the zh reference spacing
* fix(scripts): check every BIND_HOST hostname character before splitting labels
read -r -a stops at the first newline, so is_hostname validated only the
first line of a value like $'localhost\nbad' and the rest reached the
rendered listen directive. Check the allowed characters across the whole
value first, and cover multi-line, CR and tab values for each validator.
* feat(dev): add a two-Gateway multi-instance harness script
There was no way to run the deployment.multi_instance topology locally:
scripts/serve.sh pins one Gateway to port 8001, the local nginx config has a
single upstream, and getting two processes past the multi-instance startup gate
needs Postgres, Redis, a shared DEER_FLOW_HOME, shared auth secrets and a
per-process DeerMem retrieval index by hand.
scripts/dev_multi_instance.sh (make dev-multi / dev-multi-check /
dev-multi-down) starts throwaway Postgres and Redis containers on loopback,
renders a config from the developer's config.yaml (or config.example.yaml),
and runs Gateway A on 8001 and Gateway B on 8011 with one shared home, one
copy of extensions_config.json and generated AUTH_JWT_SECRET,
DEER_FLOW_INTERNAL_AUTH_TOKEN and DEER_FLOW_CREDENTIALS_KEY values, plus an
optional round-robin nginx on 2027. B starts after A answers /health: Alembic
migrations are advisory-locked, but the LangGraph checkpointer/store setup()
that follows is not.
scripts/dev_multi_instance.py holds the testable parts: the config overlay
(neutralizing and reporting settings the gate refuses), the secrets file, the
nginx config, and `check`, which automates the scriptable cross-instance
acceptance checks: readiness with stream_bridge ok, A-issued sessions and the
internal token accepted by B, thread/upload/artifact visibility, a skill
toggle seen by B, nginx balancing, and SSE join on B with Last-Event-ID resume
on A. Tests pin that the rendered config passes the real
_enforce_postgres_for_multi_worker gate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link the two-Gateway harness entries to #6613
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(dev): keep the multi-instance harness inside its own state
Address review findings on the two-Gateway harness (#6613):
- Containers carry a deerflow.harness.state-dir label; up/down remove only
containers of their own state dir and refuse those of another checkout or
DEERFLOW_MI_STATE_DIR instead of force-removing them.
- Containers are removed with `docker rm -f -v`, so the anonymous volumes
the postgres and redis images declare no longer leak on every up/down.
- `up` installs an EXIT trap once it starts containers, so any later failure
(including B's port being taken after preflight) tears down A and both
containers; start_gateway returns failures instead of exiting the script,
and the Gateway is disowned so stopping it prints no job notice.
- The overlay removes any checkpointer section (a postgres one would write
checkpoints and the Store into the developer's database), points an
explicit sandbox.ownership and a Redis checkpoint_cache at the harness
Redis (keeping ownership timing), drops DeerMem storage_path (both
spellings) and the blob_storage root so data lands in the harness home,
gives a local AIO sandbox a state-dir-specific container_prefix, and
reports a non-DeerMem memory backend it keeps sharing.
- nginx paths are rendered as quoted string literals ('$' and control
characters are refused), so a state dir with spaces works.
- `check` skips the run-based SSE check when model discovery did not
succeed, unless --with-llm, and enforces the SSE time budget per raw line
so heartbeat-only traffic cannot hold it open.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(scripts): keep the harness guidance under the AGENTS.md soft limit
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): add DEER_FLOW_CREDENTIALS_KEY for at-rest credential encryption
There was no at-rest encryption key. ChannelCredentialCipher had no
production caller: the channel service, the scheduled-task notification
outbox and the channel-connections router all built
ChannelConnectionRepository without a cipher, so store_credentials refused
(RuntimeError) and get_credentials always returned None.
- New harness module deerflow.config.credentials_key: env-only
DEER_FLOW_CREDENTIALS_KEY (comma-separated Fernet keys; first encrypts,
all decrypt via MultiFernet), CredentialsCipher with encrypt_text /
decrypt_text / rotate_text, a fernet:v2: prefix with legacy fernet:v1:
values still readable, and get_credentials_cipher() as the process-wide
entry point. Malformed keys are refused without echoing key material.
Without the variable a single instance generates
{base_dir}/.credentials_key. Never derived from AUTH_JWT_SECRET.
- New deerflow.config.secret_file.read_or_create_secret_file: O_EXCL
create (0600) plus read-back with a bounded settle window, shared by
.credentials_key, .jwt_secret (fixes two replicas cold-starting on a
shared volume keeping different JWT secrets) and the managed-model key.
- ChannelCredentialCipher now builds on CredentialsCipher (same
constructors); the Gateway loads the cipher off the event loop at
startup when channel_connections is enabled and passes it to every
ChannelConnectionRepository. Undecryptable values stay "missing + warn".
- Startup gate: a malformed key stops startup in any topology; a declared
multi-instance deployment with channel_connections enabled and no key
is refused. Plain multi-worker shares the generated file and is allowed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(deploy): generate and inject DEER_FLOW_CREDENTIALS_KEY in Helm, compose and deploy.sh
- Helm: the app Secret generates a Fernet key (randBytes 32 translated to
the urlsafe alphabet) and preserves it across upgrades via lookup like
AUTH_JWT_SECRET; the gateway reads it through an optional secretKeyRef
so a user-managed existingAppSecret without it still boots (the Gateway
gate refuses only multi-instance deployments that store credentials).
README, NOTES and values document backup, rotation and existingAppSecret.
- scripts/deploy.sh resolves the key shell -> .env -> persisted file ->
generated, persisting it as $DEER_FLOW_HOME/.credentials_key (0600,
noclobber/O_EXCL with read-back): the same file the Gateway generates in
its runtime home, so either side's key is the one both keep.
- Both compose files pass DEER_FLOW_CREDENTIALS_KEY=${...:-} to the gateway.
- .env.example entry with a generation hint; README and CHANGELOG notes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): make abandoned secret-file replacement single-winner
A rename is last-writer-wins, so replicas that restart together and all
find the same abandoned empty secret file could each replace it and keep
a different value. The process that exclusively creates
<name>.replacing now owns the replacement, re-checks the file is still
empty, and publishes by renaming the claim over it; peers wait for that
value, bounded so a crashed replacer cannot block startup forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link DEER_FLOW_CREDENTIALS_KEY entries to #6611
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): name the setting that turns off a credentials-key consumer
The multi-instance refusal ended with "or disable
channel_connections.enabled=true"; it now says "or set
channel_connections.enabled=false".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): never expose a secret file before its value is written
O_EXCL published the name before its content, so a creator that stalled
mid-write left an empty file a peer replaced after the settle window; the
resumed creator then returned a value nobody persisted, or deleted the
peer's valid file when its own write failed. Creators now write and sync
a private temporary file and hard-link it to the name, which fails if
the name exists; a failed creator only removes its own temporary file.
Filesystems without hard links publish through the same single-winner
<name>.replacing claim that replaces abandoned empty files, so creation
and replacement are serialized there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(models): recover an empty managed-model key when no catalog exists
An interrupted first save could leave an empty key file; every later save
then passed empty bytes to Fernet and the endpoint returned 503 until the
file was removed by hand. An empty key with no catalog encrypted nothing,
so it now goes through the secret-file recovery path; next to an existing
catalog it is still refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(docker): pass the dotenv DEER_FLOW_CREDENTIALS_KEY through make docker-start
scripts/docker.sh runs Compose from docker/ without --env-file, so the dev
gateway's DEER_FLOW_CREDENTIALS_KEY=${DEER_FLOW_CREDENTIALS_KEY:-} entry
interpolated to empty and overrode the key the checkout .env supplied via
env_file. Export it from the dotenv like AUTH_TRUSTED_PROXIES.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Fail closed on delegated fetch backends to close SSRF gap
* fix(community): resolve delegated-fetch credentials from one config snapshot
Address review feedback on the fail-closed SSRF guard:
- firecrawl/fastcrw client construction no longer re-reads get_app_config(),
so the API key and endpoint come from the same revision the endpoint screen
validated (prevents cross-revision credential misrouting).
- fix the browserless test double to accept the new two-argument
_get_browserless_client signature.
- document the check-only (DNS rebinding) caveat on
validate_delegated_backend_url.
- add a CHANGELOG upgrade note for the default self-hosted backend break.
* fix(doctor): consolidate delegated fetch backend checks and deployment guidance
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scripts): bind local make dev and make start to loopback
serve.sh and backend/Makefile started the Gateway with --host 0.0.0.0,
nginx.local.conf listened on every interface, and Next.js kept its
all-interfaces default outside Windows, so a local stack on a LAN or VPN
exposed ports 2026, 8001 and 3000, including /setup before the first
admin existed. The Docker stack and the README already document a
loopback-only default.
Bind the Gateway and frontend to 127.0.0.1 and keep nginx on loopback
unless BIND_HOST is set, the variable the Docker stack already honors.
nginx-local-conf.sh renders a copy listening on BIND_HOST, and serve.sh
resolves it before stopping anything so an invalid value cannot tear
down a running stack.
* docs(changelog): reference #6587
* docs(changelog): separate the loopback entry from the next bullet
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(release): reject a stale backend/uv.lock in the version gate
verify_versions.sh compared only Chart.yaml, pyproject.toml and
package.json, so bumping those three by hand passed the v* gate while
backend/uv.lock still recorded the previous root package version. The
backend image installs with uv sync --locked, so the chart and the other
images published while the backend image failed to build.
Run uv lock --check in backend/ as part of the gate, letting uv own the
PEP 440 normalization, and fail when uv is missing. The reusable
verify-versions workflow installs the uv version the backend image pins.
* docs(changelog): reference #6588
* fix(release): explain an unparsed lock version and stop blaming the lock for every uv failure
Print a fallback when the lock's root entry is not found, so the job log
does not show an empty field, and word the uv lock --check annotation so a
network error or uv crash is not reported as a stale lock.
* fix(auth): key the compose login lockout by client, not by nginx
Failed logins are counted per client IP, and the Gateway honors X-Real-IP
only from a peer listed in AUTH_TRUSTED_PROXIES. The compose files never
set it, and every browser request reaches the Gateway from the nginx
container, so all logins shared nginx's address: five wrong passwords
from anyone locked every user out for five minutes.
AUTH_TRUSTED_PROXIES now also accepts hostnames, resolved off the event
loop and cached for 10 seconds (failed lookups included, and only when
X-Real-IP is present), and both compose files default it to the bundled
nginx service, whose address Docker assigns. nginx overwrites X-Real-IP
with $remote_addr on every Gateway route; a test pins that for the three
maintained configs.
* docs(changelog): reference #6519
* fix(auth): share one trusted-proxy lookup across concurrent cache misses
A burst of setup-status / login requests on an expired hostname entry
started one getaddrinfo per request while DNS was slow, occupying the
default executor that password verification also uses. The cache check
and an in-flight task registration now happen without a yield between
them; later callers join that task through asyncio.shield, so a cancelled
request cannot cancel the lookup the others joined, and the task caches
its own result.
The compose default keeps `:-` on purpose: every browser request arrives
through nginx, so an empty AUTH_TRUSTED_PROXIES would only bring back the
shared lockout. The compose comment and .env.example say so, and the
rendered-compose test now covers the explicitly empty value.
* fix(docker): keep a .env AUTH_TRUSTED_PROXIES under make docker-start
docker.sh start runs Compose from docker/ without --env-file, so the dev
compose file's ${AUTH_TRUSTED_PROXIES:-nginx} interpolated from the shell
only, and the resolved environment entry outranked the operator's value
loaded through env_file. Export it from .env alongside the proxy variables
(a shell export still wins), and strip a CRLF before the quotes.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(deploy): fail fast when DEER_FLOW_HOME is not writable
`make docker-start` bind-mounts the host `backend/` directory into the gateway
container, whose process runs as root, so any state it creates under
`backend/.deer-flow` ends up owned by root on the host. A later `make up` runs
`scripts/deploy.sh` as the invoking user and only fails deep into the run with a
bare "Permission denied" while persisting generated secrets (deploy.sh:234).
Check writability right after `mkdir -p "$DEER_FLOW_HOME"` and, when it fails,
print the exact `chown` recovery command instead of letting the run get halfway
through. `make down` is excluded so stopping the stack is never blocked.
Fixes#6462
* test(deploy): cover runtime home writability preflight
* fix(deploy): diagnose persisted secret permissions and preserve teardown
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(wizard): add Opper as an LLM provider
Opper (https://opper.ai) is an EU-hosted AI gateway: 700+ models from
50+ providers behind one OpenAI-compatible API and one key. This adds it
to the setup wizard's LLM_PROVIDERS next to OpenRouter, with the same
shape: langchain_openai:ChatOpenAI pointed at
https://api.opper.ai/v3/compat, env var OPPER_API_KEY.
Models use Opper's bare model ids: claude-sonnet-4-6 (default), gpt-5.5,
gemini-3.8-flash and gpt-5.4-mini. Because the GPT ids are bare,
langchain-openai drops the 0.7 temperature for them the same way it does
for direct OpenAI gpt-5 models.
Checked: python -m py_compile scripts/wizard/providers.py, and each
model id against Opper's public model list
(GET https://api.opper.ai/v3/models). Not run locally: make lint and
make test.
Disclosure: Opper is my startup.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* docs: add Opper to the config example and READMEs
Add an Opper model example next to the other OpenAI-compatible
gateways, as requested in review:
- config.example.yaml: commented opper-claude-sonnet-4-6 block after
the other gateway examples (ChatOpenAI + base_url, OPPER_API_KEY)
- README.md and the translated READMEs: the same entry in the manual
model configuration examples, after OpenRouter
- .env.example and backend/docs/CONFIGURATION.md: OPPER_API_KEY, in the
shape of the Novita lines
claude-sonnet-4-6 is a pooled id in Opper's public catalogue
(GET https://api.opper.ai/v3/models). config.example.yaml still parses
with yaml.safe_load; config_version is unchanged because the schema is.
Disclosure: Opper is my startup.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(deploy): let make up start on a checkout without .env files
.env and frontend/.env are gitignored and make up does not create them,
but docker/docker-compose.yaml listed both as short-form env_file entries,
which Compose treats as required. On a fresh clone make up, make down and
make prod-logs all stopped with "env file .../.env not found" (exit 14)
before building or touching anything.
Mark the three entries optional with the long-form path/required syntax
the development compose file already uses. Compose loads them when they
exist and skips them otherwise; an unreadable or non-file .env still
fails. This needs Docker Compose 2.24+, the floor the README already
documents, and deploy.sh's startup-failure hint now names it for older
clients.
deploy.sh deliberately does not seed the files from the examples the way
docker.sh start does: .env.example carries placeholder API keys the
production Gateway would receive, and make config skips files that exist.
* docs(changelog): reference #6370
* test(deploy): gate Compose clients and cover make up
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(make): name what make clean deletes and refuse under a live gateway
`make help` described `make clean` as cleaning up "temporary files", but it
deletes backend/.deer-flow: the local database, users, threads, uploads,
memory, and secrets. Both Docker stacks mount that directory into the
deer-flow-gateway container, and `make stop` only stops local services, so
the data could be deleted under a running Gateway.
Say what is deleted in the help text and before deleting, and refuse while
the deer-flow-gateway container is running. The check runs before `make
stop`, which would otherwise stop the running stack's sandboxes first; a
missing or unreachable Docker passes.
* docs(changelog): reference #6351 for the make clean guard
* fix(make): share make clean's data list and pin stop before the delete
The help line listed "database, threads, uploads, secrets" while the
deletion notice also named users and memory. Define the list once in
RUNTIME_DATA_CONTENTS and use it in both.
Running sandbox containers left behind by `make down` are already stopped
by `make stop` (serve.sh --stop reaps deer-flow-sandbox*), which runs after
the guard and before the delete. Note that in the guard script, and pin
the ordering: the stub stop now records that the data still exists.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): read uvicorn's WEB_CONCURRENCY in the remaining worker-count readers
* fix(gateway): name the worker spelling an operator can act on
The browser refusal and the dedupe warning both took the count but printed
GATEWAY_WORKERS, so on the launches that pass uvicorn no --workers the advice
silences the gate while the extra processes keep starting. Resolve which
variable set the count and print that, and let an unparsable GATEWAY_WORKERS
fall through to WEB_CONCURRENCY instead of reporting a single worker.
Also stop the comments from claiming the readers are equivalent: they share
the two spellings and the blank-means-unset rule, but channel_connections.py
turns an unparsable value into 0 and refuses while these stay inert.
* docs: state the single-worker requirement under both spellings
The browser tool group and the inbound dedupe warning now refuse or warn on
WEB_CONCURRENCY as well, so guidance that names only GATEWAY_WORKERS=1 is
incomplete for the shipped Dockerfile and serve.sh launches, where uvicorn
takes the process count from WEB_CONCURRENCY.
* test(features): isolate the browser capability surface from the shell's worker count
browser_capability() asks for the Gateway worker count, which this change makes
readable from WEB_CONCURRENCY, so an exported count in the invoking shell turned
test_features_reports_browser_control_enabled_when_configured_and_runtime_available
into a failure that has nothing to do with the router.
* test(scripts): cover discord extra auto-detection
The discord mapping in scripts/detect_uv_extras.py worked but was neither
listed in the "currently maps" docstring nor covered by a test, unlike the
buzz mapping that grew up beside it.
* fix(channels): point the Discord missing-dependency hint at the discord extra
discord.py is the only IM transport that is not a core dependency — it ships in
the optional `discord` extra, which detect_uv_extras.py, backend/Dockerfile and
deploy.sh already expand. Its ImportError hint told users to run `uv add
discord.py` instead, which rewrites pyproject.toml/uv.lock in a tree every
restart syncs with `--locked`. Mirror the buzz hint wording and note the extra
in config.example.yaml next to the buzz block.
---------
Co-authored-by: huyengiang101086-spec <270684046+huyengiang101086-spec@users.noreply.github.com>
* fix(config): upgrade the config.yaml the Gateway loads
scripts/config-upgrade.sh resolved DEER_FLOW_CONFIG_PATH, then
backend/config.yaml, then <checkout>/config.yaml. make dev exports
DEER_FLOW_PROJECT_ROOT=<checkout>, so the Gateway reads the checkout copy
first: with both files present the script upgraded a file nothing loads
and left the live config outdated while reporting success. It also
ignored DEER_FLOW_PROJECT_ROOT and a DEER_FLOW_CONFIG_PATH set in .env,
and fell back to another file when DEER_FLOW_CONFIG_PATH named a
missing one.
Ask the harness resolver (AppConfig.resolve_config_path) instead of
copying its order, defaulting DEER_FLOW_PROJECT_ROOT to the checkout as
serve.sh does. A missing DEER_FLOW_CONFIG_PATH or invalid project root
now fails with the Gateway's error; no config anywhere still creates
<checkout>/config.yaml from the example.
* docs(changelog): reference #5991
* test(config): pin config-upgrade seeding a missing config from the example
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(doctor): check the config file the Gateway actually loads
make doctor always inspected <checkout>/config.yaml and ignored
DEER_FLOW_CONFIG_PATH and DEER_FLOW_PROJECT_ROOT. A missing override
that stops the Gateway from starting still reported "config.yaml found"
and "config.yaml loadable", and a valid override pointing elsewhere got
the wrong file checked.
Doctor now resolves the path through the harness's own resolver,
AppConfig.resolve_config_path, after defaulting DEER_FLOW_PROJECT_ROOT
to the checkout as scripts/serve.sh does for make dev. An override the
Gateway would reject fails "config.yaml found" with the Gateway's error
and the config checks skip; an unimportable harness is reported rather
than raised.
* docs(changelog): reference #5987
* fix(doctor): hand the config-location variables over like serve.sh
Review follow-up. serve.sh sources the checkout .env over the shell and
then replaces an unset or empty DEER_FLOW_PROJECT_ROOT with the
checkout. Doctor loaded .env shell-first and only defaulted an unset
root, so a shell DEER_FLOW_CONFIG_PATH / DEER_FLOW_PROJECT_ROOT that
differed from .env, or an empty root, made it check a different file
than the Gateway loads.
For those two variables the .env value now overrides the shell (other
keys keep doctor's shell-first loading), and an empty root becomes the
checkout, before the harness resolver runs.
* fix(doctor): report any harness import failure and expand ~ like source
Review follow-up.
- The harness-import guard caught only ImportError, so a harness that is
present but broken differently (TypeError/SyntaxError while the module
body runs) still crashed doctor. Catch Exception, as
check_config_loadable does, and name the exception type in the report.
- serve.sh's `source .env` expands an unquoted leading `~` in
DEER_FLOW_CONFIG_PATH / DEER_FLOW_PROJECT_ROOT and keeps a quoted one
literal; dotenv_values expands neither. Expand the unquoted form so a
`~/...` path in .env no longer fails a setup that starts, and keep the
quoted form literal so doctor does not pass one the Gateway rejects.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): add Unbrowse web fetch provider
Add a community web_fetch provider backed by Unbrowse
(https://unbrowse.ai), a hosted service that returns a page as markdown,
over plain HTTP when that is enough and through a cloud browser when the
page needs JavaScript. The tool sends one JSON-RPC tools/call POST
(unbrowse.scrape) to Unbrowse's MCP endpoint, so it needs no extra Python
package and no MCP client (uses httpx, already a dependency).
Unbrowse is not a general web search engine, so only web_fetch is added.
Changes:
- backend/packages/harness/deerflow/community/unbrowse/{__init__,tools}.py
web_fetch_tool with the same output shape as the Sofya, Exa and
Firecrawl fetch tools (# <title> plus markdown, 4KB cap). API key is read
from the config.yaml api_key field or the UNBROWSE_API_KEY env var.
Optional render: auto|never|always; unknown values log a warning and fall
back to auto. JSON-RPC errors, isError tool results, HTTP errors and
malformed payloads come back as "Error: ..." strings.
- backend/tests/test_unbrowse_tools.py: key resolution, render handling,
request shape, structuredContent vs text payloads, error paths.
- scripts/doctor.py, scripts/wizard/providers.py: register the key and the
web_fetch provider; backend/tests/test_doctor.py covers both doctor paths.
- config.example.yaml, .env.example, backend/docs/CONFIGURATION.md,
tools/AGENTS.md, EN/ZH harness/tools.mdx: document the provider.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* docs(changelog): add Unbrowse web_fetch provider entry
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(deploy): honor BETTER_AUTH_SECRET and DEER_FLOW_INTERNAL_AUTH_TOKEN from .env
deploy.sh only checked the shell before reloading a persisted secret or
generating a fresh one and exporting it. Compose interpolation lets shell
variables outrank --env-file, so a secret written to the repo-root .env --
the surface every deployment doc points at -- was silently replaced by the
generated one: sessions were signed with a secret the operator never chose
and Gateway workers outside the stack got 401 with the configured token.
Leave a .env-provided secret unexported so Compose parses it itself
(shell -> .env -> persisted file -> generated), and keep generating when
the shell exports an empty value, since Compose would otherwise pass the
empty string through.
* docs(changelog): reference #5928 in the deploy.sh dotenv secrets entries
* fix(deploy): decide whether .env provides a secret with Compose's own parser
read_dotenv_value only matches KEY=VALUE and returns the literal text, but
Compose also accepts `KEY: VALUE` lines and interpolates `${VAR}` inside
values. So `TOKEN: abc` looked absent (and got shadowed by a generated
token) while `TOKEN=${UNSET}` looked present (and suppressed generation,
leaving the stack with an empty token).
Ask Compose for its resolved interpolation environment instead
(`docker compose config --environment`, same --env-file, stub project on
stdin, project directory docker/) and treat a resolved-empty value as
missing. Clients older than Compose 2.28 lack the flag and fall back to
the plain reader. Tests cover the canned Compose answers, the fallback,
and every dotenv spelling against the real Compose client when installed.
* fix(deploy): resolve .env secrets through Compose on every supported client
`config --environment` needs Compose 2.28, so 2.24–2.27 clients took the
plain KEY=VALUE fallback and kept both defects: `TOKEN: abc` was missed
and shadowed, `TOKEN=${UNSET}` counted as present and left the stack with
an empty token.
Drop the fallback. Render a stub project whose only environment entry is
`${KEY}` through `docker compose config` — same --env-file and project
directory as the real command — and read the interpolated value back;
`""` means empty or unset. This is plain interpolation, available on every
Compose v2, and verified identical on 2.24.0, 2.27.0 and 2.32. A failing
probe stops the script instead of guessing. The real-Compose tests now run
against the docker CLI and any standalone binaries listed in
DEER_FLOW_TEST_COMPOSE_BINARIES.
* fix(dev): keep whole worktree paths when serve.sh builds DEERFLOW_ROOTS
serve.sh collects every deer-flow worktree root so `make stop` / `make dev`
can reclaim the shared dev ports (8001/3000/2026) from a sibling checkout.
It parsed `git worktree list --porcelain` with `awk '{print $2}'`, but that
output does not quote paths, so a worktree at `.../deer flow two` was
recorded as `.../deer`. `_is_deerflow_pid` matches open files against
"<root>/", which the truncated root never satisfies, so a Gateway or
frontend started from that worktree was treated as a foreign process and
the start aborted with "port already in use".
Take the remainder of the line after the `worktree ` key instead. The
dedupe and longest-first ordering that follow are unchanged.
The new test builds a scratch repo with a spaced linked worktree and runs
the DEERFLOW_ROOTS block from serve.sh against it, from both the main
checkout and the spaced one.
* docs(changelog): reference #5856 in the worktree-roots entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(release): refresh backend/uv.lock when bumping the version
`scripts/bump_version.sh` rewrote backend/pyproject.toml,
frontend/package.json and the Helm chart's `version`/`appVersion`, but
`backend/uv.lock` also records the root package version (uv keeps its PEP 440
form, so `2.1.0-rc0` is stored as `2.1.0rc0`). The documented release step left
the lock pinning the previous version, and that is not cosmetic drift:
`uv lock --check` (lint CI) fails on a stale lock and `uv sync --locked`
(e2e / plugin workflows) refuses the tree, so the release commit could not pass
CI. RELEASING.md only mentioned the manual `cd backend && uv lock` step for
release candidates.
The script now refreshes the lock via `uv lock` and refuses to start without
`uv` — it exits before editing anything rather than leaving a half-bumped tree
whose lock CI rejects. `uv lock` stays the authority for the normalized
prerelease form; the lock diff for a bump is a single line. RELEASING.md and
AGENTS.md now describe the lock as part of the bump.
Tests: backend/tests/test_bump_version_lockfile_sync.py (red on main, green
here) pins both halves — the lock moves with the sources, and a missing uv is an
error instead of a silent no-op.
* docs(changelog): record the release bump lockfile fix (#5859)
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(memory): redact queued memory payloads (pii_redaction follow-up)
Closes the remaining vector from #3190 (memory storage): request-scoped
redaction leaves originals in thread state, and MemoryMiddleware queued
that raw conversation into the durable extraction buffer — so the
extraction model's input and the persisted facts carried raw PII even
with pii_redaction.enabled.
- MemoryMiddleware gains pii_redaction_config (same pattern as
summarization / durable-context / title); the payload returned by
_resolve_add_args() is redacted at the enqueue boundary before
add()/aadd(), covering both after_agent and aafter_agent
- Reuses the shared redact_text() seam from #5527 via
active_pii_detectors/_Redactor/_redact_content; the buffered payload
itself becomes clean, not just the extraction model's view
- Wired at the lead-agent assembly site
- Tests: queue-payload redaction enabled/disabled, detector toggles,
message immutability, token stability across turns
* fix(memory): reserve existing placeholder indices across the queued batch
Review round 1 on #5577: thread state mixes raw user turns with
already-redacted tool results and summaries, so a token in a later
message must not collide with a new value in an earlier one. Pre-scan
the whole queued batch with redactor.reserve() before allocating.
Also extend the two MemoryMiddleware stubs in
test_lead_agent_model_resolution.py to accept the new
pii_redaction_config kwarg (CI backend-unit-tests red).
* fix(memory): redact the compaction-triggered memory_flush_hook queue too
Review round 2 on #5577: with memory + summarization enabled,
memory_flush_hook() queues list(event.messages_to_summarize) directly
into MemoryManager.add_nowait(), bypassing MemoryMiddleware entirely —
and compaction removes those messages from state right after, so the
after-agent redaction can never repair a raw batch queued there.
Extract the batch redaction (reserve pre-scan + rebuild) into a shared
redact_queued_messages() used by both MemoryMiddleware's enqueue
boundary and the hook; the hook gains an optional pii_redaction_config
wired via functools.partial at the summarization factory (subagents use
skip_memory_flush=True and are unaffected). Regression covers the hook
path enabled/disabled/absent.
* test(summarization): unwrap the partial in the factory hook identity check
edd916e wrapped memory_flush_hook in functools.partial to carry the
pii_redaction config, but missed committing the matching update to
test_factory_attaches_memory_flush_hook_by_default — a functools.partial
is not equal to its wrapped function, so the identity assertion failed
deterministically (caught by static review on #5577).
* fix(memory): cover tool-call arguments and lenient text blocks in queued payloads
Review round 4 on #5577 — two reproduced gaps still allowed raw PII into
memory payloads with pii_redaction.enabled=true:
- OpenViking retains the full message object, so parsed tool_calls args
and provider-format arguments in additional_kwargs bypassed the
content-only redaction; queued payloads now redact tool-call args
recursively (and reserve them in the pre-scan).
- DeerMem's format_conversation_for_update reads p.get("text") of any
dict block, not only type=="text" — _redact_content now redacts every
string-valued text field so a lenient consumer cannot bypass it.
Both repro shapes are pinned as regressions.
* fix(pii): value-derived placeholders for stable capture identity
Review round 5 on #5577: sequential allocation is order-dependent — the
same unchanged message re-redacted in a later batch can get a different
token, which flips OpenViking's content-signature and replays already
recorded turns into persistent memory sessions. Placeholders are now a
pure function of the matched value (sha256-6 per category): identities
stay stable across batches, enqueues, seams, and backends with no shared
state and no stored mapping.
- _Redactor becomes stateless; the reserve machinery (index pre-scans,
_PLACEHOLDER_PATTERN) is removed along with its call sites
- redact_texts keeps its signature, now a simple per-text redaction
- reserve-semantics tests are superseded by cross-enqueue identity tests
(two successive captures where the second batch carries higher existing
placeholders); the compiled-graph echo regex matches the new format
* fix(pii): 128-bit placeholder digests; redact tool-argument keys
Review round 6 on #5577:
- [P2] The 24-bit truncation collided distinct identities (reproduced:
contact3513@ / contact3727@example.com both rendered [EMAIL_575dbb]).
Retain 128 bits of the digest so separate people stay separate at any
realistic volume; the compiled-graph echo regex and the AGENTS.md entry
follow the new format.
- [P2] _redact_strings redacted values but preserved mapping keys, so
{"contacts": {"alice@example.com": "manager"}} kept the raw email
as a key. Redact string keys along with values; deterministic 128-bit
tokens make distinct-key collisions a hash-collision non-event, and a
redacted key equal to another key merges the entries. Regression pins
the email-keyed arguments shape.
* docs(agents): trim PiiRedactionMiddleware entry; drop stale reserve wording
The chain grew past the 98304-byte AG002 hard limit again (main growth
plus the round-6 format example). Trim the entry to fit, and replace the
stale reserve-indices wording with the value-derived semantics that
replaced it.
* docs(agents): shrink PiiRedactionMiddleware entry below the chain budget
main moved (#5609) and the base comparison flipped the nearly-over chain
back to an AG002 error (98441 > 98304). Trim the entry to the facts that
are not documented elsewhere: seams, allowlist, order, enqueue boundary.
* fix(pii): letters-only value-derived tokens; wire the factory SDK path
Review round 7 on #5577:
- [P2] Hex digests carry digit runs that later digit-anchored detectors
re-scan, corrupting minted tokens into nested placeholders (reproduced:
user280@example.com -> [EMAIL_…[PHONE_…]…], ~1/140 emails). The 128-bit
digest is now encoded over a-z, so a minted token survives the full
pinned detector order.
- [P2] create_deerflow_agent (the SDK path) constructed MemoryMiddleware
and DurableContextMiddleware without pii_redaction_config; the knob is
now an explicit optional parameter threaded through
_assemble_from_features (the factory reads no config files by design,
so SDK callers pass it explicitly).
- [Nit] hoist the Command rebuild comprehension out of the per-message
loop in _redact_result.
- [Nit] drop the dead _iter_tool_calls helper and Iterator import.
* test(factory): pin pii_redaction_config wiring; document the parameter
Review round 8 on #5577:
- [P3] test_create_deerflow_agent.py had zero coverage of the
pii_redaction_config parameter; two regressions now assert that
create_deerflow_agent(pii_redaction_config=cfg) builds a
MemoryMiddleware and a DurableContextMiddleware carrying cfg, and that
the default is None (redaction off).
- [Nit] document the parameter in the create_deerflow_agent docstring:
None leaves redaction off; a middleware= takeover bypasses the
assembly that would apply it.
* fix(pii): HMAC-scoped placeholder tokens; install model-bound redaction in the SDK chain
Review round 10 on #5577:
- [P2] sha256(category||value) is an unkeyed, globally deterministic
fingerprint: a provider or memory backend can confirm candidate values
offline and link identical values across users and threads, contradicting
the documented no-re-identification guarantee. Placeholders are now
HMAC-derived over a deployment-scoped token_secret (new config field);
the docstring documents the linkability scope for both modes.
- [P2] _assemble_from_features() never appended PiiRedactionMiddleware, so
the SDK knob enabled only the downstream seams while the primary model
still received raw content. The middleware is now appended in the same
relative position as the lead-agent base chain when the gate is on,
with a model-call regression through the assembled SDK chain.
* docs(pii): document token_secret; update the placeholder format examples
Follow-up to the round-10 review on #5577:
- config.example.yaml documents the new token_secret knob: a
deployment-scoped secret makes placeholders HMAC-derived and only
linkable within one deployment; leaving it unset keeps publicly
computable fingerprints (documented trade).
- the PiiRedactionConfig class docstring's placeholder example now
matches the shipped format (128-bit HMAC digests over base-26
letters).
* docs(pii): update the module docstring's placeholder format example
The module docstring still showed the sequential [EMAIL_1] example;
align it with the shipped value-derived format (128-bit HMAC digests
over base-26 letters).
* fix(pii): require a non-empty token_secret when redaction is enabled
Review round 11 on #5577: with the secret optional, the default enabled
configuration minted empty-key HMAC digests — publicly computable, globally
linkable fingerprints of the raw values, confirmable by offline guessing for
low-entropy identifiers such as phone numbers.
- PiiRedactionConfig now rejects enabled=true without a non-empty
token_secret (minimum 16 characters); the unkeyed mode is gone entirely.
- The request-scoped redactor in _process_request now carries the configured
key: it built its redactor without it, so user-message placeholders
silently diverged from every other seam for the same value. The empty-key
guard in _placeholder_token is the backstop that keeps every path keyed.
- Module/config docstrings and config.example.yaml no longer describe the
placeholders as irreversible or the secret as optional; the stale
reserve-semantics wording and the "memory path not covered" note are
corrected.
- Regressions: validation errors for missing/blank/short secrets, and the
user-message boundary minting the same keyed token as redact_text.
* fix(memory): redact original_user_content on queued memory payloads
Review round 12 on #5577: UploadsMiddleware preserves the raw user turn in
additional_kwargs.original_user_content, but redact_queued_messages rewrote
only content, tool_calls, and provider-format tool_calls — the queued copy
still carried the raw value, so the memory queue and backend received raw PII
with redaction enabled.
The shared helper now redacts the provenance string field on the queued copy
(same batch redactor, merged with the existing additional_kwargs rewrite,
rebuilt via model_copy — thread-state messages stay untouched), covering both
the enqueue boundary and the compaction flush hook. Regressions pin the
redacted provenance field through both paths and the untouched original.
* fix(memory): redact human_input_response on queued memory payloads
Review round 13 on #5577: clarification replies keep the raw user answer in
additional_kwargs.human_input_response.value (DeerMem reads that mapping via
its read_human_input_response mirror), but redact_queued_messages rewrote
only content, tool_calls, and original_user_content — the queued copy still
carried the raw value into the memory queue and backend.
The shared helper now rebuilds that mapping with value redacted (same batch
redactor) while preserving the protocol fields (version/kind/source/
request_id/option_id/response_kind), merged with the existing
additional_kwargs rewrite and rebuilt via model_copy so thread-state
messages stay untouched. Regressions pin the redacted value, preserved
protocol fields, the untouched original, and the flush-hook path.
* chore(config): bump config_version to 47; generate token_secret on upgrade
Review round 13 on #5577: existing deployments can legitimately sit at
config_version 46 with pii_redaction.enabled: true and no token_secret; the
mandatory-secret validator would fail their startup while config.example.yaml
still said 46, so the upgrade checker considered them current and emitted no
migration warning.
- Bump config_version to 47 so the outdated-config warning fires for v46.
- Add a v47 migration to scripts/config-upgrade.sh: when pii_redaction is
enabled and token_secret is missing/blank, generate a random deployment-
scoped secret (secrets.token_urlsafe) and persist it, so the upgraded
deployment starts under the mandatory validation.
- Regressions: a v46 enabled config is flagged outdated against the real
example; the upgrade run generates a >= 16-char secret and the result
passes AppConfig validation; a disabled config gets no secret.
- Also switch the two v41 script tests from a hardcoded "bash" to
SCRIPT_BASH/skipif, matching the documented Windows constraint and the
neighboring tests (hardcoded bash resolves to the WSL stub there).
* chore(helm): sync chart config_version to 47
Review round 14 on #5577: check_config_version.sh fails whenever the chart's
embedded config_version lags config.example.yaml, and the chart workflow is
triggered by config.example.yaml — so the v47 bump left this head failing
that check. Update the values.yaml config block and the README example to
47; scripts/check_config_version.sh now passes.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(models): add declarative reasoning capability contract (#5073)
Introduce an optional per-model `reasoning:` block beside the legacy
`supports_thinking` / `supports_reasoning_effort` booleans. It declares
thinking availability (unsupported | optional | required), the accepted
effort values with aliases, default, and serialization path, the payload
dialect, and the reasoning-history requirement. When present the booleans
are derived from it and contradictory profiles fail at config load.
`deerflow.models.reasoning` normalizes legacy and declared profiles into
one immutable contract and resolves every caller's generic request against
it. `create_chat_model` is the single enforcement point: required-thinking
models never receive a synthesized disable payload, unsupported effort
values never reach the provider, and the dialect synthesizes the on/off
payload when no template exists. Legacy profiles keep the historical path
byte-for-byte. The lead agent and subagent descriptor resolve the same
policy first so run metadata reports effective values.
`/api/models` and `DeerFlowClient` project the contract as `reasoning`;
the frontend derives mode clamping and effort choices from it through
`core/models/reasoning.ts`, and user preferences accept provider-specific
effort tokens. The setup wizard's Z.AI GLM-5.3-Flash profile migrates to
the contract, restoring its low/high/max effort control.
* docs(models): move the reasoning contract guidance into models/AGENTS.md
backend/AGENTS.md sits within 46 bytes of the 96 KiB effective-chain hard
limit for the middlewares, runtime, sandbox, and subagents guides, so the
new section pushed scripts/check_agent_guidance.py over it. The models
guide is the nearest file for the factory and the contract module, and
its chain has ample room.
* fix(models): address review on the reasoning contract
- vLLM dialect: the enable path now mirrors whichever chat_template_kwargs
switch the template declares, like the disable path already did, so a
template using the older `thinking` key no longer also grows
`enable_thinking`.
- Validate operator-supplied effort values at `effort.path` inside
`when_thinking_enabled`, `when_thinking_disabled`, and the `thinking`
shortcut against the contract, not only the profile-level value; they are
forwarded when the caller chooses nothing, so the "unknown values never
reach the provider" guarantee now holds for them too.
- Constrain `effort.path` to dotted identifiers and reject single-segment
paths that would replace a whole mapping (`extra_body`, `thinking`,
`model_kwargs`, `default_headers`, `default_query`).
- Z.AI GLM-5.3-Flash profile: `default: high` instead of `max`, because the
default also governs summarization, title generation, and subagent calls
that never choose an effort; `max` stays selectable. Documented in the
config example, the configuration manual, and the design note.
* fix(models): close reasoning contract review gaps
* fix(models): preserve native Ollama reasoning config
* fix(models): keep native reasoning strings and project reasoning into the descriptor
- `ModelConfig.reasoning` also accepts the level string ChatOllama takes
for gpt-oss style models (`low` / `medium` / `high`); the factory forwards
both the boolean and the string on the legacy path, so profiles that
passed `reasoning: high` through before the contract field existed keep
loading and keep sending it.
- The assembly descriptor no longer excludes `reasoning` from
`model_parameters`: the native ChatOllama value and a declared contract's
dialect/history/effort change the request payload, so they now move the
assembly fingerprint.
- e2e: the sidecar spec pinned the pre-contract body where a model without
effort control still received `reasoning_effort: minimal`; the preset is
now dropped for such models (the Gateway discarded it anyway), so the
assertion expects the key to be absent.
* fix(doctor): validate CLI credential contents
* fix(doctor): keep provider results when CLI auth state is undecodable
A credential file holding non-UTF-8 bytes made read_text raise
UnicodeDecodeError, which _load_json_object did not catch. check_llm_auth
wraps the whole model loop in one handler, so a single bad file replaced
every provider result with one generic failure and dropped the
provider-specific fix hints. Catch it alongside the malformed-state cases
and pin the behaviour with a two-model regression test.
Also cover the acceptance branches the change claims but did not test
(malformed JSON, non-object JSON, a directory as the auth path, blank
tokens, non-numeric expiresAt) and record credential_loader.py as the
source of truth for the mirrored rules, including the one place doctor is
deliberately stricter.
* fix(docker): add 'make prod-logs' entry point and hint when dev logs are empty
`make up` starts the production stack (deploy.sh: project `deer-flow`,
docker/docker-compose.yaml) while `make docker-logs` tails the dev stack
(project `deer-flow-dev`, docker-compose-dev.yaml), so after `make up`
it printed nothing (#5529), and no make entry point showed production
logs at all.
- scripts/docker.sh logs gains `--prod`: targets the stack deploy.sh
started, passes --env-file ../.env when present, and exports the same
interpolation defaults deploy.sh exports before every compose call —
without them the production volume specs fail to parse on checkouts
without .env.
- dev-only `logs` with no running containers now prints a hint pointing
at `make prod-logs` instead of staying silent.
- Makefile gains `prod-logs`, listed under Docker Production Commands.
New tests cover production targeting and the empty-state hint; they are
red on unfixed main and green here. Verified live: with a deer-flow
redis running, `logs --prod --redis` streams its logs.
Fixes#5529
* fix(docker): append --env-file after compose detection rebuilds COMPOSE_CMD
compose_preflight() probes the Compose binary and rebuilds COMPOSE_CMD,
so an --env-file appended before it was silently dropped. Append it
after preflight instead, and drive the regression test through the real
detection path (stub Docker Compose version v5.3.1, not require_compose_version)
so the append cannot regress silently.
Reviewed-in: #5538
* fix(docker): allow aio DooD socket preflight on Windows Git Bash
* test(deploy): add regression tests for deploy.sh DooD socket preflight
* fix(docker): restrict Windows DooD socket bypass to default path
Limit the Windows socket bypass in docker.sh and deploy.sh to the default /var/run/docker.sock path, and add regression tests ensuring custom missing socket paths are rejected.
* test(docker): skip unreachable socket controls on hosts with live docker socket
Add @pytest.mark.skipif on Path('/var/run/docker.sock').is_socket() to prevent false test failures on daemon-live hosts.
* fix(scripts): detect the ollama extra from configured models
make dev synced without --extra ollama, uninstalling langchain-ollama from a working setup. Same failure #2754 hit with postgres; ollama predates the detector added in #2767 and never got a rule.
* fix(deps): declare the ollama extra on backend, pin model use: matching
Review follow-ups.
`ollama` was the only extra in the detector's map that the root `backend`
project did not declare, so any consumer syncing without
`--all-packages` failed outright:
$ cd backend && uv sync --locked --extra ollama
error: Extra `ollama` is not defined in the project's
`optional-dependencies` table
`serve.sh` and `docker/dev-entrypoint.sh` both pass `--all-packages` and
were unaffected, but `backend/Dockerfile` does not, and
`config.example.yaml` documents `UV_EXTRAS` as an image build-arg — so
`UV_EXTRAS=ollama docker compose build` would have hard-failed on a
value this branch makes first-class. Declaring
`ollama = ["deerflow-harness[ollama]"]` alongside the other delegating
extras closes that, and `uv.lock` is regenerated to match.
The `use:` match also accepted any nesting depth inside `models:`, so a
`use` in a sub-mapping was read as the model's provider:
models:
- name: doubao
use: deerflow.models.patched_deepseek:PatchedChatDeepSeek
when_thinking_enabled:
use: langchain_ollama:ChatOllama
That yielded `--extra ollama` despite the model's own provider pointing
elsewhere, and `when_thinking_enabled` appears fifteen times in
config.example.yaml, so the shape is common rather than contrived. Pin
matching to the list item's own key indent, mirroring how
`section_value()` pins `child_indent` and documents deeper nesting as
ignored on purpose. Covered by a regression test that fails on the
looser parser.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* chore(deps): drop unrelated standard-aifc marker churn from uv.lock
Review follow-up. Regenerating the lock with a newer local uv (0.11.19)
also rewrote the `standard-aifc` entry, adding
`python_full_version >= '3.13'` markers to its `audioop-lts` and
`standard-chunk` dependencies. Unrelated to this change, so restore
upstream's lines and keep the lock diff to the `ollama` extra.
`uv lock --check` passes with these lines under both the CI- and
Dockerfile-pinned uv 0.11.1 and uv 0.11.19.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(scripts): detect ollama in unindented model lists and after nested sequences
Review follow-ups on the models parser.
`yaml.safe_dump` — used by the setup wizard (`make setup`) and
scripts/config-upgrade.sh — writes list items unindented:
models:
- name: qwen3-local
use: langchain_ollama:ChatOllama
The parser treated any column-0 line as the end of the `models:` section,
so the first model ended it and nothing was detected. That is the layout
new users get from the recommended setup path, so `make dev` still synced
without `--extra ollama`. Model entries are now recognised before the
section-end test, the same ordering `tools_include_name()` already uses
for the unindented tools list (#4367).
Separately, every sequence item reset the key indent, including items
inside a model option. With `stop:` / `- END` before `use:`, the key indent
jumped to the nested item's and the model's own `use` was skipped, so
detection depended on key order within the model.
The first sequence item under `models:` now fixes the model-list indent;
only items at that indent start a model and set where its keys sit.
Nested sequence items and deeper mappings are skipped without moving it.
Regression tests cover the real setup-wizard output via
`build_minimal_config()`, a hand-written unindented list, and `use:`
after a nested `stop:` list — all three fail on the previous parser — plus
a guard that a `- use:` nested inside a model option is still ignored.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(doctor): skip tool checks when tools block is empty
Follow-up to #5296, which fixed this for `models:`. The same defect
remains for `tools:`: `.get("tools", [])` returns None when the key is
present but empty, because the default only applies when the key is
absent. Iterating that None raises TypeError, which the surrounding
broad handler renders as a check result:
! web search configured ('NoneType' object is not iterable)
! web fetch configured ('NoneType' object is not iterable)
! web capture configured ('NoneType' object is not iterable)
! image search configured ('NoneType' object is not iterable)
✗ sandbox configured ('NoneType' object is not iterable)
Line 476 is reached by all four web/image checks through the shared
check_web_tool helper, and line 645 by check_sandbox.
Unlike the models case, a default install does not hit this: `make
config` ships ten real tool entries, so a user has to empty or comment
out that block first.
The web checks now fall through to their normal "no tool in config"
warning and the sandbox check evaluates normally. Parentheses on the
comprehension are for readability; `or` already binds correctly there.
Regression tests use the commented-out `tools:` shape that reproduces
the failure, matching the tests added in #5296.
Fixes#5300
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(doctor): skip non-mapping tool entries, tighten regression tests
Review follow-ups on the line this PR already changes.
A `tools:` list holding a scalar (`tools:\n - web_search`) reached
`t.get("name")` and raised AttributeError, which the broad handler
rendered as the check result:
! web search configured ('str' object has no attribute 'get')
That is the same leakage this PR removes for the null case, so it is
fixed here rather than deferred. `check_sandbox` already guards the same
way via `isinstance(tool, dict)`.
The empty-tools test asserted that "NoneType" was absent from the
detail, which pins the failure mode rather than the behaviour — it would
still pass if the detail became some other internal error text. Both
tests now assert the expected message directly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(doctor): assert sandbox outcome instead of the failure mode
Review follow-up, same class as the web-tool assertion fixed earlier in
this PR. The sandbox regression test still asserted that "NoneType" was
absent from the detail, which pins the failure mode rather than the
outcome — it would keep passing if some other internal error text leaked
out of the broad handler.
On this config the path is deterministic: an empty `tools:` means no
bash tool, so exactly one result. Assert the fields directly
(`CheckResult` has no `__eq__`, so whole instances cannot be compared by
value).
Verified against `main`'s scripts/doctor.py, where the same config
yields status=fail and detail="'NoneType' object is not iterable", so
the new assertions are red there and green here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
`config.example.yaml` ships a `models:` key with every entry commented
out, so it parses as None rather than an empty list and the `[]` default
in `.get("models", [])` never applies. Iterating that None raised
TypeError, which the surrounding broad handler rendered as a check
result:
✗ LLM API key check ('NoneType' object is not iterable)
✗ LLM auth check ('NoneType' object is not iterable)
✗ LLM package check ('NoneType' object is not iterable)
Every fresh install hit this before configuring a model, turning one
actionable error into four and hiding the real "models configured" hint
behind internal exception text.
Fall back on a falsy value at the three iteration sites so the checks
return no results when nothing is configured. `check_models_configured`
gets the same treatment for consistency; it was already correct because
it tests truthiness rather than iterating.
The existing tests missed this because they use `models: []`, an
explicit empty list, which iterates fine. The added regression tests use
the commented-out shape that `make config` actually produces.
`make doctor` now reports 1 error instead of 4 on a fresh clone.
Fixes#5295
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): add Sofya web search provider
Add a community provider backed by Sofya (https://sofya.co). Its search
endpoint returns the content of the result pages, not only their snippets,
and its fetch endpoint returns a page as markdown. Both are plain JSON over
HTTP, so this needs no extra Python package (uses httpx, already a
dependency).
Changes:
- backend/packages/harness/deerflow/community/sofya/__init__.py
- backend/packages/harness/deerflow/community/sofya/tools.py
Implements web_search_tool and web_fetch_tool using httpx.
API key is read from the config.yaml `api_key` field or the SOFYA_API_KEY
env var. Follows the same interface and output shape as the existing
ddg_search and serper providers, including the max_results parameter with
config override and the structured "No results found" error.
- backend/tests/test_sofya_tools.py
Unit tests covering API key resolution, config overrides, result mapping,
time range, HTTP errors, empty results, and fetch failures.
- config.example.yaml: add commented-out Sofya web_search and web_fetch
examples alongside the other providers
- .env.example: add SOFYA_API_KEY placeholder
- backend/docs/CONFIGURATION.md: list Sofya under web_search, web_fetch and
the environment variables
* fix(sofya): honor caller max_results, validate search_depth, join time_range contract test
- Caller-supplied max_results now wins; config is used only when the
argument is omitted, matching GroundRoute.
- search_depth is clamped to basic/snippets; an unsupported value logs a
warning and falls back to basic.
- Sofya added to the shared time_range schema contract test.
* fix(sofya): cap per-result content so a search stays inline
An unbounded search payload (up to 20 read pages) crossed the tool output
budget middleware's externalize_min_chars threshold, which replaces the
result list with a file reference. Cap each result's content at
contents_max_characters (default 2000, 0 disables), matching Exa's config
key. Five capped results stay under the 12000 char threshold.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5
* fix(sofya): list Sofya in the recency contract, coerce non-string content
_clip subscripted its input, so a non-string content or description from
the API raised TypeError instead of degrading. Coerce to text first, the
way _sofya_post and _response_results guard the shapes around it. Also add
Sofya to the Web Search Recency section in backend/AGENTS.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5
* fix(sofya): coerce web_fetch content, list sofya in the tools guide, add changelog
web_fetch sliced its content the same way web_search did before the last
push: a truthy non-string from the API passed the falsiness guard and then
raised TypeError. Reuse _clip, keeping the `or ""` so empty content still
reports "No content found".
Also add sofya to the community provider inventory in
packages/harness/deerflow/tools/AGENTS.md and an [Unreleased] changelog entry.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5
* docs(zh): add the missing InfoQuest and Firecrawl web_fetch tabs
The ZH web_fetch tab list named five providers where EN names seven. Both
tabs mirror their EN counterparts, so the two locales list the same
web_fetch providers again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
