* feat(deploy): select production Compose env files
Use an operator-selected dotenv file consistently for interpolation, container environment, secret probes and build extras. Preserve default selection and shutdown after a profile is removed.
Co-Authored-By: Codex <noreply@openai.com>
* fix(deploy): address production env selection review
Clarify Docker launcher entry points and render real Compose through the launcher to verify both service environments.
Co-Authored-By: Codex <noreply@openai.com>
* test(deploy): cover unreadable env selectors
Reject unreadable explicit selectors before Docker is invoked, using POSIX permission checks and restoring the fixture mode in finally.
Co-Authored-By: Codex <noreply@openai.com>
* test(deploy): scope dotenv assertions to selected secrets
---------
Co-authored-by: Codex <noreply@openai.com>
* fix(security): add DEER_FLOW_CREDENTIALS_KEY for at-rest credential encryption
There was no at-rest encryption key. ChannelCredentialCipher had no
production caller: the channel service, the scheduled-task notification
outbox and the channel-connections router all built
ChannelConnectionRepository without a cipher, so store_credentials refused
(RuntimeError) and get_credentials always returned None.
- New harness module deerflow.config.credentials_key: env-only
DEER_FLOW_CREDENTIALS_KEY (comma-separated Fernet keys; first encrypts,
all decrypt via MultiFernet), CredentialsCipher with encrypt_text /
decrypt_text / rotate_text, a fernet:v2: prefix with legacy fernet:v1:
values still readable, and get_credentials_cipher() as the process-wide
entry point. Malformed keys are refused without echoing key material.
Without the variable a single instance generates
{base_dir}/.credentials_key. Never derived from AUTH_JWT_SECRET.
- New deerflow.config.secret_file.read_or_create_secret_file: O_EXCL
create (0600) plus read-back with a bounded settle window, shared by
.credentials_key, .jwt_secret (fixes two replicas cold-starting on a
shared volume keeping different JWT secrets) and the managed-model key.
- ChannelCredentialCipher now builds on CredentialsCipher (same
constructors); the Gateway loads the cipher off the event loop at
startup when channel_connections is enabled and passes it to every
ChannelConnectionRepository. Undecryptable values stay "missing + warn".
- Startup gate: a malformed key stops startup in any topology; a declared
multi-instance deployment with channel_connections enabled and no key
is refused. Plain multi-worker shares the generated file and is allowed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(deploy): generate and inject DEER_FLOW_CREDENTIALS_KEY in Helm, compose and deploy.sh
- Helm: the app Secret generates a Fernet key (randBytes 32 translated to
the urlsafe alphabet) and preserves it across upgrades via lookup like
AUTH_JWT_SECRET; the gateway reads it through an optional secretKeyRef
so a user-managed existingAppSecret without it still boots (the Gateway
gate refuses only multi-instance deployments that store credentials).
README, NOTES and values document backup, rotation and existingAppSecret.
- scripts/deploy.sh resolves the key shell -> .env -> persisted file ->
generated, persisting it as $DEER_FLOW_HOME/.credentials_key (0600,
noclobber/O_EXCL with read-back): the same file the Gateway generates in
its runtime home, so either side's key is the one both keep.
- Both compose files pass DEER_FLOW_CREDENTIALS_KEY=${...:-} to the gateway.
- .env.example entry with a generation hint; README and CHANGELOG notes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): make abandoned secret-file replacement single-winner
A rename is last-writer-wins, so replicas that restart together and all
find the same abandoned empty secret file could each replace it and keep
a different value. The process that exclusively creates
<name>.replacing now owns the replacement, re-checks the file is still
empty, and publishes by renaming the claim over it; peers wait for that
value, bounded so a crashed replacer cannot block startup forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link DEER_FLOW_CREDENTIALS_KEY entries to #6611
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): name the setting that turns off a credentials-key consumer
The multi-instance refusal ended with "or disable
channel_connections.enabled=true"; it now says "or set
channel_connections.enabled=false".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): never expose a secret file before its value is written
O_EXCL published the name before its content, so a creator that stalled
mid-write left an empty file a peer replaced after the settle window; the
resumed creator then returned a value nobody persisted, or deleted the
peer's valid file when its own write failed. Creators now write and sync
a private temporary file and hard-link it to the name, which fails if
the name exists; a failed creator only removes its own temporary file.
Filesystems without hard links publish through the same single-winner
<name>.replacing claim that replaces abandoned empty files, so creation
and replacement are serialized there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(models): recover an empty managed-model key when no catalog exists
An interrupted first save could leave an empty key file; every later save
then passed empty bytes to Fernet and the endpoint returned 503 until the
file was removed by hand. An empty key with no catalog encrypted nothing,
so it now goes through the secret-file recovery path; next to an existing
catalog it is still refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(docker): pass the dotenv DEER_FLOW_CREDENTIALS_KEY through make docker-start
scripts/docker.sh runs Compose from docker/ without --env-file, so the dev
gateway's DEER_FLOW_CREDENTIALS_KEY=${DEER_FLOW_CREDENTIALS_KEY:-} entry
interpolated to empty and overrode the key the checkout .env supplied via
env_file. Export it from the dotenv like AUTH_TRUSTED_PROXIES.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(deploy): fail fast when DEER_FLOW_HOME is not writable
`make docker-start` bind-mounts the host `backend/` directory into the gateway
container, whose process runs as root, so any state it creates under
`backend/.deer-flow` ends up owned by root on the host. A later `make up` runs
`scripts/deploy.sh` as the invoking user and only fails deep into the run with a
bare "Permission denied" while persisting generated secrets (deploy.sh:234).
Check writability right after `mkdir -p "$DEER_FLOW_HOME"` and, when it fails,
print the exact `chown` recovery command instead of letting the run get halfway
through. `make down` is excluded so stopping the stack is never blocked.
Fixes#6462
* test(deploy): cover runtime home writability preflight
* fix(deploy): diagnose persisted secret permissions and preserve teardown
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(deploy): let make up start on a checkout without .env files
.env and frontend/.env are gitignored and make up does not create them,
but docker/docker-compose.yaml listed both as short-form env_file entries,
which Compose treats as required. On a fresh clone make up, make down and
make prod-logs all stopped with "env file .../.env not found" (exit 14)
before building or touching anything.
Mark the three entries optional with the long-form path/required syntax
the development compose file already uses. Compose loads them when they
exist and skips them otherwise; an unreadable or non-file .env still
fails. This needs Docker Compose 2.24+, the floor the README already
documents, and deploy.sh's startup-failure hint now names it for older
clients.
deploy.sh deliberately does not seed the files from the examples the way
docker.sh start does: .env.example carries placeholder API keys the
production Gateway would receive, and make config skips files that exist.
* docs(changelog): reference #6370
* test(deploy): gate Compose clients and cover make up
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(deploy): honor BETTER_AUTH_SECRET and DEER_FLOW_INTERNAL_AUTH_TOKEN from .env
deploy.sh only checked the shell before reloading a persisted secret or
generating a fresh one and exporting it. Compose interpolation lets shell
variables outrank --env-file, so a secret written to the repo-root .env --
the surface every deployment doc points at -- was silently replaced by the
generated one: sessions were signed with a secret the operator never chose
and Gateway workers outside the stack got 401 with the configured token.
Leave a .env-provided secret unexported so Compose parses it itself
(shell -> .env -> persisted file -> generated), and keep generating when
the shell exports an empty value, since Compose would otherwise pass the
empty string through.
* docs(changelog): reference #5928 in the deploy.sh dotenv secrets entries
* fix(deploy): decide whether .env provides a secret with Compose's own parser
read_dotenv_value only matches KEY=VALUE and returns the literal text, but
Compose also accepts `KEY: VALUE` lines and interpolates `${VAR}` inside
values. So `TOKEN: abc` looked absent (and got shadowed by a generated
token) while `TOKEN=${UNSET}` looked present (and suppressed generation,
leaving the stack with an empty token).
Ask Compose for its resolved interpolation environment instead
(`docker compose config --environment`, same --env-file, stub project on
stdin, project directory docker/) and treat a resolved-empty value as
missing. Clients older than Compose 2.28 lack the flag and fall back to
the plain reader. Tests cover the canned Compose answers, the fallback,
and every dotenv spelling against the real Compose client when installed.
* fix(deploy): resolve .env secrets through Compose on every supported client
`config --environment` needs Compose 2.28, so 2.24–2.27 clients took the
plain KEY=VALUE fallback and kept both defects: `TOKEN: abc` was missed
and shadowed, `TOKEN=${UNSET}` counted as present and left the stack with
an empty token.
Drop the fallback. Render a stub project whose only environment entry is
`${KEY}` through `docker compose config` — same --env-file and project
directory as the real command — and read the interpolated value back;
`""` means empty or unset. This is plain interpolation, available on every
Compose v2, and verified identical on 2.24.0, 2.27.0 and 2.32. A failing
probe stops the script instead of guessing. The real-Compose tests now run
against the docker CLI and any standalone binaries listed in
DEER_FLOW_TEST_COMPOSE_BINARIES.
* fix(docker): allow aio DooD socket preflight on Windows Git Bash
* test(deploy): add regression tests for deploy.sh DooD socket preflight
* fix(docker): restrict Windows DooD socket bypass to default path
Limit the Windows socket bypass in docker.sh and deploy.sh to the default /var/run/docker.sock path, and add regression tests ensuring custom missing socket paths are rejected.
* test(docker): skip unreachable socket controls on hosts with live docker socket
Add @pytest.mark.skipif on Path('/var/run/docker.sock').is_socket() to prevent false test failures on daemon-live hosts.
README documents DeerFlow as deployed by default "in a local trusted
environment (accessible only via the 127.0.0.1 loopback interface)", but both
compose files published nginx as `"${PORT:-2026}:2026"`, which Docker binds to
0.0.0.0 and [::]. The shipped artifact did not match its own documented
default, so running it on a LAN or cloud host produced a wider surface than
the docs implied without the operator changing anything -- and the agent can
execute commands.
Publish as `"${BIND_HOST:-127.0.0.1}:${PORT:-2026}:2026"` in both compose
files, so the default matches the documented model while operators who front
the stack with their own TLS/auth can still widen it via BIND_HOST. The
Gateway keeps binding 0.0.0.0:8001 inside the container (nginx reaches it over
the compose network) and its port stays unpublished, so the published nginx
port is the entire external surface.
BREAKING CHANGE: a deployment that relied on the previous 0.0.0.0 default
becomes unreachable from other hosts after this upgrade. Set BIND_HOST=0.0.0.0
in .env to restore it, after putting authentication in front and completing
first-run setup.
Also:
- .env.example documents BIND_HOST and PORT with the reasoning.
- deploy.sh reports the address the stack actually bound and, when it is not
loopback, tells the operator to complete first-run setup immediately. It
reads BIND_HOST/PORT from .env via a new read_dotenv_value helper following
compose precedence; the shell does not source .env, so reading the
environment alone would have reported "loopback only" for a stack .env had
exposed. The pre-existing ${PORT} summary line had the same defect and is
fixed with it.
- test_compose_default_bind_host.py pins the loopback default, that BIND_HOST
stays overridable, and that no service in either compose file publishes a
port without an explicit bind address, so a later addition cannot drift back
to 0.0.0.0 unnoticed.
* feat: add redis stream bridge
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix(gateway): address redis stream bridge review
Redis was imported eagerly through deerflow.runtime and declared as a hard dependency, which made memory-only installs load redis.asyncio at startup and left the lazy factory import ineffective. Move redis behind an optional extra, remove the public eager re-export, and keep make_stream_bridge as the only runtime import path with an actionable install hint when the extra is missing.
Because Docker deployments now default the stream bridge to Redis via DEER_FLOW_STREAM_BRIDGE_REDIS_URL, install the redis extra explicitly in Docker/dev container flows and teach the local uv-extra detector to infer redis from both stream_bridge.type and the Redis URL env var. This keeps Docker working while preserving slim non-Docker installs.
Harden the Redis bridge by batching XREAD replay, replacing brittle ResponseError string matching with a single fallback to 0-0 for malformed Last-Event-ID values, documenting connection/retention/fail-hard behavior, and adding fake plus opt-in real Redis coverage for XADD/XREAD, replay, invalid IDs, and MAXLEN trimming.
* fix(config): bump config version for stream bridge
* fix redis stream bridge terminal handling
* fix: repair uv.lock, format redis.py, and align Dockerfile extras test
The uv.lock file was missing a closing bracket for the redis extras
section, redis.py had a formatting issue caught by ruff, and the
Dockerfile extras test did not account for the hardcoded --extra redis
flag.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(security): mount host Docker socket only in aio (DooD) sandbox mode
The default Compose stack mounted /var/run/docker.sock read-write into the
root gateway container in every sandbox mode, including the default `local`
mode that never uses it -- an unnecessary host-escape surface (DooD =
root-equivalent host control). deploy.sh already gated the socket *check* on
sandbox_mode != local, but the Compose files mounted it unconditionally.
Move the socket mount to an opt-in docker/docker-compose.dood.yaml overlay
that deploy.sh / docker.sh append only when detect_sandbox_mode() returns
`aio`. Default (local) and provisioner/Kubernetes modes no longer expose the
host daemon. Tighten the socket existence check from != local to == aio.
Document the DooD threat model in SECURITY.md.
Reported by @greatmengqi.
* refactor(docker): address review on socket-hardening PR
- docker.sh: use absolute path for the dood overlay (match deploy.sh, drop cwd dependency)
- deploy.sh: drop now-dead DEER_FLOW_DOCKER_SOCKET exports in down/build paths
- docker-compose.yaml: fix stale header comment to point at the overlay
Addresses codex + reviewer feedback on #3517.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(auth): share internal gateway token across workers
* fix: restore deploy script executable bit
* Update deploy.sh
to skip the auth_token setup for the down command
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(deploy): fall back to python/openssl when python3 is absent for secret generation
Bare python3 call in deploy.sh exits 49 on systems without python3 in
PATH (e.g. some Alpine/minimal containers, or Windows environments where
only 'python' is on PATH). Add a fallback chain: python3 → python →
openssl rand -hex 32. If all three are unavailable, emit a clear error
message and exit with a non-zero status instead of a cryptic recipe
failure.
Closes#2922
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
- Updated documentation and comments to reflect the transition from LangGraph Server to Gateway.
- Changed default URLs in ChannelManager and tests to point to Gateway.
- Removed references to LangGraph Server in deployment scripts and configurations.
- Updated Nginx configuration to route API traffic to Gateway.
- Adjusted frontend configurations to utilize Gateway's API.
- Removed LangGraph service from Docker Compose files, consolidating services under Gateway.
- Added regression tests to ensure Gateway integration works as expected.
Co-authored-by: Copilot <copilot@github.com>
* refactor: extract shared utils to break harness→app cross-layer imports
Move _validate_skill_frontmatter to src/skills/validation.py and
CONVERTIBLE_EXTENSIONS + convert_file_to_markdown to src/utils/file_conversion.py.
This eliminates the two reverse dependencies from client.py (harness layer)
into gateway/routers/ (app layer), preparing for the harness/app package split.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: split backend/src into harness (deerflow.*) and app (app.*)
Physically split the monolithic backend/src/ package into two layers:
- **Harness** (`packages/harness/deerflow/`): publishable agent framework
package with import prefix `deerflow.*`. Contains agents, sandbox, tools,
models, MCP, skills, config, and all core infrastructure.
- **App** (`app/`): unpublished application code with import prefix `app.*`.
Contains gateway (FastAPI REST API) and channels (IM integrations).
Key changes:
- Move 13 harness modules to packages/harness/deerflow/ via git mv
- Move gateway + channels to app/ via git mv
- Rename all imports: src.* → deerflow.* (harness) / app.* (app layer)
- Set up uv workspace with deerflow-harness as workspace member
- Update langgraph.json, config.example.yaml, all scripts, Docker files
- Add build-system (hatchling) to harness pyproject.toml
- Add PYTHONPATH=. to gateway startup commands for app.* resolution
- Update ruff.toml with known-first-party for import sorting
- Update all documentation to reflect new directory structure
Boundary rule enforced: harness code never imports from app.
All 429 tests pass. Lint clean.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: add harness→app boundary check test and update docs
Add test_harness_boundary.py that scans all Python files in
packages/harness/deerflow/ and fails if any `from app.*` or
`import app.*` statement is found. This enforces the architectural
rule that the harness layer never depends on the app layer.
Update CLAUDE.md to document the harness/app split architecture,
import conventions, and the boundary enforcement test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add config versioning with auto-upgrade on startup
When config.example.yaml schema changes, developers' local config.yaml
files can silently become outdated. This adds a config_version field and
auto-upgrade mechanism so breaking changes (like src.* → deerflow.*
renames) are applied automatically before services start.
- Add config_version: 1 to config.example.yaml
- Add startup version check warning in AppConfig.from_file()
- Add scripts/config-upgrade.sh with migration registry for value replacements
- Add `make config-upgrade` target
- Auto-run config-upgrade in serve.sh and start-daemon.sh before starting services
- Add config error hints in service failure messages
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix comments
* fix: update src.* import in test_sandbox_tools_security to deerflow.*
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: handle empty config and search parent dirs for config.example.yaml
Address Copilot review comments on PR #1131:
- Guard against yaml.safe_load() returning None for empty config files
- Search parent directories for config.example.yaml instead of only
looking next to config.yaml, fixing detection in common setups
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: correct skills root path depth and config_version type coercion
- loader.py: fix get_skills_root_path() to use 5 parent levels (was 3)
after harness split, file lives at packages/harness/deerflow/skills/
so parent×3 resolved to backend/packages/harness/ instead of backend/
- app_config.py: coerce config_version to int() before comparison in
_check_config_version() to prevent TypeError when YAML stores value
as string (e.g. config_version: "1")
- tests: add regression tests for both fixes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: update test imports from src.* to deerflow.*/app.* after harness refactor
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
