mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-26 21:26:39 +00:00
* fix(security): add DEER_FLOW_CREDENTIALS_KEY for at-rest credential encryption
There was no at-rest encryption key. ChannelCredentialCipher had no
production caller: the channel service, the scheduled-task notification
outbox and the channel-connections router all built
ChannelConnectionRepository without a cipher, so store_credentials refused
(RuntimeError) and get_credentials always returned None.
- New harness module deerflow.config.credentials_key: env-only
DEER_FLOW_CREDENTIALS_KEY (comma-separated Fernet keys; first encrypts,
all decrypt via MultiFernet), CredentialsCipher with encrypt_text /
decrypt_text / rotate_text, a fernet:v2: prefix with legacy fernet:v1:
values still readable, and get_credentials_cipher() as the process-wide
entry point. Malformed keys are refused without echoing key material.
Without the variable a single instance generates
{base_dir}/.credentials_key. Never derived from AUTH_JWT_SECRET.
- New deerflow.config.secret_file.read_or_create_secret_file: O_EXCL
create (0600) plus read-back with a bounded settle window, shared by
.credentials_key, .jwt_secret (fixes two replicas cold-starting on a
shared volume keeping different JWT secrets) and the managed-model key.
- ChannelCredentialCipher now builds on CredentialsCipher (same
constructors); the Gateway loads the cipher off the event loop at
startup when channel_connections is enabled and passes it to every
ChannelConnectionRepository. Undecryptable values stay "missing + warn".
- Startup gate: a malformed key stops startup in any topology; a declared
multi-instance deployment with channel_connections enabled and no key
is refused. Plain multi-worker shares the generated file and is allowed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(deploy): generate and inject DEER_FLOW_CREDENTIALS_KEY in Helm, compose and deploy.sh
- Helm: the app Secret generates a Fernet key (randBytes 32 translated to
the urlsafe alphabet) and preserves it across upgrades via lookup like
AUTH_JWT_SECRET; the gateway reads it through an optional secretKeyRef
so a user-managed existingAppSecret without it still boots (the Gateway
gate refuses only multi-instance deployments that store credentials).
README, NOTES and values document backup, rotation and existingAppSecret.
- scripts/deploy.sh resolves the key shell -> .env -> persisted file ->
generated, persisting it as $DEER_FLOW_HOME/.credentials_key (0600,
noclobber/O_EXCL with read-back): the same file the Gateway generates in
its runtime home, so either side's key is the one both keep.
- Both compose files pass DEER_FLOW_CREDENTIALS_KEY=${...:-} to the gateway.
- .env.example entry with a generation hint; README and CHANGELOG notes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): make abandoned secret-file replacement single-winner
A rename is last-writer-wins, so replicas that restart together and all
find the same abandoned empty secret file could each replace it and keep
a different value. The process that exclusively creates
<name>.replacing now owns the replacement, re-checks the file is still
empty, and publishes by renaming the claim over it; peers wait for that
value, bounded so a crashed replacer cannot block startup forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link DEER_FLOW_CREDENTIALS_KEY entries to #6611
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): name the setting that turns off a credentials-key consumer
The multi-instance refusal ended with "or disable
channel_connections.enabled=true"; it now says "or set
channel_connections.enabled=false".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): never expose a secret file before its value is written
O_EXCL published the name before its content, so a creator that stalled
mid-write left an empty file a peer replaced after the settle window; the
resumed creator then returned a value nobody persisted, or deleted the
peer's valid file when its own write failed. Creators now write and sync
a private temporary file and hard-link it to the name, which fails if
the name exists; a failed creator only removes its own temporary file.
Filesystems without hard links publish through the same single-winner
<name>.replacing claim that replaces abandoned empty files, so creation
and replacement are serialized there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(models): recover an empty managed-model key when no catalog exists
An interrupted first save could leave an empty key file; every later save
then passed empty bytes to Fernet and the endpoint returned 503 until the
file was removed by hand. An empty key with no catalog encrypted nothing,
so it now goes through the secret-file recovery path; next to an existing
catalog it is still refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(docker): pass the dotenv DEER_FLOW_CREDENTIALS_KEY through make docker-start
scripts/docker.sh runs Compose from docker/ without --env-file, so the dev
gateway's DEER_FLOW_CREDENTIALS_KEY=${DEER_FLOW_CREDENTIALS_KEY:-} entry
interpolated to empty and overrode the key the checkout .env supplied via
env_file. Export it from the dotenv like AUTH_TRUSTED_PROXIES.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
153 lines
5.4 KiB
Python
153 lines
5.4 KiB
Python
"""Tests for AuthConfig typed configuration."""
|
|
|
|
import os
|
|
from unittest.mock import patch
|
|
|
|
import pytest
|
|
|
|
import app.gateway.auth.config as cfg
|
|
|
|
|
|
def test_auth_config_defaults():
|
|
config = cfg.AuthConfig(jwt_secret="test-secret-key-123")
|
|
assert config.token_expiry_days == 7
|
|
|
|
|
|
def test_auth_config_token_expiry_range():
|
|
cfg.AuthConfig(jwt_secret="s", token_expiry_days=1)
|
|
cfg.AuthConfig(jwt_secret="s", token_expiry_days=30)
|
|
with pytest.raises(Exception):
|
|
cfg.AuthConfig(jwt_secret="s", token_expiry_days=0)
|
|
with pytest.raises(Exception):
|
|
cfg.AuthConfig(jwt_secret="s", token_expiry_days=31)
|
|
|
|
|
|
def test_auth_config_from_env():
|
|
env = {"AUTH_JWT_SECRET": "test-jwt-secret-from-env"}
|
|
with patch.dict(os.environ, env, clear=False):
|
|
old = cfg._auth_config
|
|
cfg._auth_config = None
|
|
try:
|
|
config = cfg.get_auth_config()
|
|
assert config.jwt_secret == "test-jwt-secret-from-env"
|
|
finally:
|
|
cfg._auth_config = old
|
|
|
|
|
|
def test_auth_config_missing_secret_generates_and_persists(tmp_path, caplog):
|
|
import logging
|
|
|
|
from deerflow.config.paths import Paths
|
|
|
|
old = cfg._auth_config
|
|
cfg._auth_config = None
|
|
secret_file = tmp_path / ".jwt_secret"
|
|
try:
|
|
with patch.dict(os.environ, {}, clear=True):
|
|
os.environ.pop("AUTH_JWT_SECRET", None)
|
|
with patch("deerflow.config.paths.get_paths", return_value=Paths(base_dir=tmp_path)), caplog.at_level(logging.WARNING):
|
|
config = cfg.get_auth_config()
|
|
assert config.jwt_secret
|
|
assert any("AUTH_JWT_SECRET" in msg for msg in caplog.messages)
|
|
assert secret_file.exists()
|
|
assert secret_file.read_text().strip() == config.jwt_secret
|
|
finally:
|
|
cfg._auth_config = old
|
|
|
|
|
|
def test_auth_config_reuses_persisted_secret(tmp_path):
|
|
from deerflow.config.paths import Paths
|
|
|
|
old = cfg._auth_config
|
|
cfg._auth_config = None
|
|
persisted = "persisted-secret-from-file-min-32-chars!!"
|
|
(tmp_path / ".jwt_secret").write_text(persisted, encoding="utf-8")
|
|
try:
|
|
with patch.dict(os.environ, {}, clear=True):
|
|
os.environ.pop("AUTH_JWT_SECRET", None)
|
|
with patch("deerflow.config.paths.get_paths", return_value=Paths(base_dir=tmp_path)):
|
|
config = cfg.get_auth_config()
|
|
assert config.jwt_secret == persisted
|
|
finally:
|
|
cfg._auth_config = old
|
|
|
|
|
|
def test_auth_config_empty_secret_file_generates_new(tmp_path):
|
|
from deerflow.config.paths import Paths
|
|
|
|
old = cfg._auth_config
|
|
cfg._auth_config = None
|
|
(tmp_path / ".jwt_secret").write_text("", encoding="utf-8")
|
|
try:
|
|
with patch.dict(os.environ, {}, clear=True):
|
|
os.environ.pop("AUTH_JWT_SECRET", None)
|
|
with patch("deerflow.config.paths.get_paths", return_value=Paths(base_dir=tmp_path)):
|
|
config = cfg.get_auth_config()
|
|
assert config.jwt_secret
|
|
assert len(config.jwt_secret) > 20
|
|
assert (tmp_path / ".jwt_secret").read_text().strip() == config.jwt_secret
|
|
finally:
|
|
cfg._auth_config = old
|
|
|
|
|
|
def test_jwt_secret_creation_keeps_a_peer_replica_value(tmp_path, monkeypatch):
|
|
"""Two replicas cold-starting on a shared home volume must sign with one key.
|
|
|
|
Deterministic interleaving: the peer creates ``.jwt_secret`` after this
|
|
process found none but before it writes. A truncating write would replace
|
|
the peer's value and both replicas would keep different secrets.
|
|
"""
|
|
from deerflow.config.paths import Paths
|
|
|
|
secret_file = tmp_path / ".jwt_secret"
|
|
|
|
def peer_wins_the_race(_nbytes):
|
|
secret_file.write_text("peer-replica-secret-value-0123456789", encoding="utf-8")
|
|
return "this-replica-secret-value-0123456789"
|
|
|
|
monkeypatch.setattr(cfg.secrets, "token_urlsafe", peer_wins_the_race)
|
|
with patch("deerflow.config.paths.get_paths", return_value=Paths(base_dir=tmp_path)):
|
|
secret = cfg._load_or_create_secret()
|
|
|
|
assert secret == "peer-replica-secret-value-0123456789"
|
|
assert secret_file.read_text(encoding="utf-8") == "peer-replica-secret-value-0123456789"
|
|
|
|
|
|
def test_concurrent_jwt_secret_creators_converge(tmp_path):
|
|
import threading
|
|
|
|
from deerflow.config.paths import Paths
|
|
|
|
workers = 8
|
|
barrier = threading.Barrier(workers)
|
|
results: list[str] = []
|
|
lock = threading.Lock()
|
|
|
|
def run():
|
|
barrier.wait(timeout=10)
|
|
value = cfg._load_or_create_secret()
|
|
with lock:
|
|
results.append(value)
|
|
|
|
with patch("deerflow.config.paths.get_paths", return_value=Paths(base_dir=tmp_path)):
|
|
threads = [threading.Thread(target=run) for _ in range(workers)]
|
|
for thread in threads:
|
|
thread.start()
|
|
for thread in threads:
|
|
thread.join(timeout=30)
|
|
|
|
assert len(results) == workers
|
|
assert len(set(results)) == 1
|
|
assert (tmp_path / ".jwt_secret").read_text(encoding="utf-8") == results[0]
|
|
if os.name == "posix":
|
|
assert (tmp_path / ".jwt_secret").stat().st_mode & 0o777 == 0o600
|
|
|
|
|
|
def test_unusable_jwt_secret_location_keeps_the_actionable_error(tmp_path):
|
|
from deerflow.config.paths import Paths
|
|
|
|
not_a_directory = tmp_path / "home"
|
|
not_a_directory.write_text("", encoding="utf-8")
|
|
with patch("deerflow.config.paths.get_paths", return_value=Paths(base_dir=not_a_directory)), pytest.raises(RuntimeError, match="AUTH_JWT_SECRET"):
|
|
cfg._load_or_create_secret()
|